Evaluation database

Evaluation report

2003 IDS: Psychosocial Interventions Evaluation of UNICEF Supported Projects (1999- 2001)



Author: Melville, A.; Scarlet, F.

Executive summary

Background

Throughout 1999-2001, Indonesia was experiencing a surge of internal armed conflict. During this period, UNICEF launched two projects to promote psychosocial well-being and recovery of children who have experienced armed conflict in their communities. The first, the Psychosocial Help Training Project, aimed to support teachers and counsellors to provide basic psychosocial assistance to children, first in West Timor and then throughout the conflict zones in Indonesia. The second was the EMDR project, which trained professionals and paraprofessionals in the use of Eye Movement and Desensitisation Reprocessing. The project aimed to treat the effects of trauma, and as such, was a treatment intervention.

Purpose / Objective

This evaluation assessed the design, implementation and impact of the two major psychosocial projects sponsored by UNICEF Indonesia from October 1999 until the end of 2001.

Methodology

This evaluation was conducted utilizing three different sources of information: desk study of relevant documents; interviews with the project staff and the trainees; and focus groups, interviews and questionnaires with beneficiaries. Focus group discussions with IDP children were held in Madura and Lhokseumawe, and questionnaires distributed to children in Madura and two locations in Aceh. 40 children were involved in focus group discussions. 152 children who participated in the projects completed questionnaires while 225 children not involved in the project completed questionnaires for comparison.

Key Findings and Conclusions

PSYCHOSOCIAL HELP TRAINING PROJECT
Design and Implementation:
Overall, the design of the Psychosocial Help project was logical and effective, particularly the design of the first phase in West Timor. The project consisted of a series of psychosocial trainings for teachers, counsellors and youth volunteers in West Timor, with substantial follow-up to support the activities in the East Timorese refugee camps. The second phase was a psychosocial training workshop in Jakarta for counsellors and children's workers from throughout the conflict zones in Indonesia, which was followed by 3 district level trainers for caretakers of separated children. The main beneficiaries where children and parents experiencing psychosocial difficulties as result of armed conflict but not those with clinical psychological problems - that is, the project did not provide psychological or psychiatric treatment.

The strengths of this design were that in West Timor, it was a comprehensive, multilevel intervention - that is, the first level of intervention was by parents, teachers and youth volunteers and for those needing more specialised assistance, they could be referred to the second level of intervention conducted by counsellors. In West Timor, a comprehensive field-based assessment was conducted, and the projects included strong follow-up and support to activity implementation. Due to the continual presence of UNICEF staff in the field, the project was flexible to the changing circumstances and needs, and emerging problems were effectively overcome. In West Timor the activities were integrated with UNICEF educational and health projects, while the district-level trainings in other parts of Indonesia were integrated within UNICEF's Children in Need of Special Protection Section. In both West Timor and the project for other conflict areas, the activities focused on strengthening children's support networks, and emphasised child resiliency, which are ethical, effective and efficient ways to promote children's healing. Activities included recreational and expressional activities, and in other conflict zones also included skills building (such as relaxation) and vocational training. These activities were mostly conducted in groups which was culturally appropriate and an efficient way to reach large numbers of affected children. The trainers were very knowledgeable, experienced and used an effective training style - that is, they were elicitive, participatory, flexible and culturally sensitive. All activities were conducted in close collaboration with relevant organizations, and build local capacity. Finally, the project was timely given the constraints, particularly of working in the emergency situation in West Timor.

Some areas to improve for future projects would be to involve children in the project design and implementation, to incorporate clear protocol from beginning of project, and to implement activities by members of the community (such as youth volunteers and teachers) first so as to build confidence with the community and children, and after this implement activities by counselors. Overall, the training topics were good, but there needed to be a better balance between knowledge/attitude change and skills building, and there was a need for more specific group counseling techniques for children. There may be a need to supplement group activities with some specific individual interventions - for instance, individual follow-up for vulnerable children. There was also a tendency to overestimate the skill level of trainees - for instance, it was also clear that 5 days is not enough to train counselors. Selection of participants for these trainings need to be done very carefully, and should not be left entirely to counterparts. While it is important to coordinate and involve many different organizations in the project as beneficiaries or advisors, the number of partners responsible for implementation should be limited. For the Jakarta training there was little follow-up and this was a major short-coming of this training. This appeared to be at least partly due to the initiation of EMDR and the subsequent lack of funds to continue with psychosocial interventions.

Impact:
This project had a substantial impact on a large number of primary beneficiaries - that is, children and their parents. In West Timor approximately 9,000 children participated in recreational and/or expressional activities, 1,000 children received counseling, and 500 parents participated in community meeting or support groups. 20,000 children in other conflict areas (primarily Madura, Central Kalimantan, Malukus and East Java) benefited from recreational and/or expressional activities. It is estimated that 19% of the targeted beneficiaries were reached by this project, which is very substantial proportion in the Indonesian context. It was found that children who participated in psychosocial activities had a lasting (at least one year) significantly better psychosocial outcome than those who did not have any psychosocial assistance. Benefits included: children were happier, more animated, developed their social skills, had better family relationships, were more able to make up their mind, were less scared, hyperactive, and regressive and had fewer psychosomatic problems; the refugee camps were more animated, and care-takers were empowered to improve the situation of children in these camps. This was also an efficient project, as it cost approximately $9 per beneficiary. Beneficiaries were primarily children under 12 years old, particularly in West Timor, so future projects need to focus more on the adolescent age group.

For professionals and para-professionals, the impact was also important. A total of 60 counselors were trained, 160 teachers/care-takers and 60 youth volunteers were trained. Key changes in the trainees included: greater self-confidence and motivation to work with children in armed conflict, understanding of children as active learners, recognition of the importance of self-knowledge and expression, and skills to help children express/play. Finally this project served as framework for much of the psychosocial work conducted in Indonesia - for instance, the training manual and approach has heavily influenced the psychosocial work of two key Universities in Indonesia.

In conclusion, the psychosocial training project is an easily implementable, practical project that has proven psychosocial benefits for children and their communities. It is cost efficient, and generally consistent with UNICEF policies and guidelines. It is an approach that should be continued and strengthened, particularly based on the West Timor model with minor adjustments to integrate the lessons learnt that are highlighted in this evaluation.

EMDR PROJECT
Design and Implementation:
The EMDR project consisted of a series of 5 trainings and 1 week field supervision to train mental health professionals and para-professionals in the use of EMDR for children and families traumatised by violence. As such, it aimed to treat clinical psychological problems resulting from exposure to sudden, life-threatening events. It mainly focused on Aceh and the Malukus.

The strengths of the design and implementation of this project were that: it build local capacity (through training) and was a collaboration with local organisations; the design was logical as it included step-based training with practical assignments and supervision; it did contribute to the individual aspect of healing process; it helped some beneficiaries to overcome their problems, particularly uncontrollable fear of the perpetrators of violence; and did provide a psychological technique to deal with the effects of trauma to some paraprofessionals who previously had none. The project was more effective in Maluku than Aceh, primarily because in Maluku a core group of the trainees had experience and opportunity working with children, and had basic counseling skills (see below). Generally, EMDR was implemented along with other activities, such as play groups, or cognitive, behavioral or relaxation therapies.

None-the-less there where some serious problems. First, the project focused exclusively on problems resulting from exposure to trauma, but many beneficiaries stated that psychosocial problems resulting from other issues, such as dislocation, family problems, loss of loved one etc. were a higher priority for them. It did not help to normalize beneficiaries' lives and was not a holistic approach to their problems. Resilience was not core concept, it did not focus on strengthening children's support networks and it utilized a medical model, which can be disempowering for the beneficiaries. There was no children involvement in the design, and the needs assessment was biased and not sufficiently field-based. The training topics did not contextualize EMDR in the Indonesian context - for instance it did not address signs of trauma in Indonesia. Despite substantial experience, the first set of trainers used a didactic and inflexible training approach. EMDR is a treatment methodology which is not the usual focus of UNICEF's programming, and as described above, this approach was not consistent with any of UNICEF's policies, except building local capacity. Many beneficiaries, and some trainees, found the process difficult, strange, and time-consuming which calls into question the cultural appropriateness of this technique within Indonesia. The individual nature of EMDR for adults was inefficient for dealing with large numbers of affected people. Finally, and perhaps most importantly, the field based paraprofessionals often did not have basic counseling skills to use EMDR effectively or ethically. This raised the danger of unsupervised and unsupported paraprofessionals raising the beneficiaries traumatic memories and not being able to effectively process them (this occurred in a least a few cases).

Impact:
There were a limited number of beneficiaries of this project, and there was little impact of these activities that could be demonstrated. Beneficiaries were estimated at 500-600, of which an estimated 175-250 were children. It is estimated that less than 1% of the targeted population benefited from EMDR services. The cost per beneficiary was $430-500 or $860-1000 per UNICEF targeted beneficiary (children or parents). The majority of beneficiaries were from Maluku, and a more limited number from Aceh and Madura/Surabaya. There was no difference found between the overall psychosocial outcomes for children who participated in EMDR activities, and those that did not participate in any psychosocial activities. This may have been partly due to the fact that the quality of EMDR interventions varied widely. A recent meta-analysis concludes that EMDR does have an impact, but that it is similar to certain other types of treatment, and that the bi-lateral stimulation which is the supposed core of the treatment is not crucial to its effectiveness. In this project, the impact on the trainees was mixed, with some trainees who had basic counselling skills happy and effectively able to add EMDR to their range of intervention techniques and other paraprofessionals without basic skills very happy to have at least one technique to deal with traumatized children/adults. Other trainees were frustrated because they found EMDR so difficult to implement.

In conclusion, EMDR is difficult to implement and generally inappropriate for the Indonesian context, particularly given the limited basic counseling skills of child workers in conflict zones but also because of the questionable cultural appropriateness of the technique. In this project, EMDR had limited, if any, lasting positive effects on children that could be demonstrated, and it does not appear to be more effective or efficient than other therapies which may be more easily implemented in Indonesia. It is inconsistent with UNICEF policies and guidelines, and not cost effective. At this stage, EMDR is not appropriate for helping children deal with the psychological and social effects of armed conflict in Indonesia.

Recommendations

It is recommended that the UNICEF Indonesia psychosocial programming should have two primary goals:
- Strengthening community-based social supports for children, including re-establishing stable family life, and mobilising para-professionals
- Building children's resilience, including normalisation of their life, healing past wounds and building their psychosocial skills

The lessons learnt and recommendations for future programming from this evaluation are summarised below for easy reference:

Design:
- Psychosocial projects are a child's right. Such programmes are important to children in armed conflict but sometimes have to overcome adult focus on material needs.
- Programme decisions and priorities must derive from a situation analysis on the ground.
- Children should have an active role in designing and implementing programmes.
- Where possible, projects should have multilevel design. Promotion activities, to help parents, teachers, youth volunteers and children themselves better support children, should be combined with prevention activities, by counsellors and other psychosocial specialists, to help children who need more specialised assistance.
- Structured psychosocial activities should be integrated with supportive psychosocial activities such as family reintegration and educational programmes. Often these supportive psychosocial activities will be managed under a separate project, but they should form an integrated assistance package.
- Incorporate clear evaluation protocol from beginning of project.
- Support activity implementation and follow-up trainings for all projects.
- Should take care to ensure that approaches are culturally appropriate and workable before implementation.

Approach:
- Psychosocial programmes should focus on strengthening child resiliency and children's support networks.
- Projects should focus on healing, including normalizing life. Should not only focus on expressional or recreational activities, but should take children through a healing process to help build their skills to overcome their problems.
- Treatment of psychological problems resulting from trauma should be dealt with as part of an intervention that deals with other sources of stress for the beneficiaries, such as those resulting from dislocation, family tension/violence, grief or schooling/economic pressures.
- Need to ensure trainees have or are trained in basic counselling skills before providing a training in a treatment methodology such as EMDR.
- Should not use medical model where the active expert is seen as having the knowledge to cure the sick, passive patient.
- Ensure integration of peace-building concepts in psychosocial projects, such as tolerance, cooperation etc, particularly in areas where there conflict levels have significantly reduced.

Implementation:
- Implement activities by youth volunteers and teachers first to build confidence with community/children, and after implement activities by counsellors.
- Training topics should have a balance between knowledge/attitude change and skills building.
- Need to include specific art/play group counselling techniques for children.
- Need longer than 5 days to train counsellors.
- Training should be elicitive, participatory, flexible and culturally sensitive.
- Should carefully select trainees.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2003

Region:
EAPRO

Country:
Indonesia

Type:
Evaluation

Theme:
Health - Mental Health

Partners:

PIDB:

Follow Up:

Language:
English

Sequence Number:
2003/800

New enhanced search