Evaluation database

Evaluation report

2002 BHG: Evaluation of the UNICEF School-Based Psychosocial Program for War-Exposed Adolescents as Implemented During the 2000-2001 School Year



Author: Layne, C.M.; Davies, R.; Burlingame, G.M.; Saltzman, W. R.; Thomas, N.; Pynoos, R. S.

Executive summary

Background

This report summarizes an evaluation of the UNICEF School-Based Psychosocial Program for War-Exposed Adolescents. This specialized program has been implemented since the 1997-1998 school year in Bosnian secondary schools by specially trained school counselors under the supervision of trained mental health professionals working in the community. The program is designed to identify and therapeutically support Bosnian adolescents with histories of severe war trauma who continue to experience significant psychosocial problems after the war. A centerpiece of the program is a manualized, 20+-session trauma/grief-focused group treatment program in which the school counselors lead a specialized trauma/grief-focused therapy group with up to 10 of the most severely traumatized students at the school.

Purpose / Objective

The purpose of this report is to summarize, from a broad perspective, the most significant impacts of the program at varying levels. Thus, it will selectively review those evaluation findings that are expected to be of greatest interest in answering the question of whether the program was effective, and how. This program evaluation is divided into three sections that correspond with three basic evaluative questions. The first section will address the question, "What happened and how does this compare with what was expected?" In this section, the results of the evaluation will be reviewed according to program objectives. The second section will address the question, "Why and how did it happen or not happen?" by reviewing factors that appear to have contributed to the program's effectiveness in some regions, and its lack of effectiveness in other regions. The third section will address the question, "What should be done about it?" by making recommendations that reflect the findings of this evaluation and the cumulative experience of the Team and its local partners.

Methodology

A total of 10 secondary schools located in the Central Bosnia (Travnik and Novi Travnik) and Sarajevo Cantons took part in the evaluation. Quantitative data were collected throughout the school year, as indicated by the implementation plan, and utilized measures completed by the student group members, their primary caretakers, and their teachers. The post-treatment follow-up was conducted at the end of the school year and utilized the same sources. Data were collected from both students and group leaders using a semi-structured focus group format. Student groups ranged from 7-9 members and were held in participating Sarajevo and Travnik area schools. Group leader focus groups ranged from approximately 10-15 members and were held at the end of the post-year supervision meeting in Travnik.

Key Findings and Conclusions

The evaluation revealed consistently positive results across a broad variety of program dimensions. In particular, focus groups conducted with the school counselors and students revealed a broad program impact on the levels of individual student group members, their families, peers, the classroom and school environment, and the larger community.

The school counselors generally reported that participation in the program had brought about a change/expansion in their roles in the schools from one of disciplinarians of disruptive students to valued providers of mental health services. The counselors attributed their participation in the program as instrumental in increasing their expertise and expanding their professional roles. A large and skilled professional network of school counselors, community mental health professionals, and local universities has been created via the program. This network has great potential for assuming an advocacy role for public policy, defining the professional role of the pedagogue/psychologist in the schools, and mental health program development and implementation in the schools.

The great majority of school counselors reported that they incorporated the program materials and training into their other professional work at the schools, which greatly increased the number of direct beneficiaries.

Students identified a wide range of program benefits, including the acquisition of effective coping and problem-solving skills, increased self-esteem, hope in the future, improved family and peer relationships, and improved school performance. They also reported that they advocated for the program by passing on skills and materials to family and peers, making referrals to the school counselor, and sticking up for other students with similar problems. Teachers generally concurred with the students, identifying gains in school performance and peer relationships at higher rates in the treatment group compared to the control group.

Quantitative analyses revealed significant pre- to post-treatment reductions in targeted distress symptoms (as measured by measures of posttraumatic stress, depression, and grief symptoms). Rates of reliable improvement were higher in the treatment group compared to the wait list control group, indicating that treatment was generally of greater benefit than holding intermittent assessment visits with the school counselor.

The supervision structure of the program (in which the counselors meet with community mental health professionals in group supervision meetings) worked well throughout the 2000-2001 school year. In particular, the program operated with only minimal off-site support by the UCLA/BYU Team throughout the entire 2000-2001 school year.

Lessons Learned

Since the spring of 1997, a total of 43 secondary schools located in 6 major regions throughout the country have been trained to implement the program. At the close of the 2000-2001 school year, a total of 22 schools were participating in three regions throughout the country, producing a retention rate of 58% among the schools and 50% among the regions. Why did the program "take root" in some regions, and die out in other regions with equal or higher levels of need for psychosocial support? Lessons learned will now be reviewed and summarized in the form of good practice recommendations:

1. Design a program that the local participants perceive and treat as locally owned and operated rather than as an externally imposed set of obligations. In particular, the program will take root and become sustainable only when the local participants become personally convinced that the program helps them to carry out their professional work in a more competent, effective fashion.

2. Select the right government administrators to support program implementation. The program has generally flourished in regions where informed, competent, and committed administrative support is present, and perished or struggled in areas where such administrative support is lacking. The importance of selecting, educating, supporting, and regularly monitoring the activities of government coordinators cannot be overstated.

3. Recruit and retain competent and committed community mental health professionals to work as program supervisors.

4. Recruit school counselors to implement the program who have received specialized training in implementing the program, who have good rapport with their students, good interpersonal skills, competence in their professional roles, and who are compassionately dedicated to helping their students.

5. The program must contain built-in incentives to continue implementation, such as observing improvements in students, receiving recognition for the counselors' increased expertise, and receiving expressions of gratitude from parents and teachers. Attending supervision meetings and increasing their professional expertise also served as powerful incentives.

6. Create a local program administrative infrastructure that is as self-supporting and self-sustaining as possible. The supervisors, in particular, have played an invaluable role in providing timely and effective on-site support.

7. As needed, especially in the initial stages of implementation, support sustained external commitment. Repeated and intermittent pulsed visits by external consultants that are focused on current problems and upgrading skills have helped to generate momentum, maintain local commitment, reinforce interpersonal and inter-institutional ties, update understanding of local needs, and to promote the quality of program implementation and evaluation.

Recommendations

Group members identified skills as being the most important and valuable positive outcome for the trauma group program. These skills are adaptable for use in a classroom setting with a well-trained and well-supported teacher or pedagogue, and carry a relatively low risk of negative impacts. Thus, efforts should be undertaken to systematically review the program materials and to import and/or adapt them for classroom instruction.

Teachers and pedagogues have requested in-depth training prior to implementing a general classroom-based psychosocial program. Specifically, they requested training in the underlying theoretical model, and in the correct practice/ presentation of the specific skills and concepts covered. Thus, the development of an implementation strategy should also include plans for a formal initial training of the implementing teachers/pedagogues. Ongoing supervision should also be made available as needed.

The working group that is charged with developing recommendations and methods for guiding this adaptation/development of a general psychosocial program should draw liberally on the experiences of the school counselors and their supervisors who have taken personal initiative in adapting the program for use in their regular counseling work and in the classroom.

A valuable impact of the program is the development of a large professional network of trained school counselors, and the establishment/strengthening of formal ties between local schools, mental health clinics, universities, and Pedagogical Institutes. This network has the potential of evolving to become an advocate for the role of the pedagogue/psychologist in the schools and the professional training of pedagogues and psychologists during and after their formal schooling. Generally, this network may serve as an important mechanism for promoting and expanding mental health services in various forms throughout Bosnian schools. Thus, the support and continuation of this network should be designated a high priority, and it should be provided the support needed to continue both its appointed work and its own evolution.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2002

Region:
CEE/CIS

Country:
Bosnia-Herzegovina

Type:
Evaluation

Theme:
Health - Mental Health

Partners:

PIDB:
MS 109 (1054)

Follow Up:

Language:
English

Sequence Number:
2002/008

New enhanced search