2000 BHG: Evaluation of Program Efficacy: UNICEF School-based Psychosocial Program for War Exposed Adolescents as Implemented During the 1999-2000 School Year
Author: Layne,C. M.; Saltzman, W. R.; Burlingame, G. M.; Houston, R. F.; Pynoos, R. S.
This report contains an evaluation of two programs sponsored by UNICEF during the 1999-2000 school year. The first program evaluated, General Psychosocial Intervention, was a trial version of a classroom-based general psychosocial support program, implemented in two secondary schools. The program is implemented by teachers (or other trained personnel) in the classroom and promotes basic "mental health hygiene." Content areas of the program include communication skills, problem-solving skills, friendship-building skills, and learning to control one's emotions by controlling one's thoughts.
The second program evaluated, Specialized Trauma Project, was the UNICEF School-Based Psychosocial Program for War-Exposed Adolescents. This specialized program is implemented in Bosnian schools by specially trained school counselors (psychologists and pedagogues) under the supervision of trained mental health professionals working in the community (clinical psychologists and psychiatrists). The program is designed to identify and therapeutically support Bosnian adolescents with histories of severe war trauma who continue to experience psychosocial problems after the war. A centerpiece of the program is a manualized, 20-session trauma/grief-focused group treatment program in which the school counselors lead a specialized therapy group with 6 to 10 of the most distressed students at the school.
Purpose / Objective
This evaluation of the Specialized Trauma Project is divided into three sections that correspond with three basic evaluative questions. The first section will address the question, "What happened and how does this compare with what was expected?" In this section, the results of the evaluation will be reviewed according to program objectives. The second section will address the question, "Why and how did it happen or not happen?" by reviewing factors that appear to have contributed to the program's effectiveness in some regions, and its lack of effectiveness in other regions. The third section will address the question, "What should be done about it?" by making recommendations that reflect the findings of this evaluation.
For the General Psychosocial Intervention, this evaluation relies solely upon qualitative data. Evaluation data were collected in focus groups held with participating teachers, pedagogues, community mental health professionals, and students during the UCLA Team's May/June 2000 on-site visit.
An evaluation of the Specialized Trauma Project was conducted during its 1999-2000 school year implementation. A total of 17 secondary schools throughout B&H took part in the evaluation. All of the pedagogue teams conducted screening and psycho-educational activities in their schools, formed by 28 school counselors. Six group leader teams conducted one full cycle of the group program, and 9 group leader teams conducted a partial cycle of the group program. Program evaluation data were collected at two points during the 1999-2000 school year (see Appendix C for a summary of evaluation activities). Pre-treatment data were taken from the UNICEF Screening Survey as administered in the classrooms in Fall 1999/Winter 2000. Post-treatment data were taken from multiple sources, including a self-report questionnaire administered to group members; focus groups held with group leaders, group supervisors, and student group members; and a self-report questionnaire completed by group leaders.
Key Findings and Conclusions
General Psychosocial Intervention
An evaluation of the first program revealed generally positive findings (especially given the absence of training and limited support provided for the implementation). Focus groups held with participating students revealed that the students generally enjoyed the materials and access to an informal forum in which they could discuss personal matters. In addition, focus groups held with participating faculty revealed that the teachers also generally enjoyed the materials and access to a forum in which they could step out of their formal roles and engage on a more personal level with their students. Notably, the developmental level of the students appeared to be strongly linked to student's reactions to the program: the younger students appeared to be much more receptive to the program and to consider it relevant. In contrast, although several 3rd and 4th year classrooms gave the program a good review, the majority expressed the opinion that these things were "beyond them."
The teachers felt that the allotted 45-minute period was insufficient to accomplish the session tasks. Aspects of the program that the faculty thought was most useful were the exercises focusing on mutual respect, listening to each other, listening to themselves, developing a vocabulary for emotions, and learning to think through decisions before acting.
Specialized Trauma Project
Although weak in some aspects of its methodology, the evaluation revealed consistently positive results across a broad variety of program dimensions. A strong link was identified between project inputs and tangible outputs. These included:
- Students reported high levels of satisfaction with the program, and strongly endorsed questions about whether the program belonged at their schools and whether they would encourage other distressed students to attend it.
- Students' distress scores (as measured by tests of post-traumatic stress, depression, and grief symptoms) decreased significantly between pre- and post-treatment.
- Decreases in distress scores were significantly associated with increases in measures of positive psychosocial adjustment, as measured by students' reports of their classroom rule compliance, social withdrawal, peer relationships and school interest.
- Students' reports of group processes (including catharsis, cohesion, and insight) were positively associated with measures of positive psychosocial adjustment, as measured by students' reports of their classroom rule compliance and school interest.
- The school counselors reported high levels of satisfaction with training seminars co-led by the UCLA Trauma Psychiatry Team and supervising local mental health professionals.
- Focus groups and self-report questionnaires conducted with the school counselors revealed a program impact that stretched well beyond its original objectives.
- School counselors reported generally high levels of confidence in their abilities to implement the program.
- Nearly all counselors reported that participating in the program had increased their knowledge of the prevalence of war-related trauma among their students and associated distress symptoms.
- Of particular importance to sustainability, the large majority of school counselors reported that they had adapted and imported? the materials into their teaching and professional work at the schools.
- Of interest to both impact and sustainability, the counselors reported that participation in the program had brought about a change/expansion in their roles in the schools from one of "disciplinarians" of disruptive students to providers of mental health services. They reported that they welcomed this change, and identified their participation in the program as instrumental in increasing their expertise and expanding their professional roles.
- The counselors expressed the intention to continue the program during the 2000-2001 school year, and estimated that some 20% to 30% of the students at their schools continue to experience war-related difficulties.
- The supervisory structure of the program (in which the counselors meet with community mental health professionals in group supervision meetings) worked reasonably well. In particular, this structure was helpful in evaluating and referring a highly distressed (suicidal) student to an appropriate source of specialized care.
General Psychosocial Intervention
- School staff from both regions voiced the strong opinion that the program should be continued and expanded, after it has been adapted for local use.
- Although materials have been created for use in the 2nd, 3rd, and 4th years of secondary school, this type of approach appears to have the most appeal for younger students. Thus, the program should probably focus, at least in its initial year, on the first, and perhaps second, grade level of secondary school.
- A significant portion of the program's favorable reception appears to be linked to process features, in which both students and teachers step out of their formal roles and engage each other on a more personal and informal level. This finding suggests that this program may benefit from linking up with active learning-based programs, as implemented by UNICEF and other agencies in B&H.
Specialized Trauma Project
- UNICEF should give serious consideration to continuing the trauma group program beyond the 2000-2001 school year. This observation is particularly relevant in light of (a) the counselors' perception that some 20-30% of students at their schools continue to experience trauma-related difficulties, (b) the large investment made into program development and training, and (c) the relatively small investment required to continue the program (e.g., supporting the regular supervision meetings and photocopying program materials). As part of this consideration process, we recommend that UNICEF sponsor an empirical evaluation of the need for the program's continuation.
- UNICEF should sponsor a systematic effort to incorporate as many useful elements from the trauma group materials into the general psychosocial support program as is possible. These efforts should draw heavily on the experiences of the school counselors and their supervisors who have taken the initiative of adapting the program for use in their regular counseling work and in the classroom.
Full report in PDF
PDF files require Acrobat Reader.