2012 Croatia: External evaluation of the "For Safe and Enabling School Environment"
Author: Andreja Brajsa Zgance, PhD; Toni Babarovic, PhD; Ivana Feric, PhD; Renata Franc, PhD; Jelena Maricic, PhD; Marina Merkas; Goran Milas, PhD; Stanko Rihtar; Ines Sucic, PhD; Marija Sakic, PhD; Lynette Sikic - Micanovic, PhD; Tihana Stojs; Maja Tadic
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."
The project “For Safe and Enabling School Environment” is designed, implemented and managed by the UNICEF Office for Croatia. It was launched in 2003 and it is implemented in partnership with the Croatian Ministry of Science, Education and Sport (MoSES) and Education and Teacher Training Agency (ETTA). The incentive for the project was based on several relevant national and international documents and guidelines.
The project consists of two parts: the public campaign and the school project. The goals of the public campaign were to increase public awareness of peer violence among boys and girls and of its consequences, to reduce tolerance to violent behaviours and to motivate the public to participate in and to support the project. The school project was established to prevent and reduce peer violence in schools, to promote working and living conditions in schools that nourish tolerance, understanding and respect for others, as well as to support schools and local communities to create a protective network for boys and girls. Schools applied for the project on a voluntary basis and a mentor trained by UNICEF was assigned to each school in order to train the teachers and support the implementation of the project in the school.
For the duration of the entire project from 2003 to 2011, 301 (mostly) primary schools implemented the project, 163 schools gained the title of VFS, and 85 schools managed to renew their VFS title after three full years of project implementation.
Objectives of the evaluation were:
1. Review and assess implementation of the project and evaluate role, contribution and impact of every component of the project on the prevention and reduction of violence among children in the school environment and creation of safe and enabling environment.
2. Asses potential modalities and strategies of the project to be replicated and scaled with particular focus on stakeholders’ participation with the view of documenting and disseminating best practices developed through the project and its contribution to the National Plan of Activities for the Rights and Interests of Children 2006 – 2012, National Programme of Activities for Prevention of Violence among Children and Youth adopted in 2004 and National Programme of Activities for Prevention of Violence among Children and Youth adopted in 2009 and the National Protocol on Procedures in Cases of Violence among Children and Youth. This will specifically necessitate an assessment of the sustainability of the project, degree of ownership by concerned governments and actual possibilities existing for the best practices initiated through project “For Safe and Enabling Environment in Schools” to be mainstreamed in the education system and policies.
3. Make recommendations for the project improvement and adjustment for institutionalization process, as well as make recommendations regarding the future of the project (possibilities regarding scaling up, replicating and mainstreaming in the education system and policies within a reasonable timeframe and affordable costs).
In order to address all the objectives and goals of the evaluation, a combination of quantitative and qualitative approaches was used. The study was based on administering outcome measures to schools which implemented the project and control schools which did not implement the project. Thus, post-test only design was used in which project and control schools were compared at a single point in time. Firstly, a sample of 10 project schools with the VFS certificate was selected. Secondly, a control sample of 10 non-project schools was selected. To ensure the comparability of project and control schools, control schools were selected to represent matched pairs to the project schools with respect to several relevant variables.
In order to evaluate the project and its implementation according to the evaluation criteria, four stakeholders in project implementation in schools were surveyed using a questionnaire – boys and girls, teachers, parents and project mentors. Samples of boys and girls, teachers, and parents were drawn from a random sample of 10 schools participating in the project and a control sample of 10 nonproject schools.
Quantitative data gathered through surveys was supplemented with a qualitative analysis, namely focus groups and in-depth interviews with relevant stakeholders, so as to gain a deeper understanding of behaviour, opinions and motives regarding peer violence in schools and a fine-grained and individual overview of the project and its implementation. To this end, two schools participating in the project were selected along with two non-project schools. Separate focus group discussions with boys and girls, teachers, parents and members of the local community were conducted at each school, consisting of 12 focus groups in total. Semi-structured in-depth interviews served to provide a more comprehensive overview of the organizational and management aspects of project implementation in schools.
Findings and Conclusions:
The results of the external evaluation show that the project was not particularly effective in achieving the outcome of reducing the level of peer violence and bullying in schools, since the frequency of peer violence and bullying, pupils’ feeling of safety in school and their tendency to react when witnessing peer violence and bullying do not differ in project and control schools. However, it seems that the project was effective in raising the level of knowledge and self-perceived competence regarding peer violence and bullying among key right-holders and duty-bearers and in improving the social climate in project schools.
The results generally indicate that the project was functional. Teachers mainly implemented the principles of the project in their daily work.
The project had an impact at the level of duty-bearers and stakeholders within schools. It also had an impact on the general public and relevant stakeholders outside of the schools, but this was of a limited scope.
The project is recognized and formally supported by the governmental institutions. It is set in accordance with the relevant national and international documents and policies, and is addressing a relevant issue in the realm of child welfare and protection in Croatia.
The majority of children, teachers and parents from project schools support the continuation of the project in schools and its mainstreaming to the national educational level. The majority of stakeholders in control schools think that a project that reduces peer violence should be introduced in their school and state that they would become engaged in such a project. However, when it comes to defining the roles of the relevant governmental institutions with respect to the projects’ scaling up to the national educational level, it seems that a certain transfer of responsibilities occurs in which MoSES and ETTA see UNICEF as the primary manager of the project.
Recommendations based on the results of the present evaluation focus on three areas: restructuring of project content, improvement of project implementation and increasing its relevance and sustainability.
Restructuring of content and intensification of frequency of workshops for pupils, with greater focus on development and internalization of relevant skills through learning-by-doing paradigm.
Continuous development of new workshops and activities based on good practices in order to satisfy the specific and changing needs of duty-bearers and right holders.
Improvement of project implementation:
Ensuring that parents embrace and apply the project principles in their interactions with children through more active training of parents.
More frequent communication and sharing of information with parents in order to encourage them to become involved in the project.
More frequent communication and sharing of information with members of local community in order to encourage them to become involved in the project.
Increasing relevance and sustainability:
Development of systematic monitoring and evaluation in order to gather information about other similar projects and to modify this project emphasizing its comparative advantages.
Use the bottom-up approach to stimulate the interest of the government and its relevant bodies regarding the ownership of the project.
Strengthening the Network of VFS by expanding its role and obligations in program management and dissemination in order to achieve sustainability.
Based on the results of the evaluation, the current amount of support was obviously not enough. Evaluation showed that teachers from both control and project-schools suggested that harsher punishments for students that behave violently would improve the programme: the popularity of such an amendment exposes the conflict between the different approaches to the education, socialisation, and disciplining of children. These contradictory philosophies are at the root of the difficulties implementing the programme in Croatian schools.
Ownership of the violence prevention programme should be prioritised along with other goals such as lowering violent incidents, having school protocols, adopting a non-blaming response, etc.
The amount of support given to teacher participants in the programme was not sufficient to ensure a long lasting change in the teachers’ processes. Educational authorities need to find ways to motivate teachers to invest efforts to combat children’s violence.
In planning further programme development activities, it would be worthwhile to place additional emphasis on the topic of collaborating with parents, and include it in the training provided to teachers and schools.
Ensuring the sustainability of the programme, and its transfer to education authorities was one of the most difficult tasks highlighted in the evaluation. Besides the reasons such as lack of resources, devotion, and policy the programme signalled a change of school philosophy, policy, and relationships made it seem somewhat too complex.
Full report in PDF
PDF files require Acrobat Reader.