2012 Bosnia and Herzegovina: Evaluation of the Project “Protection of Children at Risk and Children in Contact with the Justice System in Bosnia and Herzegovina”
Author: Vera Devine, Selma Agovic
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."
According to the Terms of Reference (ToR; see Annex I of this report), this report presents the findings of the evaluation of the project “Protection of Children at Risk, and Children in Conflict with the Justice System in BiH”.
The specific objectives of the evaluation, according to the ToR, were to:
1) Evaluate the project results against the planned activities
2) Make strategic and forward-looking recommendations for potential future interventions
The evaluation was also to ”identify lessons learned and make strategic recommen-dations and elements for decision-making in the future, informing the design of the project as well as the government reforms in the field of justice for boys and girls. The evaluation [should] assess progress against the project results at the State, entity and municipal level (in the 9 selected project locations).”
During an extensive briefing with UNICEF at the onset of the in-country mission (3 July 2012), as well as during separate briefings with the two donor agencies (Sida and SDC), it was agreed that the emphasis of the assignment should be on the second ob-jective, i.e. the making of strategic recommendations to guide future project interven-tions. Therefore, this report provides recommendations for future efforts in relation to each specific output. The project under evaluation will officially end in December 2012, with a no-cost extension for a further 6 months currently being discussed. Stra-tegic recommendations, then, are to inform both the remainder of the current project, and the design of potential follow-up projects.
The evaluation was carried out from 20 June to 31 July 2012, by a team of two evaluators, supported by Indevelop’s evaluation director and manager. The assign-ment is a conflation of what was initially foreseen as two separate exercises - a mid-term evaluation (originally scheduled for spring 2011) and a pre-end evaluation, to take place 6-7 months prior to the project end. The two evaluators were financed by Sida and SDC, respectively; The field phase took place from 3 July to 12 July 2012. A detailed schedule of meetings is attached as an Annex to this re-port. To maximise coverage, the evaluators split up for the field visits: one of the evaluators visited 3 HRO municipalities (Zenica, Tuzla, Bijelina), the other Trebinje, which is a municipality covered by ZDS.
The evaluators had a series of meetings with the UNICEF Country Office BiH, both in preparation of the meetings in the field, as well as in order to seek clarifications after meetings with stakeholders. Early on in the process, the differences between an evaluation and an audit were discussed and clarified. The evaluation looked at budg-etary issues only insofar as these helped explain the design and delivery of the pro-ject, as well as to inform overall assessments of project efficiency. The evaluators had a telephone conference call on 30 July 2012 discussing UNICEF comments on the first draft report. A further round of feedback and changes to the report were made on 13 August 2012.
The evaluators had agreed at the onset of the field visit that any problematic or criti-cal issues would be flagged to counterparts during the meetings in order to avoid un-pleasant surprises; specifically, one of the implementing NGOs, ZDS, has been sig-nalled that their project reporting skills were an issue of concern that would be high-lighted in the report.
Findings and Conclusions:
Against the DAC criteria, the evaluation concludes the following:
Design: The project worked on four inter-related objectives, adopting an integrated, multi-disciplinary approach of work with duty-bearers and rights-holders, at state and entity-levels, as well as at the level of nine selected municipalities. Activities also targeted institutions for children in conflict with the law, as well as raising the general level of awareness on juvenile justice (JJ) issues among the population of BiH. The project was accompanied by advocacy work by UNICEF, in close partnership with Sida and SDC, with key national decision-makers and the international community in BiH. There is continued need to work on all four components, with alternative meas-ures featuring highest in the perceived needs of stakeholders and institutions.
Relevance: The project has been highly relevant for the context of BiH; and the choice of objectives, results and outputs, as well as implementing modalities reflect this. The project’s objectives and outcomes are consistent with and supportive of government policies and sectoral priorities, and have shaped the EU accession agenda; these objectives are to a large extent still valid for BiH. The relevance of the project is also reflected in the fact that it has demonstrated flexibility by adapting its approach to emerging needs as they emerged for partners and stakeholders.
Effectiveness: Overall effectiveness has been affected by delays on some project components. In part, this is a function of the highly complex and unstable political environment characterising BiH, which strongly affects policy decisions regardless of the actual issue at hand (in the case of this project, the state-level Juvenile Justice Strategy 2010-2014), and which is clearly outside of the control of UNICEF. [...]
[Read more in full report]
We make two sets of recommendations, 1) for the ongoing project to consider until its (likely) conclusion in June 2013 and 2) for a potential follow-up project beyond that time.
For the remainder of the project:
• Monitor that the child-friendly rooms at police stations equipped through the pro-ject are used, and consider organising specific training for police representatives to increase confidence in using it.
• Discuss with the implementing NGOs Zdravo da Ste (ZDS) and HRO the exit strategies/conclusions of activities in the 9 municipalities by June 2013. As the municipalities are not at the same level of progress with the municipal-level pre-vention Action Plans, different scenarios are likely to apply; discussions should also focus on municipalities that could be involved in a follow-up project.
• Assess the results of the prevention of the School Safety Networks, a model that, despite having been widely used, including in the evaluated project, has not been subject to systematic capturing of impact and lessons learned.
• Continue advocacy work in partnership with SDC, Sida, and other international organisations with the relevant authorities on the need for a state-level approach to JJ, either as a stand-alone strategy, or as part of an overall justice sector reform strategy.
• Continue advocacy work to hold the FBiH authorities accountable to the com-mitments made to improve the conditions in juvenile detention centres, and to rigorously support plans for the building of a new centre in Orašje.
• Conduct extensive stakeholder consultations about their needs in a follow-up phase spanning a period from 3 – 6 years from June 2013.
• Critically assess the usefulness of quantitative and qualitative indicators employed by the existing project to assess progress against objectives; collect, build up, or fill in lacking baseline data to use in future interventions, and to determine more specifically (as opposed to having rightly identified the general needs) in the spe-cific parts of the system (Centres for Social Welfare; Centres for Mental Health; police; judiciary/prosecution; institutions etc.), and using existing guidance on JJ indicators (such as the UNODC/UNICEF Manual for the Measurement of Juve-nile Justice Indicators); put in place a mechanism to monitor impact and sustain-ability after engagement with the current municipalities finishes.[...]
[Read more in full report]
Full report in PDF
PDF files require Acrobat Reader.