2012 Bosnia and Herzegovina: Joint Evaluation of Social Protection and Inclusion
Author: Paro Chaujar; Zehra Kacapor
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 3' of the report."
The Programme for Enhancing Social Protection and Inclusion Systems for children in Bosnia and Herzegovina (SPIS) is a joint programme of the Government of BiH and UNICEF, financially supported by the European Commission Delegation as part of IPA agreement with the BiH Government, DfID and the Government of Norway. The programme has been increasingly funded more by the EUD than other donors. At the time of this evaluation the EUD is the only donor to the SPIS programme.
This evaluation is jointly commissioned by the Government of BiH and UNICEF to assess outcomes and impacts of the entire SPIS programme at the end of its third/most current phase. The evaluation is also expected to review and assess the framework of social protection and inclusion systems in the country.
There are two broad objectives of the evaluation as per the Terms of Reference:
1) To assess, analyse and evaluate:
a) The elements which form the social protection and inclusion systems (advantages and disadvantages of the current systems) and
b) The services offered through the social protection and inclusion systems (social welfare, education, and health), such as the resources and interventions which are needed to better link these three key social sectors.
2) To assess the effectiveness and contribution of the SPIS programme in achieving its objectives in strengthening the social protection and inclusion systems in Bosnia and Herzegovina, primarily focusing on identifying better strategic entry points for the SPIS programme to penetrate the SPI systems in BiH (which specific services to address; aspect and target of the intervention), through discussions with key stakeholders.
The evaluation is expected to focus on the Integrated Approach to Social Protection and Inclusion systems and how the SPIS programme contributed to overall progress in the sector of social protection and inclusion in the country.
As such there are two parts to this report. The first part is a brief description and assessment of the various components of the ‘systems’ for social protection and inclusion in the country based on existing documents. And the second, larger part presents findings related to effectiveness and contribution of the SPIS programme that has experimented with an integrated approach to social protection and inclusion.
Two sets of terms of reference were used for designing this evaluation:
• Joint Evaluation Of Integrated Approach To Social Protection And Inclusion Systems And Contribution Of The SPIS Programme Unicef Bosnia And Herzegovina. Terms Of Reference & Work Plan (Draft As Of 9 April 2012). This document discusses proposed methodology (UNEG criterion) and scope of evaluation (2009 onwards) as well as provides a set of questions to be answered in the evaluation, under each UNEG criterion.
• UNICEF BIH: Terms Of Reference For International Consultant / Team Leader For Mid-Term Review (MTR) Country Programme Of Cooperation 2010-2014, UNICEF And Bosnia And Herzegovina & Joint Evaluation Of Social Protection And Inclusion (SPI) In Bosnia And Herzegovina. This document provides terms and conditions for the consultants and overview of the purpose of evaluation.
As per the first document listed above, “Joint Evaluation will cover SPIS Programme phases I and II, and will cover the following implementation period: January 2009 – up to date”. The ToRs did not specify implementation period to be covered and so the period suggested in the former document has been adhered to.
In the initial stage of the conceptualisation of this evaluation, two methodologies were identified:
• Desk review and based on desk review, development of an analytical report that identifies lessons learned, adjustments, and recommendations of the SPIS programme
• Consultations with stakeholders, partners and beneficiaries, which will be used along with desk review to develop a final report
By the time the evaluation took off, the terms of reference had been modified and instead of two sets of reports: an analytical and a final, a comprehensive, single report format was agreed upon, with table of contents and main questions to be answered listed.
Findings and Conclusions:
Based on feedback received from respondents, existing literature (studies, assessments and monitoring reports), this evaluation concludes that SPIS has contributed significantly to the cause of promoting an integrated approach to social protection and inclusion for children. Through intensive capacity building of state institutions at local levels and setting up of mechanisms for institutionalizing multi-sectoral collaboration, a good governance model has been demonstrated and replicated within the time frame of the programme. The programme, in order to move from a model approach in selected municipalities now needs to be expanded to all municipalities in the country, albeit through a programme and more efficient approach.
SPIS has certainly emerged as innovative and successful in demonstrating good governance at the local level, characterized by institutionalized mechanisms for multi-sectoral cooperation and piloting of joint action plans.
Under the SPIS programme close cooperation between health, education, and social protection sectors has been strengthened, particularly at the local level. The evidence for this are the Protocols adopted in municipalities for referral and inter-sectoral cooperation as well as the establishment Municipal Management Boards which were later elevated to the status of “Commission for the Social Protection and Inclusion of Children”. This body comprises representatives of different public institutions- schools, health centres, CSW, police, etc. The transition to the Commission implies that this body is now recognised as a permanent executive body at the municipal level.
One of the most critical shortcoming of the project has been its insufficient engagement with broader social protection sector reforms in the country, in particular the ongoing debates and discussions on improving the social assistance/ benefits segment of social protection.
a) Urgently ensure that the roles played by external consultants placed in the various Ministries are included in job descriptions of regular staff so that the focus on SPIS continues after consultants leave.
b) Institutionalise children’s participation in the local development processes
c) Improve coordination and information sharing between the government at all levels and various resource agencies such as NGOs and consulting firms, so they may synergise efforts as well.
d) There is need to make municipal budget allocations more efficient.
e) The focus so far has been on inclusion of children with disability, both in terms of innovative services (IECD) as well as in terms of special focus projects. SPIS model for identification of priorities needs to be made more sensitive and proactive on these other, albeit more politically sensitive factors. Also gender based exclusions need to be better examined and progress in inclusion needs more explicit monitoring along gender as much as other excluding factors.
f) Any revised framework developed for furthering the progress made so far should be designed for impacts at the level of the final beneficiaries (and not just outcomes) and it should have a programme approach:
• Improved policy frameworks and services should translate into better results at the level of final beneficiaries. The programme framework should be impact oriented.
• Results should be monitored and evaluated after actions/ services have been completed and followed up until their lead to ultimate goals. For this there is need to develop baseline and conduct end line surveys.
Some of the lessons learned from the SPIS experience so far are:
a) Despite political differences and different policy/legal frameworks:
• It is possible to operationalize a uniform method for multi-sectoral cooperation at the level of local governance, across the country; and
• Entities work with each other and exchange experiences and good practices towards goals related to children’s rights, to a large extent.
b) Creating and facilitating multi-entity forums for the highest level decision making on programmes is possible and effective
c) Efficiency is a challenge in multi-sectoral and multi entity collaboration at policy level
d) Even though there is a history of disparate frameworks across entities, through collaboration nurtured as part of this programme, the new services that are initiated have higher likelihood of being standardised.
e) Including all levels of governance is critical for success of programmes that require horizontal and vertical linkages
f) There is need to address all sectors (education, health, welfare) equally and address all forms of vulnerabilities (disability, nationality, poverty, etc.)
g) Strengthening systems needs a programme approach, spread over a reasonable period of time and aiming for impacts at the level of final beneficiaries
The ‘SPIS model’ demonstrates good practice models and has demonstrated its application to other sectors within the context of BiH (in joint UN programme DEG as well as in UNICEF supported programme for prevention of juvenile offending). It is certainly a model that can be replicated in different countries and contexts, particularly where multiple layers of governance are involved.
SPIS is not only relevant to the current social policy frameworks in the country but has served to inform and guide emerging frameworks on protection and inclusion.
Full report in PDF
PDF files require Acrobat Reader.