2016 Global: Evaluation of UNICEF’s humanitarian response to the Syria crisis
Author: Hetty van Doorn, Brian Majewski, Ian Heigh
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 4’ of the report, and the executive feedback labelled as 'Part 5'.
Starting from non-violent protests in February 2011, the crisis in the Syrian Arab Republic accelerated into an all-out conflict inflicting untold suffering and hardship on civilian populations. By the end of 2015, an estimated 13.5 million people, including 6 million children, were in need of humanitarian assistance. Of the people affected, some 7.6 million were within Syria - in 2015 alone, more than 1.4 million people were displaced inside the country, many for the second or third time. An additional 4.6 million Syrian refugees were registered in neighbouring countries, with approximately 3.5 million hosted in Jordan, Lebanon and Turkey.
Conservative figures estimate that more than 191,000 people have lost their lives in the conflict. With at least 10,000 children killed in the Syrian Arab Republic since 2011, child casualty rates are the highest recorded in any recent conflict in the region.
The regional neighbours have made tremendous efforts to accept the flood of refugees – but the dimension of the crisis is placing these countries – and vulnerable host communities – under stress, that their situation may become politically and socially unsustainable. The influx has pushed up demand for already scarce supplies and resources, such as increased competition for livelihoods and access to basic social services. The political challenges faced locally have changed and evolved, while the security situation is under constant flux. This environment is impacting the implementation challenges and conditions faced by humanitarian organizations assisting the local population.
As the crisis began to deepen, UNICEF joined the international effort to reach affected populations with humanitarian assistance by mounting complex, large-scale operations across the sub-region. Six UNICEF country offices (COs) supported the delivery of programmes designed to meet the needs of crisis-affected children inside the Syrian Arab Republic and in neighbouring countries.
The evaluation is intended to serve both an accountability function (historical/summative) and a learning function (forward-looking/formative). The scale and funding for the crisis necessitates an accountability function; the fact that the crisis is becoming a protracted emergency necessitates the learning function. Equal weight is attached to both. The evaluation aims to support further strengthening of UNICEF’s performance in protecting children’s rights and well-being in the region and in responding to large-scale multi-country emergencies.
The purpose of the evaluation is to provide a comprehensive assessment of UNICEF’s overall response to the Syria crisis against its own mandate and standards, its stated objectives, and standard evaluation criteria. The evaluation, based on collation and analysis of relevant data and information, generates evidence, conclusions and key lessons and recommendations concerning UNICEF’s future humanitarian responses both in the sub-region and elsewhere.
The main objective is to provide an independent and robust evaluation of UNICEF’s emergency response under three main headings:
- UNICEF’s strategy and key programme interventions, programme choices and related operations, including attributable results.
- UNICEF’s engagement with other actors, with a primary focus on its role in sector coordination where relevant; and a secondary reflection on its collaborations with key stakeholders, including governments, other United Nations agencies, beneficiaries and implementing partners.
- UNICEF’s management structures and operational processes, including its L2 and L3-related procedures, in relation to its Syria crisis response and performance.
The UNICEF Evaluation Office formulated the purpose, objectives, questions, outputs and scope of the evaluation based on a six-month scoping, research and consultation exercise. The evaluation was undertaken between March-November 2015 by a specialist consultancy firm, and was independent of the Syria response.
The EO managed the evaluation and the Reference Group for the evaluation guided the process, with the overall guidance from the Evaluation Office Director.
The UNICEF response was evaluated using a data analysis framework comprising five focus areas identified by UNICEF senior managers as the main operational outcomes, to assess:
- Effectiveness: How well did UNICEF deliver assistance?
- Relevance: Was UNICEF's response appropriate for the environment and needs of the affected population, over time?
- Coverage: How well was UNICEF able to scale up and meet the assessed needs?
- Efficiency: How efficient was the response (speed, cost and quality)?
- Coherence: Have UNICEF’s humanitarian guidance tools been used and of use in the context of the Syria crisis?
Guidance from OECD/DAC and the ALNAP was applied to identify criteria and design indicators against which the evaluation has been carried out. These criteria are aligned with the focus areas above.
The focus areas informed the design of the data collection tools and primary data were collected from four main stakeholder groups: (1) UNICEF staff (COs, RO and HQ), (2) implementing partners, (3) coordinating partners and (4) the affected population.
Secondary data came from an extensive literature review conducted during the evaluation period and supporting data came from a web-based survey. All data inputs were triangulated during the analysis. Four analysis steps were used to produce the evaluation findings, and outcomes were cross-referenced with supporting data and reviewed with relevant technical, field operations and managerial UNICEF staff members and the Evaluation Reference Group.
Findings and Conclusions:
Key focus areas for child protection across the countries were emergency psychosocial support; case management and advocacy, develop policies, influence laws and establish SOPs; and support the Government in undertaking its overall leadership role and child protection coordination.
As part of its education mandate, UNICEF advocated for, and facilitated school enrolment and certification in both camp settings and host communities. UNICEF also supported the establishment of non-formal education services in camps, host communities and out-of-camp settlements and scaled up support to adolescent programming with a focus on vocational training and life skills.
After an initial focus on measles vaccination, polio vaccination was prioritized within health programming across the region, following the outbreak of poliomyelitis in the Syrian Arab Republic in October 2013, with a focus on vaccine procurement and logistics, cold chain and communications.
The WASH programme was regionally prioritized and delivered assistance in close collaboration with partners. Different country offices prioritized different activities, broadly water supply (Jordan and Syria), wastewater management (Jordan) and infrastructure repair and rehabilitation (Jordan, Lebanon and Syria).
Cross-sector initiatives such as the No Lost Generation initiative, a regional-level strategic framework and the Makani/My Space initiative, a country-level integrated programme in Jordan, were also developed.
Despite the described difficulty of the operating environment and the limitations of its approach, UNICEF was substantially able to deliver on its core objectives. The response was slow to start (2012), but the evaluation found evidence that the organization invested significantly in implementing its programmes, incrementally built its capacity and improved performance through 2013 and 2014, with significant scale-up and reach of programming achieved from 2014.
The recommendations were formulated to:
- improve UNICEF’s response in the Syrian Arab Republic and the sub-region; and
- improve UNICEF’s global humanitarian response activities for the future, based on lessons learned from the Syria crisis
Recommendation 1: Develop an overarching sub-regional UNICEF strategy, based on comprehensive needs assessment and situation analysis (including risk analysis and conflict analysis) aimed at strengthening the coherence and consistency of the overall response, and linked to UNICEF’s global priorities and responsibilities. This should include, for each country office, a long-term, country specific approach.
Recommendation 2: Clarify the future roles and accountabilities of Headquarters, MENARO/Syria Hub and country offices, including lines of communication, and provide appropriate guidance through updated standard operating procedures.
Recommendation 3: Optimize the selection and management of implementing partners.
Recommendation 4: Develop a systematic approach to information sharing, feedback and accountability mechanisms for the affected population and integrate these into country plans, programme proposals and monitoring and evaluation processes.
Recommendation 5: Develop UNICEF-specific guidance for measuring the efficiency of programming and operational support that is contextualized for the crisis.
Recommendation 6: At the global level, taking the lessons of the Syria crisis into account, develop key guidance, tools and the knowledge base needed to carry out humanitarian response activities in similar contexts (i.e. complex, multi-country, protracted emergencies, driven by conflict, featuring urban and camp settings and large-scale population displacement). Follow through to make such guidance widely available, accessible, known and understood.
- If further assessment within sector’s are needed, suggest to have separate sector wide evaluations, specifically for the flagship programmes such as WASH, Education, Child Protection and Health rather than one large evaluation. Subsequently, the sector evaluations, can be synthesized within a meta-evaluation. This approach will minimize certain constraints faced during the implementation of this evaluation and a sound risk mitigation strategy.
- For large protracted emergencies, it is important that UNICEF has dedicated resources to produce result level data (preferably Outcome level or above) prior to the start of an evaluation. Presently most data is focused on population coverage, with few data on quality of response.
- Need to strengthen the Accountability to Affected Populations (AAP) dimension of UNICEF’s work and ensure that proper feedback mechanisms are put in place. The mechanism should ensure feedback is used to improve programme delivery.
- Before an evaluation is commissioned, financial data, quality of service data and other relevant information on efficiency per outcome needs to be available. The evaluation team will then be able to put these data together to assess the efficiency of the response.
Note: You will find the report below labeled as follows:
- REPORT - Volume 1: Main Report
- PART 2 - Volume 2: Appendices
- PART 3 - Executive Summary in Arabic
- PART 4 - GEROS 2016
- PART 5 - GEROS 2016 Executive Feedback Summary
- PART 6 - Evaluation Brief
Other Evaluation Resources on the Syrian Crisis.
Full report in PDF
PDF files require Acrobat Reader.