2016 Global: Evaluation of the UNICEF Response to the Crisis in the Central African Republic
Author: Andrew Lawday, Ian Clifton Everest, Soledad Posada, Ana Rodriguez and Colleen McMillon
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as 'Part 4'.
The Evaluation of UNICEF’s response and programme strategies to the humanitarian crisis in CAR follows the declaration of Level 3 corporate emergency procedure in Dec 2013, after the situation developed from a silent emergency into a more visible and complex humanitarian and protection crisis. An estimated 2.5 million people were in need of assistance. There was a lack of basic services, and insecurity resulted in over 500,000 internally displaced, and 420,000 seeking refuge in neighbouring countries. Children in particular have borne the brunt of the crisis, with many being displaced, separated, maimed, mutilated, abducted, killed, raped, and recruited into armed groups.
In response, UNICEF strengthened its presence in CAR and its capacity to accelerate the delivery of humanitarian assistance, focusing on life-saving interventions to address vaccine-preventable and water-borne diseases, malaria and malnutrition, and reducing risks faced by displaced populations. UNICEF and its partners provided psycho-social assistance to children affected by armed conflict; safe learning spaces, and work on the identification, release and community-based reintegration of children associated with armed groups.
The evaluation was undertaken to independently evaluate UNICEF’s response and programme strategies to the humanitarian and protection crisis in CAR. The findings indicate that despite the many needs and challenges, UNICEF and its partners were effective in delivering programmes for children and programmes were generally satisfactory in achieving results. However, the evaluation also reveal that there are still major challenges and demands that needs to be addressed to save lives, protect the rights of the most vulnerable, and ensure their survival and livelihood.
The evaluation also provides some recommendations for going forward for more effective actions by UNICEF and its partners towards enhancing equitable programme delivery for women and children of CAR.
This is an evaluation of UNICEF’s response and programme strategies in the Central African Republic from July 2013 to December 2014. Its purpose is to inform UNICEF’s programming in the country, provide impartial evidence and offer practical recommendations. The evaluation follows the declaration of the L3 emergency procedure for CAR in December 2013. It serves the primary purposes of:
- a formative purpose, to 'Inform UNICEF’s programming (in CAR) for 2015/2016 as well as the L3 transition strategy,'
- a summative purpose, to 'Provide impartial evidence and generate information on how UNICEF has responded to the crisis since July 2013 to December 2014,' and
- an advisory purpose, to 'Examine the application of the L3 procedures in supporting the response, including the role of Headquarters (HQ) and the Regional Office (RO), and provide practical and actionable recommendations on the L3 transition strategy.' It should be noted that actually the evaluation period was extended into 2015, due to various reasons.
Three user-focused objectives were established at the inception phase:
- A strategic objective: To provide well-founded recommendations to the Representative Country Office (Rep/CO) and the General Emergency Coordinator (GEC)/RO for developing the CAR response, programme strategies and components in 2015;
- An accountability objective: To provide an impartial assessment of the CAR response, programme strategies and components for UNICEF’s response partners and oversight function; and
- A learning objective: to explain lessons learned, strengths, and areas for improvement in the CAR response and to enable knowledge managers to understand effectiveness and main factors.
The evaluation was conducted in three stages in 2015. At the inception stage, the evaluation team refined the design, engaged users, and defined stakeholders; conducted a document review; and carried out global-level interviews. At the field research stage, the evaluation team conducted structured consultations in CAR with CO Managers, implementing partners, strategic partners, and intended beneficiaries. In the analysis and reporting stage, the team conducted two levels of analysis, drafted the report according to guidelines, invited reflection and validation from users, and finalized the report.
A stakeholder analysis was conducted at the inception phase, to inform a structured consultation of stakeholders. It identified three categories of primary stakeholders in the UNICEF response - Implementing partners; strategic partners, including government and humanitarian actors; and beneficiaries, including children, and representatives of the affected population.
Consultations with UNICEF and strategic partners were conducted through managers and nominated representatives; those for implementing partners and beneficiaries relied on purposive sampling. 15 implementing partners were selected according to budget size and across seven programmes. Sixty beneficiaries were consulted in three provincial cities, where the evaluation team spoke to a range of individuals at the community level, including displaced and hosts; Muslims and Christians; and children and young people.
120 stakeholders were consulted, either using in-depth interviews or in focus groups or in participatory classroom discussions. Stakeholders in CAR were consulted through face-to-face interviews, and international stakeholders through Skype and telephone. The primary approach to stakeholder consultation was therefore qualitative. In addition, an opinion polling exercise was conducted among the institutional respondents, to provide a quantitative aspect and inform evaluative assessments.
Findings and Conclusions:
Relevance/Appropriateness: UNICEF’s response and programme strategies were relevant and appropriate. UNICEF’s plans were well aligned with United Nations plans and the CCCs. The organization reacted to major constraints, its advocacy contributed to protection and its strategy was relevant to some rights of the child.
Coherence: UNICEF response and programme strategies were generally coherent but not integrated. The response committed to an integrated approach, although it lacked guidance and the expertise to implement it, though notable synergies and complementarities were noted.
Coverage: UNICEF’s coverage of needs was good and large in scale especially in Bangui, for IDP sites and easy-to-reach areas. UNICEF and partners increased coverage as it scaled up operations, and achieved wide coverage.
Connectedness: UNICEF’s response strategy was insufficiently connected with longer-term development goals. UNICEF supported some development in practice (especially in health and education), even while it remained focused on the emergency response.
Effectiveness: UNICEF was effective in delivering programmes in CAR. Programmes were effective in achieving results, albeit with some variation across sectors.
Efficiency: While UNICEF was slow to respond to needs outside of Bangui in 2013, it acted quickly to scale up its operations in 2014, and expanded coverage to support a large number of people in need across its core sectors. However, it is difficult to measure how efficiently the programmes were delivered, as UNICEF does not collect all necessary information that allows for full efficiency assessment.
Coordination: UNICEF’s response and programmes were coordinated with other actors, and its wide-ranging agenda highly valued. UNICEF carried the most weight within the cluster system, and ensured that all the clusters it led had a dedicated cluster coordinator.
UNICEF should update risk analyses and put in place preparedness arrangements in all COs facing chronic and complex emergencies. UNICEF should: i) reinforce response capacity at declaration of L2 status; ii) develop preparedness plans that include RM, HR, leadership, partnerships, supplies & logistics, procurement, and M&E; iii) conduct early and regularly updated situation/strategic risk analysis to respond to the chronic and complex emergency, and prepare for further acute phases.
UNICEF should review its partnership model and operations function, and consider: i) reviewing the comparative advantage of UNICEF’s partnership model in emergencies where government and INGO partners are not sufficiently available to implement programmes; ii) developing a protocol for direct delivery iii) revising management systems to ensure the entire operations function supports achievement of programme results in an emergency.
UNICEF should develop, long-term strategic response to the CAR’s chronic and complex emergency, and apply such an approach in similar situations. UNICEF HQ level should provide guidance and support on the development of strategic plans to address the aspects of development, recovery/resilience, and rights advocacy in chronic and complex emergencies, especially in cases where government’s capacities are low.
UNICEF CAR should consider: i) developing a comprehensive protection strategy to address the protection and human rights crisis, informed by assessment of rights; ii) developing an advocacy strategy to support it; iii) conducting comprehensive needs assessments at household level; and iv) plan or advocate for major investments to support social cohesion and promote a culture of peace and tolerance.
UNICEF should develop an M&E and learning framework and system to support strategic decision-making, accountability to stakeholders and learning for improvement. This is essential for strengthening strategic management of the response.
- If further assessment within sector’s are needed, it would be useful to have separate sector wide evaluations, specifically for the flagship programmes, rather than one large evaluation. Subsequently, the sector evaluations, can be synthesized within a meta-evaluation. This approach would ensure more evidence is generated to inform programming.
- For large protracted emergencies, it is important that UNICEF has dedicated resources to produce result level data (preferably Outcome level or above) prior to the start of an evaluation. Presently most data is focused on population coverage, with few data on quality of response.
- Need to strengthen the Accountability to Affected Populations (AAP) dimension of UNICEF’s work and ensure that proper feedback mechanisms are put in place. The mechanism should ensure feedback is used to improve programme delivery.
- Before an evaluation is commissioned, financial data, quality of service data and other relevant information on efficiency per outcome needs to be available. The evaluation team will then be able to put these data together to assess the efficiency of the response.
- M&E and Learning was found to be quite weak - therefore, in protracted emergencies, it is important to invest in M&E, including ensuring clarity of what data is needed to support strategic decision-making, learning, and accountability; and developing systems for data collection, analysis, and reporting. This should include innovative methods for collecting data, consulting samples of beneficiaries, integrating and developing third party monitoring, enhancing field monitoring, reviews, and incorporating evaluations, and ensuring that all the evidence collected is used to improve programme delivery.
Note: You will find the report below labeled as follows:
- REPORT - CAR report [English]
- PART 2 - CAR report [French]
- PART 3 - GEROS 2016
- PART 4 - GEROS Executive Feedback Summary
- PART 5 - CAR Brief [English]
- PART 6 - CAR Brief [French]
Full report in PDF
PDF files require Acrobat Reader.