We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2017 EO: TOWARDS IMPROVED EMERGENCY RESPONSES: Synthesis of UNICEF Evaluations of Humanitarian Action 2010 – 2016



Author: Julia Betts, Volker Huls

Executive summary

Background:

The period 2010-2017 has seen unprecedented change in the humanitarian landscape. The advent of several concurrent major emergencies, along with their increasing complexity and protracted nature, has left vastly increased numbers of children in need of humanitarian assistance. From 66 million per year in the 1990s, in 2016, an estimated 535 million children – nearly a quarter of the world’s children – lived in countries affected by armed conflict, violence, disaster and chronic crises. The global humanitarian system has undergone significant shifts. The international response to the 2010 Haiti earthquake exposed some serious fault lines, with the response criticized for being late, slow and uncoordinated. Post-Haiti, the Transformative Agenda, initiated in 2011, sought stronger leadership, greater coordination and increased accountability for humanitarian action. The World Humanitarian Summit of 2016 brought actors together around common dilemmas, culminating in the Grand Bargain on humanitarian financing and other key agreements.

UNICEF has played a critical role in global humanitarian reforms. Concurrently, its own engagement in humanitarian response has grown rapidly. Its expenditure on humanitarian activities – across the spectrum of preparedness, response and building resilience – increased from just over $900 million in 2012 to $2.1 billion in 2015. The report synthesizes the findings of 30 evaluations of UNICEF’s humanitarian action, and asks three questions:

  • How has UNICEF’s humanitarian action from 2010 to 2015 performed, and how has it improved over time?
  • What factors have supported or constrained improvement?
  • What can be learned, and what improvements made for the future?

It builds on and updates a similar exercise conducted in 2013. Its overarching aims are to support accountability, contribute to learning, and help UNICEF realize its humanitarian objectives for the vulnerable children it serves.

Purpose/Objective:

The overall aim of this synthesis is to determine factors supporting or constraining relevant, effective, efficient, connected, coherent, and well-coordinated humanitarian action, drawing on evaluation reports and other evaluative documents within the context of UNICEF’s humanitarian work; and identifying lessons learned and applied by UNICEF.  The synthesis is intended to foster discussions on major issues emerging from the analysis, and to provide a summary of evaluation evidence concerning UNICEF’s recent humanitarian action, and related policies, systems and practice.  The synthesis also highlights success factors and constraints in programming in these contexts, identifying gaps in the evaluative work, and suggesting areas for further enquiry.   

The synthesis provides: 

  • An overview of UNICEF’s EHA reports and related materials at all levels of the Organization, including their number, focus, coverage, themes, quality; level of investment in EHA compared to appropriate benchmarks, and compared to other humanitarian organizations and UNICEF’s own risk profile. 
  • A gap analysis of where further inquiry is required, identifying areas/sectors with little or no EHA data concerning the impact of UNICEF’s programming for children and women as per its core mandate, or of its fulfilment of its role as Cluster Lead Agency. 
  • A contextual overview providing an indication of key events and trends relevant to the analysis, both within UNICEF and externally;
  • A synthesis of recurrent findings, issues, themes on how UNICEF has performed against its own corporate commitments, standards and stated objectives (in terms of the modified DAC/ALNAP criteria for EHA); identifying factors that contributed to success and those that constrained UNICEF’s efforts; the key lessons that UNICEF learned about its humanitarian action through the evaluations; and a few recommendations on enhancing UNICEF’s humanitarian action.  

Methodology:

This Synthesis has been developed from a selection of 30 evaluations, distilled from the wider pool of 76 evaluations. Criteria for inclusion were:

  • First, only documents with a strongly evaluative approach were included. This excluded, e.g., reviews, research reports and other material such as Lessons Learned documents.
  • Second, only reports receiving a ‘satisfactory’ rating in the UNICEF quality assurance mechanism for evaluations were included, to ensure that evidence was sufficiently valid and reliable.

Two other forms of evidence provided triangulation:

  • Inter-Agency Humanitarian Evaluations (IAHEs), provided evidence of the system-wide response but do not report on UNICEF performance specifically.
  • The 2017 report Learning from Humanitarian Action: a synthesis of non-evaluative documents on UNICEF’s humanitarian action from 2010-2016, which brought together evidence from reviews and other relevant sources.

The 30 evaluations cover all major emergencies to which UNICEF has responded since 2010. The 30 evaluations were systematically reviewed, applying an analytical framework to ensure consistent extraction of key findings.  This included the evaluation criteria of relevance, effectiveness, efficiency, coherence and co-ordination plus other lines of analytical fields of interest. 

A second-layer quality assessment for individual data pieces was then applied. Evidence was rated for validity and reliability on a scale of 1 (low) to 4 (high), with only reliable evidence – scoring at least 2 – included. This approach allowed the strength of evidence underlying each finding to be made explicit.

The availability (density) of evidence against individual themes overall covered within evaluations was organized as follows:  weak: 1-9 evaluations out of 30; moderate: 10-19 evaluations out of 30; strong: 20 or more evaluations out of 30). 

Findings and Conclusions:

The Synthesis reflect an organization that has evolved considerably since the Haiti 2010 earthquake.  New procedures have been implemented, new ways of working developed, and learning generated and shared. Reforms to the wider humanitarian system, are reflected in improvements to corporate and operational practice. 

Programming also aligns strongly with national responses, priorities and plans. UNICEF participates in joint responses to emergencies, and prioritizing partnerships – though its connections with government or national authorities are stronger than those with its partner UN agencies. 

There are some key results for children facing conflict and crisis. UNICEF has contributed to reduced transmission of disease; helped prevent hunger and under-nutrition; and provided clean water and education to many vulnerable children. It has protected children in high threat environments and built the capacity of local and national actors in humanitarian situations.

Consistent weaknesses identified include: 

  • Needs assessments for affected populations – even under accessible conditions – are sometimes incomplete or too general. Consequently, opportunity-based, rather than needs-based programming, persists.
  • Strategies and programme designs are sometimes weak, leading to a reactive rather than a proactive approach.  

UNICEF is still working to build clear links from humanitarian to development responses. The application of the Level 2 and Level 3 SSOPs has had a major effect in supporting timely responses. 

There is evidence of a more risk-willing approach, an openness to innovation, and a willingness to experiment. However, new procedures available to short-cut administrative burdens are not always applied and used by responsible staff. 

Finally, UNICEF has not consistently adopted a proactive approach. Issues such as preparedness, transition planning and AAP have not always kept pace with global shifts. 

Recommendations:

  1. UNICEF has gathered a considerable body of evidence on its humanitarian action (76 evaluations since 2010). Its Evaluation Policy states that evaluations of humanitarian action will ‘usually be undertaken’. Yet despite a set of corporate triggers, coverage remains unsystematic and patchy – particularly of Level 1 emergencies.
  2. Evaluations found that UNICEF’s humanitarian action was often insufficiently grounded in needs assessments, even where these were feasible. Programme designs require clearer links to needs.
  3. Given its highly decentralized nature, guidance and procedures issued ‘from the centre’ are only ever as influential as UNICEF’s country management and staff habits permit them to be. New protocols and procedures, such as the Level 2 and 3 SSOPs, need to be accompanied by capacity development and training to build a ‘risk-willing’ approach.
  4. UNICEF’s decentralized structure means that it benefits from a vast cadre of national staff and partners, which provide it with a core capability to prepare for humanitarian action from a localized viewpoint. Under Grand Bargain commitments, preparedness and risk identification should be approached from this perspective.
  5. The CCCs in their current formulation do not reflect the changing nature of humanitarian crises, and promote siloed rather than integrated responses.
  6. Performance monitoring of humanitarian action is a consistent challenge, yet the evaluations analyzed found considerable scope to improve UNICEF’s monitoring of its own performance in emergencies, in line with recent internal efforts to strengthen Humanitarian Performance Monitoring.
  7. A more explicit and defined strategic overview within UNICEF’s humanitarian action is needed, which is firmly geared to resilience and transition goals. This should be linked to the revisited CCCs.

Lessons Learned:

Needs assessments are the foundation of effective humanitarian action – understanding needs is not just a fundamental part of humanitarian action but the bedrock on which effectiveness and efficiency relies. Circumstances may constrain access, and inter-agency processes may not deliver all that they intend. Nonetheless, humanitarian responses must be firmly grounded in an understanding of the needs they aim to address. Engaging with affected populations is a precondition of addressing equity concerns; it cannot be shortcut, or bypassed.

There is no substitute for strategy – several of the experiences analysed show reactive, rather than proactive responses; a tendency to plan piecemeal and in silos; and a lack of coherence across countries in a regional response. Yet UNICEF has the capacity and the learning available to enable it to plan more strategically. Not doing so is a missed opportunity for effectiveness. 

Focus on the future – many of UNICEF’s changed procedures and protocols have emerged as a response to a particular experience. Documenting and learning from experience is important; but equally important is the kind of future focus and horizon scanning that enables systems and protocols to be developed, which will help identify and manage future risks.

Learning needs direction – whilst much learning can be generated and some even applied, the systematic application of documented experience into changed corporate procedures is challenging. Transforming learning from the ‘here and now’ into corporate shifts requires systemic change and staff being fully on board.  

Systems-building is a process – despite  the major effort that has been dedicated to improving systems for emergency response – reflected in the Level 2 and Level 3 SSOPs – the evidence examined here shows the limited authority and traction of such systems when unaccompanied by a culture of confidence in their use.  

 

Please find the following items attached below, labelled as follows:


  • Evaluation report - Report
  • Summary - Part 2
  • Evaluation Management Response (EMR) - Part 3
  • Updated EMR, as of 14 January 2019


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2017

Country/Office:
Evaluation Office

Region:
Global/Headquarters

Type:
Review

Theme:
Emergencies

Language:
English

Sequence #:
2017/009


New enhanced search