We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Global: Report of the Inter-agency Humanitarian Evaluation (IAHE) of the Response to the Crisis in South Sudan



Author: Nigel Clarke, Jeremy Loveless, Boniface Ojok, Sarah Routley and Tony Vaux

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation  reports. The quality rating scale for evaluation reports is as follows:  “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as 'Part 3'.

Background:

An Inter-Agency Humanitarian Evaluation (IAHE) was conducted in South Sudan in April 2015 under the auspices of the Inter-Agency Standing Committee (IASC) Transformative Agenda (TA) for the global reform of humanitarian action in large-scale emergencies. This was the first such evaluation of a response to a conflict situation and complex political emergency. It was triggered by the declaration of an L3 emergency on 11 February 2014. The purpose was to provide an independent assessment of the extent to which planned and relevant collective objectives set in the SRP were met, and the extent to which response mechanisms, including the HPC and other key pillars of the TA, supported the response, and recommend improvement-orientated-actions.


Widespread internal conflict in South Sudan began between rival government factions in December 2013, and escalated rapidly as ethnic dimensions emerged and as the fighting spread to different parts of the country. More than 1.5 million were displaced, both internally, and externally to neighbouring countries. More than 80,000 people had sought refuge in UN bases and were being protected by UN Mission in South Sudan (UNMISS). Over 3 million people were considered to be in the highest category of food insecurity.

South Sudan gained independence from the Sudan in 2011 following decades of civil war that cost one million lives and halted national development progress. Although South Sudan had been taking tentative steps towards stabilization and recovery prior to the onset of the current crisis, development indicators were among the worst in the world and in several states the proportion of children affected by acute malnutrition had reached emergency thresholds. UNMISS remained in place due to sporadic local conflicts and remaining tensions with the Sudan. Major aid operations, which were ongoing before the current crisis began in December 2013, were primarily working towards stabilization, resilience and recovery.

Purpose/Objective:

IAHEs are part of the HPC, an element of the Transformative Agenda, an initiative taken by the IASC in 2011 to improve inter-agency humanitarian action. The IAHE is a critical examination of results designed to inform management decisions. The process is “intended to promote collective accountability and system-wide strategic learning” through “an external, independent assessment of a collective humanitarian response”.

The HC and the HCT are the primary users of IAHEs, which provide information about progress made towards objectives, as well as relevant information for planning and decision-making. IAHEs are expected to promote accountability and lesson learning across the entire humanitarian system.

The IAHE in South Sudan, which was triggered by the declaration of an L3 emergency on 11 February 2014, has the following two purposes:

Purpose 1: To provide an independent assessment of the extent to which planned and relevant collective objectives set in the SRP were met.
Purpose 2: To assess the extent to which response mechanisms, including the HPC and other key pillars of the Transformative Agenda, supported the response, and to recommend improvement-orientated actions.

These purposes are achieved by addressing the four evaluation questions:

  1. To what extent are the SRP objectives appropriate and relevant to meeting the humanitarian needs? Have systems been established to measure their achievement? To what extent have the 2014 results been achieved, and what were both the positive and potentially negative outcomes for people affected by the disaster? 
  2. To what extent have national and local stakeholders been involved and their capacities strengthened through the response?
  3. Was the assistance well coordinated, and did it successfully avoid duplication and fill gaps?  What contextual factors help explain the results or lack thereof?
  4. To what extent were IASC core humanitarian programming principles and guidance applied?

Methodology:

An evaluation matrix was developed based on the four evaluation questions. The matrix outlined the sub-questions and sources of all required information and was used as the basis for all interviews and research questions and for documenting findings. Information was triangulated through different sources and reference documents.

The evaluation team spent three weeks in SS, and conducted interviews and research in a cross-section of locations, including government- and opposition-controlled areas, several conflict-affected areas, non-conflict areas, deep field locations, PoC sites and sites with IDPs and host populations. Fourteen locations in six states were visited. Individual evaluation team members took the lead on specific sectors and themes. Where possible, the team member responsible for community consultations travelled with another ET member to allow for triangulation and for communities’ voices to act as a cross check to agencies’ perceptions. 

Data collection methods and tools used in the evaluation included:

  • More than 150 interviews with people from UN, NGOs, ICRC, IOM, CSOs, national and regional government representatives, donors, UNMISS and the UN Police.
  • Twenty-six key informant interviews, transect walks and timelines were undertaken with community mobilizers, health care mobilizers, hygiene promotion staff and representatives of women and men.
  • Two focus group discussions, detailed key informant interviews and participatory rural appraisal timelines were undertaken in two POCs.
  • Twenty-four community consultations with groups of men, women, youth, children and mixed groups.
  • Thirteen inter-agency meetings, including sector cluster meetings, ICWG meetings, HCT meetings, contingency planning meetings, security briefings and crisis coordination meetings, and meetings of community leaders and women.
  • One panel discussion with five women (14-25yrs) by Internews correspondents on behalf of the IAHE team.

Findings and Conclusions:

The evaluation concluded that the main response objectives related to saving lives, providing protection and supporting livelihoods were appropriate and have generally been fulfilled. The evidence supports the claim that the response averted any major public health catastrophe, including famine. Working in collaboration with UNMISS, the HCT response reached 100,000 people with life-saving protection in Protection of Civilians (PoC) sites. Excellent and innovative work in support of livelihoods was conducted and will form a strong basis for future expansion in this sector.

The Humanitarian Coordinator (HC), raised US$1.27 billion in 2014, 70 per cent of the requested funds, which represents an excellent result relative to most other emergency appeals. At the same time, the response was slow to deploy in the early months of 2014, which undoubtedly resulted in preventable deaths and suffering. Not enough attention was paid to the spirit of the humanitarian programme cycle (HPC). Response plans were written primarily as fundraising documents rather than as strategic documents against which progress could be measured and regular course corrections made. Coordination structures were present in Juba, but outside of the capital, this was very light. The severe shortage of experienced coordinators and leaders, especially outside of Juba, was the single biggest weakness in the response and had major side effects for programme coverage and quality.

The general level of AAP was disappointing and there was lack of leadership in this area by the HCT. NGOs could have played a greater role in the response. Also, the response took a balanced and appropriate approach to working with national and local authorities, seeking to collaborate and support those elements that demonstrated a commitment to humanitarian efforts and could put politics aside.

The Transformative Agenda was found not to have had a major impact on the South Sudan response.

Recommendations:

A number of recommendations were noted, including:

  • The HC and the HCT should hold cluster lead agencies accountable for ensuring that the clusters are adequately staffed for a response of this magnitude.
  • Starting with a problem analysis and review of current initiatives, the IASC should lead concerted action to address the global crisis in human resource capacity for emergency response.
  • The HCT should focus on forward-looking strategic and contingency planning, starting with a revised SRP for 2015.
  • SRPs should be prepared using a rational strategic planning process, to make them useful as tools for programme cycle management and fundraising. Allocate adequate time to response planning and seek a genuinely inclusive process in which the views of affected people are given prominence.
  • Cluster-level plans should follow standard approaches with cross-cluster priorities agreed to at the outset. The use of common indicators in donor contracts could help align agency donor reporting with cluster reporting.
  • As the main engine of coordination, the clusters must be fully resourced with the appropriate, single-hatted staff and be held accountable by the HC/HCT.
  • The ICWG should take on proactive inter-cluster management, in line with the revised terms of reference. Its focus should be on making sure all cluster business is aligned with the strategic response plans, that the work of national and sub-national clusters is in sync and that clusters adopt consistent approaches to managing and monitoring quality and coverage.
  • Reinforce OCHA at the state level with sufficiently experienced staff.
  • Adopt a flexible approach to programming. All plans should support resilience. Expand initiatives with longer-term impact, e.g. in education, health and livelihoods, but be ready to respond to new crises as they arise.
  • AAP needs to be embraced and promoted by the HC and the HCT as something that can add great value to the response, in line with IASC guidance.

Lessons Learned:

  1. The emergency is merging with the long-term pattern of conflict and underdevelopment in South Sudan and a key lesson has been that there is no clear choice between relief and development (or resilience, recovery, etc.). Inter-agency operations in South Sudan would benefit from a fresh paradigm that can nimbly move between these approaches.
  2. The HRP should learn lessons from previous operations, HRPs, and Operation Peer Reviews to ensure same mistakes are not repeated.
  3. A lesson on implementing AAP from the Philippines L3 response to Typhoon Haiyan was that agencies must be better at listening to affected people on their terms and allowing them to set the feedback agenda.

Note: You will find the report below labeled as follows:

  • REPORT - Full Report
  • PART 2 - GEROS 2016 Rating
  • PART 3 - GEROS Executive Feedback Summary


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Region:
Global

Office:
Evaluation Office

Theme:
Emergencies

Institution:
Office for the Coordination of Humanitarian Affairs (OCHA) on behalf of the IAHE Steering Group

Partners:
OCHA, WFP, UNICEF and UNHCR

Type:
Evaluation

Language:
English

Sequence #:
2016/004 EO

New enhanced search