Evaluation database

Evaluation report

2012 WCARO, Senegal: Real-Time Independent Assessment (RTIA) of UNICEF’s Response to the Sahel Food and Nutrition Crisis, 2011–2012



Author: Ricardo Solé Arqués & Enrico Leonardi

Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."

Background:

The countries of the Sahel are among the lowest-ranking on the Human Development Index (HDI). An estimated 645,000 children die in the region every year, with up to 30% of these deaths linked to malnutrition ; GAM rates close to the emergency threshold are present in specific districts in all of them and in the “Sahel belt” regions of Chad are even over the critical 15% threshold. 

The region has suffered well-documented episodes of nutritional crisis in the recent past, is prone to epidemics of diseases such as measles, cholera, and meningitis, and is exposed to significant levels of malaria, which is endemic here. Polio is still recurrent in some countries, requiring frequent immunization campaigns. UNICEF, as the mandated agency for the protection and assistance of children, is present in the Sahel and has developed partnerships with governments, international organizations (IOs), and community-based organizations (CBOs) in order to promote policies and technical capacity to deal with the challenges facing children in the region. UNICEF is pushing for child-oriented agendas in all planning and development undertakings in each country.

Following the nutritional crises of 2005 and 2010, at the end of 2011 a new deterioration was anticipated for 2012, due to poor rains and rising food prices. Early warning systems (EWS) and other forecasts predicted that more than 12 million people would be affected across eight countries of the Sahel, with an expected SAM caseload of more than 1 million children aged under five. 

The assessments and information gathered at the end of 2011 were shared with the RO and with EMOPS, and UNICEF took part in the articulation of an Inter-Agency Standing Committee (IASC) response strategy that addressed the nutritional emergency, including warnings of outbreaks of cholera and polio in the region. UNICEF’s response to the nutritional crisis was triggered by the declaration of a de facto Level 2 emergency , which made possible an enhanced response, the mobilization of resources, and the application of specific procedures to facilitate processes. An EMT was established at WCARO level and a surge capacity for key staff was activated. 

In February 2012 UNICEF launched a Humanitarian Action Update (HAU), appealing for a total of USD 119,516,156 to address needs initially in eight countries (Burkina Faso, Mali, Mauritania, Niger, Senegal, Chad, Northern Nigeria, and Northern Cameroon). This appeal was revised in June 2012 to USD 239 million and Gambia was included in the response. UNICEF is aiming to deliver an integrated package as a strategic response to the underlying causes of the problems addressed, with the aim of ensuring the Core Commitments for Children (CCC). This includes interventions in Nutrition, Health, WASH, Education, HIV/AIDS, Child Protection and C4D. 

It should also be noted that deteriorating conditions in Mali have affected the humanitarian situation in the north of the country, and an influx of refugees has spilled over into neighboring countries. This has had consequences for the previous estimates and appeals, and plans have been developed accordingly (Consolidated Appeals Process (CAP) appeals for Mauritania, Niger, and Mali). 

The geographic scope of the Sahel crisis, the magnitude of its toll on children, and the level of response provided all underline the need for UNICEF to learn from and improve upon its response in the earliest stages of the crisis. Toward this end, it was deemed important that UNICEF undertake a Real-Time Independent Assessment (RTIA) to gain an impartial perspective on whether its response to the crisis to date has been adequate, whether the targets defined in the integrated response plans have been adequately aligned to the CCCs and are being met, and whether resources are being used in the most efficient and well coordinated way.

A quick and light RTIA was proposed to facilitate such learning, in order to feed into UNICEF’s operational decision-making and program improvement while the response was still ongoing. The exercise was intended to provide timely feedback on the efficient use of resources available and to enable the COs and RO to reallocate resources where required, adjust approaches, revise targets, and/or amend procedures, as necessary.

Purpose/Objective:

This RTIA was planned as a learning exercise to be carried out after the first phase of the response, in order to inform decisions during the second part of the intervention. A team of two external experts was mobilized and the field phase took place between June 24 and August 3, 2012. The objectives and scope of the exercise were defined in the ToR (attached as Annex 5) as follows:
o The main objective of this RTIA is to assess, as systematically and objectively as possible, what in UNICEF’s response in the Sahel has been working well, what has been working less well, and why, with a view to providing real-time feedback to improve the response moving forward. 

Specifically this RTIA examines UNICEF’s Integrated Programme Response, National Strategy Response Plans supported by UNICEF, country Cluster/Sector Lead Agency roles (where relevant) and operational support in the overall context of the benchmarks set in the CCCs, as contextualized and prioritized in the Sahel response, through rigorous evidence-based analysis, while identifying gaps or unintended outcomes, with a view to improving UNICEF’s strategy and the program’s approach, orientation, coherence and coordination.

The scope of this RTIA is limited to the nutrition response, and it was agreed that the response to concurrent crises, such as the refugee crisis resulting from the conflict in Mali, cholera outbreaks in some of the countries involved, and polio outbreaks of high relevance for UNICEF would not be assessed, although their mutual influences and interactions are noted where relevant.

With regard to timeframe, the RTIA’s ToR referred to a period between the HAU of February 2012 and May 5, 2012. Delays in the deployment of the team for this RTIA meant that the period to be assessed was extended up to the beginning of the mission, i.e. June 25, the deadline for June’s Situation Reports (sitreps), which was considered an acceptable reference point to draw a limit. This posed a particular challenge in a mission that extended until the beginning of August (see “Limitations” section below), and the consultants’ team in fact extended the assessment period until the publication of the July sitreps, just before the debriefing session in Dakar.

Methodology:

The methodology of the RTIA is based on both quantitative and qualitative approaches and has been articulated around three core elements: 
• A comprehensive desk review, which resulted in an inception note, with an evaluation framework established and approved by the AMG;
• Field visits to the Western and Central Africa Regional Office (WCARO) and four Country Offices (COs) – Nigeria, Niger, Mauritania, and Chad – where interviews were conducted with key UNICEF staff and with other actors present in the field and relevant in the context of the response to the nutrition crisis, including local authorities, leading NGOs, and other UN agencies; and
• Visits to sites and malnutrition treatment delivery posts.

To complement the above and to reinforce the participatory nature of the process and an adequate validation of its provisional conclusions, debriefing sessions were carried out at each CO and at the end of the mission at the Regional Office (RO) in Dakar. 

Findings and Conclusions:

The current response is a nutrition-oriented one, and it can be judged to have been timely and successful in terms of preparedness and the human resources (HR) surge, and in positioning supplies in advance so that all COs have been able to address their estimated caseloads of severe acute malnutrition (SAM). 

Needs were estimated on the basis of SMART surveys, and a decision was taken to cover 100% of the predicted caseload. This meant a 76% increase in the caseload compared with that reached in 2011, establishing a target of 1,095,000 children with SAM to be treated in 2012.

UNICEF’s response to the Sahel crisis: achievements
• Change of mindset regionally: aiming at a target SAM caseload 76% higher than in 2011;
• Timely pre-positioning of ready-to-use therapeutic food (RUTF) region-wide;
• Surge in HR capacity;
• Communication, awareness, utilization of social media;
• Creative solutions (VISION task forces, cash-in-hand, “5+1 axes” strategy).

The initial focus on nutrition was adopted in order to mobilize resources for the treatment of SAM at scale in a region where a number of national nutrition programs are significantly under-scale, and this focus proved useful in raising the profile of the crisis and attracting public attention. UNICEF’s initial proactive approach, its timely response in the deployment of the HR surge, and the pre-positioning of RUTF supplies across the region have been identified as significant achievements. Additionally, the different COs have been able (with occasional RO support) to deal with a number of communication demands (visits, fact sheets, media attention) and in some cases (Niger, Chad) have successfully used social media. Creative initiatives at CO level (CO VISION task forces, cash-in-hand authorized by the Division of Financial and Administrative Management (DFAM)), together with remote assistance from HQ, are worth highlighting and show that there is readiness to overcome difficulties (internal administrative constraints) even if L2 procedures have not been clarified.

However, the response has not been based on a comprehensive assessment of the needs of children and their families in the context of a nutrition crisis, other than the need to ensure the availability of RUTF. Furthermore, the absence of both a strategic plan and a subsequent operational plan integrating all sectors has been acknowledged as a significant issue to be addressed. Some aspects were not given enough attention or support at the beginning of the response, especially Education, Communication for Development (C4D), and HIV.

Early in the response, UNICEF’s HQ and the RO proposed to the COs planning matrixes for the different sectors and a Humanitarian Performance Monitoring (HPM) system that would harmonize indicators to monitor performance. The roll-out of this system has suffered due to the lack of integrated planning and has proved to be inadequate in the context of a slow-onset nutritional crisis. 

The support provided by the RO to the COs has included operational, logistical, and technical support, and has been very much appreciated. Support from HQ was also provided in the context of designing a more integrated response, but this has not yet been properly articulated. L2 procedures and their implications are not well understood at CO level. In some cases, especially in relation to technical support to individual sectors, UNICEF’s “silo” structure has prevented further integration. The traction of the Emergency Team (ET) at RO level has been only partially translated at CO level.

In some COs, specific plans and strategies have been put in place. In Niger, the response has been integrated by all actors into an already consolidated operation; in Chad an integrated plan has been established; and in Mauritania a “5+1 axes” strategy is being put in place to operationally integrate sectors in different geographical areas and to allow the adequacy of the HPM approach to be tested.

The issue of moderate acute malnutrition (MAM), including its characterization and the definition of needs and an adequate response, is still a contentious one between UNICEF and WFP. Some progress has been made, but with UNICEF having overall responsibility for nutrition, there is a need to establish better arrangements and coordination with WFP.

The nutrition response has been successful overall in pre-positioning supplies, which has potentially improved the quality of the response, but it faces specific challenges in delivering treatment to the planned caseload. Lack of capacity of local governments, contextual difficulties of access, and a shortage of implementing partners are factors limiting its capacity to reach all beneficiaries. Every country visited in the course of this assessment faces specific challenges in this respect. Overall, trends at the time the RTIA was undertaken suggest that it will be difficult to meet the estimated caseload, although we are still in the midst of the lean season and the situation may change. Problems of delivery capacity are significant, as are the lack of supervision capacity at end-user level and the lack of beneficiary participation. These are all issues that need to be addressed urgently.

Other sectors are behind schedule, which is blamed on lack of funding, while the lack of clear strategies makes it more difficult to raise funds. Of particular concern is the case of WASH, where UNICEF has specific responsibilities. The “WASH in Nut” strategic note is an interesting starting point, well known in the COs visited, but this initiative faces difficulties in operationalization. Implementation in the field of the WASH component is still limited. The Health sector has approached the response to the nutrition crisis from the perspective of its regular programs, trying to ensure adequate coverage of immunization and the distribution of insecticide-treated nets (ITNs), activities that are not directly connected with the response. The poor integration of Child Survival and Development (CSD) elements (Health, Nutrition, WASH) raises concerns about their internal articulation within UNICEF and the “siloing” of the sections identified during the RTIA. The Education and Child Protection (CP) sectors have experienced difficulties in integrating activities into the response due to the lack of clear strategies, although both sectors have advanced in some countries in terms of developing such strategies. Psychosocial stimulation of children, from a CP perspective, still represents a very limited element in the response. There is an HIV/AIDS Sahel concept note that has been largely ignored thus far.

The lack of beneficiary involvement in the response is striking, considering the potential for UNICEF to involve communities and the consolidated tools in place. Of particular concern are the limited attention paid thus far to gender issues and to disabled people, and the limited operationalization within the response of HIV and prevention of mother-to-child transmission (PMCT) aspects. 

In addition, the fact that UNICEF has taken on responsibility for being the main or almost the sole supplier of RUTF raises issues of image and accountability. Equity concerns, in addition to known socioeconomic factors, could potentially be raised in countries in non-Sahel regions where malnutrition is present but is not being addressed at the same level as in the current response. 

Recommendations:

1: Establish an integrated planning framework to guide the response.
This recommendation encompasses findings contained in paragraphs 7, 9, 18, 21, 23, 24, 45, 48, and 49–54 of this report, and includes as a logical consequence the issues of monitoring results (HPM) (paragraphs 20–22 and 25–32). 
• Develop an RO strategic plan based on a comprehensive needs assessment and on inputs from COs’ response plans. This should aim at the formulation of clear strategies to integrate other sectors into nutrition crises in the Sahel. This could include a kind of “switchboard to emergency mode” in all normal programming, allowing for preparedness and response in the event of the situation deteriorating. 
• Articulate a response plan, based on the above. Operationalization of this could build on elements of the proposed matrixes used in the current response and should articulate and include other strategic approaches from the different sectors. Of particular relevance would be the integration of Nutrition, Health, and HIV in a coherent approach to the potential performance, capacity, and limitations of the health system, with set objectives aimed at reducing under-five (U5) mortality. C4D, Social Protection approaches, Education, and Child Protection could play critical roles in creating a more integrated response.
• Harmonize HPM indicators, taking into account context-specific situations and inputs from COs.
• Establish communications and fundraising strategies to actively seek funding for an integrated response, encompassing all sectors. Establish agreements with donors so that there is flexibility to switch normal programming into emergency preparedness and response.

2. Take measures to urgently improve the delivery side of the response (these measures will be context-specific). 
Within this broad recommendation, findings in paragraphs 46–53, 55, 58–61, 66–68, and 71 are addressed. All of these address specific aspects of the delivery and demand side of the response and should be treated as a whole (some COs have already taken measures to address the more relevant aspects of each issue).
• Possible solutions to coverage constraints include extending the number of centers providing malnutrition treatment; using mobile clinics; promoting community-based SAM case management; and enhancing capacities at provincial and Field Office (FO) levels. 
• Addressing logistical bottlenecks in RUTF distribution at field level: These are problems identified at end-user level, and require aspects of service delivery to be reinforced, including stock and supply management at provincial level. Promoting local production of RUTF should be considered as a medium-term strategy to reduce dependence on external sources.
• In the medium and long terms, rapid assessments should be conducted prior to the signature of Project Cooperation Agreements (PCAs) and entering into partnerships with actors known, or identified, to be efficient, so that in the event of crisis they can be more immediately proactive:
o Facilitate the process of assisting nutrition actors who are not UNICEF partners;
o Increase allocations for transportation of supplies to implementing areas;
o Reinforce logistical and programme capacities of UNICEF FOs; 
o Establish as a component of PCAs training and capacity building in stock management.
• Addressing performance indicators for defaulters and cured rates: Data management is crucial to better address potential problems as well as underlying causes of defaults. 
• Prospective analysis of programs’ performance to refine the supplies needed consistent with the capacity for delivery, and to review implementation plans.
• Better integrating the Nutrition, Health, and HIV sectors in a coherent approach, taking into account the potential performance, capacity, and limitations of health systems, and with set objectives aimed at reducing U5 mortality. Barriers to access, including economic (recovery costs), cultural, and geographic barriers, should be addressed.
• Involvement of beneficiaries: building on tools developed in different contexts, a framework should be established for the participation of beneficiaries.
• Mobilization of local communities and utilization of volunteers: COs need to better coordinate this issue in-house and with other organizations.
• UNICEF should proactively attract NGOs and engage with them on the implementation of the emergency nutrition program in order to ensure a short-term enhanced capacity to deliver.
• Sector (cluster) and inter-sector (inter-cluster) coordination arrangements must be improved in areas under UNICEF’s responsibility. Coordination of partners could improve capacity to deliver services and enhance access and coverage.

3. Address shortfalls in implementation in the WASH sector.This is a straightforward, though challenging, recommendation, given the importance of the WASH sector and the accountability implications involved. It relates, among others, to paragraphs 51 and 71 in this report. 
COs, with the support of the RO, should start looking into strategies to tackle challenges and implementation gaps in the WASH intervention. Among possible solutions, alternative funding approaches should be investigated: e.g. making WASH one of the priorities for funding with non-earmarked contributions; negotiating with existing and new donors to cover WASH costs with nutrition funding; and establishing a special Emergency Program Fund (EPF) for WASH. The “WASH in Nut” strategy should be utilized to make clear to donors the importance of WASH in the nutrition response. The RO WASH section should evaluate the impact of the roll-out of the strategy at CO level and its local limitations, adaptations, and successful implementations, with the aim of refining it for use in future nutrition crises in the Sahel.

4. Reinforce the HR team in WCARO, and develop a regional HR strategy.
This recommendation was supported by the COs participating and also by EMOPS and the Program Division (PD). It corresponds to paragraphs 77 and 78 of this report. Action to address this recommendation depends on internal processes; it has been kept as a key recommendation due to the importance it was given in the ranking exercise.
We recommend addressing this issue from a strategic perspective: it should be based on  a diagnosis of needs in the region, the limitations of finding French-speaking candidates, HR demand in relation to specific programmatic needs, and the availability of funding. One of the weaknesses in the HR surge capacity identified by the RTIA has been the occasional deployment of human resources without a clear operational framework or the means required to carry out the intended task. The process of HR requests, searches for candidates, selection, and recruitment has to be as transparent as possible, and communication between CO and RO HR sections has to be established through agreed channels and working tools. Ensuring adequate follow-up of HR processes in the region will likely require reinforcement in HR services at WCARO.  
As with other aspects of the response, the HR strategy must be linked to a regional strategy and to adequate sectoral strategies. In line with Recommendation 1, an eventual HR strategy would benefit from the development of sector strategies that could include a “switch” mechanism to turn normal programming into emergency preparedness and response. Achieving a minimum capacity of permanent staff able to deal with emergencies would greatly benefit UNICEF’s potential in this region, and would make the surge required in the case of acute crisis more focused and more readily integrated. However, this should not detract from the need to adequately reinforce the HR section in WCARO.

5. Optimize processes for the RO to support COs.
This recommendation addresses findings contained in paragraphs 33 and 76. These aspects also concern UNICEF’s internal management; the consultants believe that they should be given attention as they can be the cause of some dysfunction. 
However, addressing them is probably not an easy task given the “silo” structure of UNICEF. In particular, the fact that the RO’s Emergency Team traction does not translate easily at CO level and tends to overburden the functions of the Emergency Officer has to be addressed. 
• More senior staff must be available to coordinate response at CO level; or Emergency Officers should be more senior and their teams stronger.
• Dissemination of L2 Standard Operating Procedures (SOPs) must be ensured, as well as discussion about their implications in the event of a new crisis.
• More attention must be paid to the need to integrate other sectors into the initial phases of the response. This seems especially true – and also feasible – in the case of a slow-onset crisis of this type.
• A travel monitoring system could be a potential tool to facilitate integration and avoid or limit “siloing”.

6. Relations with WFP, need for a Letter of Understanding (LoU), including MAM estimations and blanket feeding coordination.
This recommendation corresponds to paragraphs 10 and 39 of this report. This aspect is well known by senior staff at WCARO and will certainly be addressed in conversations with WFP, but the solution is complex. The global cluster initiative to address MAM through a specific taskforce is a good step, but additional efforts will be needed to align the organizational cultures of UNICEF and WFP (see “Discussion” section, page 46).

7. Address cross-cutting issues not adequately covered in the current response.
• Ensure a gender-based approach, with disaggregation of gender data for the nutrition program (paragraph 63).
• Characterize disabilities in the target population (paragraph 64).
• Address malnutrition in non-Sahel areas, avoiding positive discrimination (paragraphs 8 and 65).
These recommendations are arranged in matrix format in Annex 1, with a breakdown specifying their respective urgency, feasibility, and importance.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search