We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2017 Uganda: End of project Evaluation Enhanced Resilience Karamoja Program (ERKP)



Author: Christina Blanchard-Horan, Jasmine Fledderjohann,Wamuyu Maina, Denis Bwesigye, Flavia Miiro

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

The Uganda Demographic and Health Survey (UDHS), 2011 revealed that the Karamoja region had the highest rates of child stunting (45%), Severe Acute Malnutrition (SAM) (2.6%) and Global Acute Malnutrition (GAM) (7.1%) in the country. The July 2016 Food Security and Nutrition Assessment (FSNA) indicated that the prevalence of chronic under-nutrition (stunting) was 23% and that of underweight was 17.1% in Karamoja. An FSNA conducted in Karamoja in 2015 showed that GAM rates were at their highest since 2010.

In a bid to reduce vulnerability and build resilience to cope with the effects of climate change and levels of acute malnutrition, the DFID supported the Enhanced Resilience Karamoja Programme (ERKP) in 2013. The programme started in October 2013 and was collaboratively implemented by UNICEF, Food and Agricultural Organisation (FAO) and World Food Programme (WFP) from November 2013 to date. The ERKP approach worked within the context of the transitional period and worked to move from emergency support to development of programming through improving:

  • Access to high impact nutrition services.
  • Food and livelihoods security
  • Early warning systems
  • Development coordination
  • Evidence and learning

The UNICEF component of the ERKP strategy for reducing prevalence of GAM and SAM fell into four broad categories:

  1. improving high-impact nutrition interventions and food supplementation and managing acute malnutrition
  2. improving advocacy, coordination and partner interaction;
  3. increasing knowledge and understanding and securing timely and quality information;
  4. strengthening contingency planning and emergency preparedness.

Purpose/Objective:

An end-term evaluation of the UNICEF-supported components of the broader ERKP was conducted with a goal of understanding the effectiveness and impact of UNICEF’s work for the period November 2013 to December 2015 to guide future programming. The investigation was expanded to March 2016, due to timing of the evaluation in the 4th quarter of 2016.

The evaluation objectives were as follows:

  1. Assess programme relevance, appropriateness and efficiency and quality of services.
  2. Assess programme effectiveness, sustainability and equity.
  3. Document evidence-based lessons, good practices and recommendations.

The audience for this evaluation is UNICEF, Department for International Development (UK) (DFID) and ERKP representatives. To forge future collaboration and efforts, the evaluation will inform ERKP stakeholders, which include sister UN organisations (World Health Organisation (WHO), WFP, FAO) and other iNGOs providing nutrition services in Karamoja.

Methodology:

A cross-sectional study was conducted between 26th September 2016 and 30th October 2016 within the seven districts of the Karamoja sub-region, namely the Abim, Kaabong, Kotido, Moroto, Nakapiripirit, Napak and Amudat districts. UNICEF-supported health facilities in each district were selected from purposive clusters comprising of each of the livelihood zones. Both quantitative (semi-structured interviews and data abstraction) and qualitative (desk reviews, key informant interviews, focus group discussions, unobtrusive activity observations and stakeholder mapping) primary data were collected at the national, district andcommunity levels. In addition, value for money (VfM) was assessed by examining the efficiency and efficacy (cost) of the Ready to Use Therapeutic Food (RUTF) intervention and the training.

A descriptive analysis was conducted to assess changes in the district-level nutritional outcomes. Inferential models were fit as fixed effects models to adjust for the autocorrelation of measures within facilities/districts across time. Multi-Criteria Analysis (MCA) and Cost–Benefit Analysis (CBA) were explored to demonstrate which interventions have the highest VfM. Quantitative results were presented graphically in tables, figures and charts. Methodological triangulation of the qualitative and quantitative data was conducted to enhance the understanding of the ERKP nutrition components.

Findings and Conclusions:

Several targets were met, including improvements in coverage, referral and follow-up; and the number of twice-yearly Vitamin A doses for children 6–59 months. The association between admissions and Vitamin A intake suggested that higher Vitamin A intake was associated with fewer admissions.

The programme’s social change and behavioural interventions promoted the uptake of recommended maternal, IYCF and care in the region.There was a significant association between the percentage of caregivers reporting messaging on maternal nutrition and facility cure rate and the reduction in the difference between estimates and new admissions.

Interviews indicated that from the community and facility levels to the district and national levels, the programme was linked with a variety of stakeholders. At the national level, the UNICEF ERKP leveraged their strengths and relationships to build support for nutrition planning and implementation. UNICEF built strong partnerships with the MoH to improve nutrition in Karamoja, influence policy and foster integration of the IMAM programme into the mainstream health system to build sustainability.

The associations between cure rates and livelihood referrals suggest that the most critical move that UNICEF Uganda can make for the nutrition programme at this point is to build the needed processes and procedures to roll out a multi-sectoral programme that incorporates livelihood programmes with IMAM services.

Although UNICEF ERKP contributed significantly to contingency planning, this was not captured in the log frame. Furthermore, targets that were reflected in the log frame for 2015 and 2016 were not met. In 2016, this began to change as plans were going through for the approval and rolling out of district contingency plans. There were signs that the contingency planning and emergency preparedness were underway in Karamoja.

Recommendations:

  1. Improve the UNICEF ERKP log frame and develop theory of change model for future interventions.
  2. Integrate communication at all levels to present common nutrition messages that refines linkages with all stakeholders and establishes regular communication mechanisms between them, e.g. Strengthen links between UNICEF, FAO, and iNGOs by designing programmes that engage stakeholders and community health providers to identify and refer chronic cases.
  3. Streamline nutrition information and data analysis with support to build analytic structure.
  4. Recognize and study high preforming facilities to identify best practices for duplication.
  5. Construct district livelihood and nutrition profiles to help guide targeted interventions that close the gap between nutrition sensitive and nutrition specific barriers to services.
  6. Measure stunting to understand intervention impact, e.g. EBF at >90%, and continually every two years’ post intervention.
  7. Support DNCC development and ensure joint sustainability plans that clearly highlight government and community roles, responsibilities, and nutrition budget.
  8. Cultivate better coverage with community engagement strategies to serve hard-to-reach populations, such as those discussed in resilience planning meetings. Consider effective community-based management of acute malnutrition (CMAM) models that have been effective in sub-Saharan Africa.

Lessons Learned:

Document evidence-based lessons, good practices and recommendations
The UNICEF ERKP was not set up for attribution of results. To establish attribution, strong causal links must be found between the intervention and the observed outcome. It is clear, however, that the initiative achieved strong results around strengthening institutions and policies and integration of nutrition services into existing systems.

EBF increased over time, surpassing the country average. Although they slightly increased over the programme period, the rates of dietary diversity were very low. Complementary feeding was also low, at a quarter of the population practicing. Clearly, EBF education has been effective. We postulate that although mothers understood the importance of complementary feeding, compliance was complicated by the lack of food availability and resources to obtain proper available foods .

Unexpected outcomes – The number of cases of SAM for under-fives was positively associated with both the number of supervisions (b=12.4; p<0.01) and the number of VHTs who could correctly count Respiration Rate (RR) (b=12.7; p<0.05). Thus, the higher the number of supervisions and VHTs able to correctly count RR, the higher the number of admissions of children under five. Counting RR correctly may be a proxy indicator for health education level of staff.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2017

Country/Office:
Uganda

Region:
ESAR

Type:
Evaluation

Theme:
Nutrition

Language:
English

Sequence #:
2017/020

New enhanced search