Evaluation database

Evaluation report

2013 EAPRO: External Evaluation of the EU/UNICEF Partnerships on Nutrition Security. MYCNSIA Mid-Term Evaluation



Author: Annemarie Hoogendoorn, Joanne Harnmeijer, Bert Lof, Albertien van der Veen

Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report."

Background:

Various Asian countries show high absolute levels of children under five who are stunted and/or wasted. The EU/UNICEF Partnership on Nutrition in Asia is a 4-year programme (€ 28.4 million) aimed at contributing to reducing undernutrition (stunting and anaemia) in five countries (Indonesia, Philippines, Lao PDF, Nepal, and Bangladesh) with high (>30%) or very high (>40%) levels of chronic malnutrition (as evidenced e.g. by national stunting prevalence figures). The overall programme target is to achieve a 5% reduction in stunting and one-third reduction in anaemia among pregnant women and children through work under four pillars: (i) upstream policy development; (ii) capacity development; (iii) nutrition information systems; and (iv) direct nutrition interventions. The Partnership also has a component aimed at positioning nutrition security high on regional agendas which is implemented by the Programme Management Unit (PMU) spread over the UNICEF Regional offices in Bangkok and Kathmandu.

MYCNSIA is based on the 1,000 days approach which promotes good nutrition for pregnant and lactating women and infants up to the age of two years. as the best and most cost-efficient intervention for avoiding irreversible damage to physical growth and intellectual capacities from undernutrition.

MYCNSIA operates in a context with two other main international structures established to address undernutrition:
• REACH started in 2008 as a mechanism in which four UN agencies (FAO, UNICEF, WFP and WHO) collaborate in assisting country-led approaches in selected countries through support to coordination and through operational guidance;
• SUN was launched in 2010 as a mechanism to encourage high level of political commitment, uniting governments, civil society, the private sector, and citizens on Scaling Up Nutrition.  The movement currently covers 35 countries (within MYCNSIA Philippines is the only country that has not (yet) joined.

Purpose/Objective:

Strategic level:
Determine the relevance and appropriateness of the logical models and strategies (each with sub-results) that are in place for each of the four pillars (results areas) of the MYCNSIA / ANSP programmes, with the aim to identify the value added by the programmes as a whole taking into account what is already happening / existing.
Assess whether programme design and strategies have sufficient in-built focus on equity ).

Operational level:
Review  the implementation process for each of the results areas in the MYCNSIA and ANSP programmes, in particular with respect to targeting (also in relation to equity ) and operational efficiency (sufficiency of available inputs; outputs according to planned timeframes; specific bottlenecks and/or enhancing factors during implementation)   
Assess the effectiveness of the MYCNSIA and ANSP programmes in terms of the results achieved for each of the four results areas . This will include a description of the factors that influenced the achievement or non-achievement of the planned targets. Also it will be assessed what impacts can be expected (the real difference for the beneficiaries for each of the results areas, plus unintended positive or negative results if any) and sustainability (long-term benefits, continuation of generation of benefits after completion of the programmes). 
Compile an inventory of good practices and lessons learned for all four results areas.

Methodology:

The methodology combined quantitative and qualitative participation. It was based on several methodological principles including: consultation, transparency, triangulation, among others. It included.

Analysis of the country context
For each country, a short overview was prepared of the main policy frameworks and key stakeholders on nutrition security in order to get good understanding of the context in which MYCNSIA / ANSP operates.

Equity analysis
Use of “mini” case studies
A set of mini case studies has been included in the MTE for more in-d The selection was made in such a way that the set of mini cases generates a variety of observations on programme implementation for each of the four result areas.

Data collection tools
Desk review
Based on documents provided by the Regional Offices and Country Offices, plus a separate web-based search undertaken by the consultants

Semi-structured interviews
The interviews were based on a semi-structured checklist. As a visual tool, diagrams were prepared for each of the MYCNSIA target countries and at the regional level as a quick reference to the sub-results in each of the four result areas.

Field visits to project sites
The selection of sites to be visited was based on purposive sampling. The choices were made in coordination with the nutrition focal points in the Country Offices and EU Delegations.

Findings and Conclusions:

Generic logical models and log frame, country-specific and regional-level work plans
In 2011, the MYCNSIA developed two ‘generic’ logical models, however, a clear ‘theory of action’ is missing, and insufficient insight is provided into the connections between outputs, outcomes and impacts. So far the log frame has not yet been used for annual reporting.

Advocacy at regional level flourishes but it also lacks
It is very positive that UNICEF has achieved to engage with ASEAN and SAARC. Although most of the work is in the initial stages, it is a milestone achievement.  More critical observations are that after two years of operation most of the MYCNSIA documentation for advocacy (advocacy briefs, country profiles) is still under preparation, and that the usefulness of the type of materials for advocacy to decision-makers seems questionable.

UNICEF provides highly useful support to upstream policy development but with not enough focus yet on models to ensure that policy translates into action.

Mixed results for capacity development. There is little evidence of the effects achieved thus far (e.g. data on knowledge and capacities gained are very scarce). Capacity Development efforts in the five countries sometimes were hampered by contextual constraints.
   
Large differences in the packages of interventions that are being scaled up
Government policies are widely different from country to country, and often still being developed. The level of advancement on introducing CMAM varies considerably across the five countries but hardly anywhere is beyond piloting at smaller scale.

Good potential for institutional sustainability
Overall, in the five countries there now is an enabling environment with high Government interest in nutrition. However, there still is a way to go until there will be sufficient budget allocations for nutrition so that the key direct nutrition intervention programmes can be brought to scale.

Recommendations:

FOR UNICEF
More explicit emphasis on supporting Government systems for policy development and design of implementation models. Impact indicators for new programmes should also reflect ‘systems’ variables
 
Further engage in technical backstopping to national nutrition monitoring systems

Recommendations to the Programme Management Unit for remaining programme period:
i) Put more emphasis on exchange between MYCNSIA target countries
ii) Look for options to reap synergies between countries on capacity development
iii) Further engage with ASEAN and SAARC for information dissemination and capacity development
iv)  Analyse how equity can better be operationalized in the MYCNSIA M&E systems
v) Commission an impact study as add-on to the MYCNSIA final evaluation

Recommendations for UNICEF Country Offices:
i) Annual reports 2013 and 2014 to provide information on the work plan indicators next to the system of self-scoring of level of achievement.
ii) Consider scheduling a quick scan on the level of national policy implementation 
iii) Consider commissioning a costing study on one or more Pillar 4 interventions (IYCF, MNP/IFA/MMN distribution, CMAM)
iv) Advocate for use of nutrition indicators as impact indicators for other sectors and in multi-sectorial efforts
v) Promote the elaboration of a national matrix which specifies the role each agency takes up in relation to addressing undernutrition.

Lessons Learned:

Need to clearly specify in the programme design what the intended roles and contributions will be of the various stakeholders involved. The lesson learned from MYCNSIA is that the design as framed in the log frames should be clear from the beginning.

Inter-pillar complementarity brings added value, but also carries a risk; the caveat of interconnections between programme activities.  Complementarity between the pillars has been MYCNSIA strength in some of the countries. However, it is also good to realize that connecting programme activities cause dependencies between programme components. For instance, in Lao PDR, the development of the first national IYCF guidelines and training package,

In the economically more advanced countries in the region (Philippines, Indonesia), it is not appropriate for UNICEF to provide budgets or in-kind supplies to Government (IFA, ORS, Zinc, MNPs, etc.). Main focus support to these countries should be on provision of assistance to the Government for development of policies, capacities and implementation models. 

The difficulty of mainstreaming a truly multi-sectorial approach should not be underestimated. The lesson is that where the opportunity occurs it should be exploited!  Social protection schemes appear to offer such an opportunity and also have the advantage of targeting under-privileged segments of the population.

Opportunities to mainstream nutrition in existing social protection schemes have been used to a limited extent. Also from a multi-sectorial perspective the equity angle deserves more explicit attention. A multi-pillar approach would be preferable. 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2013

Region:
EAPRO

Theme:
Nutrition

Type:
Evaluation

Partners:
European Union (EU)

Language:
English

New enhanced search