Author: Hodge, M.; UNICEF NYHQ
1. Objectives of the evaluation
1. To identify which particular parts of the UNICEF I+ effort are working well or need adjustment.
2. To provide information to I+ decision takers about potential policy changes that can be identified for the targets and/or strategies for the periods 2004-2005 and 2006-2009.
For reasons related to the newness of the methodology, the evaluation objectives are targeted at the global and partially at regional levels of UNICEF. Field office level data and conclusions are not present, and the evaluation refers only indirectly to the field level.
3. Strategic Approach: Risk Assessment
An evaluation using risk assessment involves working with the client or subject of the evaluation to identify what the key potential risks are to achieving their objectives, and then analyzes how well these risks have been reduced. Risks are rated in three ways:
• Low: The maximum possible safeguards and best practices are in place.
• Medium: Some of the best practices are in place; others are missing or being developed.
• High: All or most of the safeguards and best practices are absent or under-developed.
Careful interpretation is required: serious problems may occur despite a Low rating; there may not be problems despite a High rating. What is being measured is the likelihood that UNICEF will experience problems given how well it’s policies and support systems are functioning.
4. Topics Examined
The conditions being assessed are clustered under two large themes: the policy environment (consistency, sustainability, coherence with the MTSP, partnerships, and policy review) and the enabling environment (Programme guidance, resource mobilization, human resources, and learning systems). See Table 1, Section D of the Evaluation Report for details.
An independent external evaluator was employed, under direction from the Evaluation Office. Document reviews and interviews provided the main inputs. The variables to be assessed and the risk definitions were a joint exercise involving the I+ team and others. They build upon existing Programme guidance. The ratings and most of the analysis in the report are from the external evaluator, with final adjustments by the Evaluation Office.
The methodology has strengths and limitations, as noted in section 2 of the evaluation report.
Note: the findings are valid as of June 2004 and do not reflect changes since then.
Within the 25 specific ratings given:
• Only 8% (2 ratings) show the lowest risk level, meaning maximum best practices and safeguards are in place.
• 64% (16 ratings) have a medium risk rating.
• 28% (7 ratings) show the highest risk rating.
The major comments about the findings are:
1. The risk pattern is equally strong and weak for both the policy and enabling environments.
2. There are numerous strengths that are clear resources to build on.
3. I+ is perceived as essentially a continuation of activities pre-dating the MTSP, with a focus mainly on single-vaccine preventable diseases via Accelerated Disease Campaigns (ADCs).
4. Building health systems for the sustainable delivery of routine vaccination services is receiving much less attention than the ADCs. Weakly resourced national settings cannot easily support both.
5. Other parts of UNICEF that would have been expected to adjust their actions in order to link with or reflect the I+ priority are rarely doing so, and then mildly.
6. The immunization and ‘plus’ elements remain largely separate. Vitamin A is the notable exception.
7. The I+ leadership group appears to be predominantly responding to crises rather than leading ongoing evidence-driven reviews of policy. Crises are effectively responded to but consume time needed to achieve MTSP goals.
8. External donor finance is a major risk. There are very few donors and they are pushing efforts much more toward ADC activities than toward strengthening routine immunization services.
9. UNICEF’s HR approach has not changed in ways that explicitly respond to and support the MTSP directions.
10. Certain risks are external to UNICEF and can only be mitigated through patient advocacy; others are wholly internal and can be changed much more rapidly.
11. The MTSP has centralizing and prioritizing tendencies. These tend to run counter to UNICEF’s decentralized and grass-roots programming structure. The necessary redesign of the staffing profile and of many other enterprise processes indicated by this centralizing thrust is unfinished — and in many cases not started.
7. Priority Recommendations (Short and Long-term)
1. Clarify accountabilities of regional Immunization and related advisors.
2. Assess organizational learning efforts in I+. Establish PER accountabilities for generating and using information gathered from Programme efforts.
3. Consider an additional round of this evaluation, with a focus on the regional level, and potentially some CO level case studies.
4. Examine the linkage concept behind I+ for the next MTSP. Also assess for each country the feasibility of implementing ‘Plus’ activities beyond vitamin A.
5. As campaign style ADCs decline, start fresh thinking on strengthening routine immunization.
6. Evaluate the current PD and RO structures to see if they maximize coherent inter-sectional approaches to the MTSP.
PDF files require Acrobat Reader.