2012 EAPRO: Regional Thematic Evaluation on UNICEF's Response To Decentralization, EAPRO 2013
Author: A.K. Shiv Kumar and Katherine Hay
This regional thematic evaluation attempts to (i) assess UNICEF’s strategies and interventions geared towards the realization of the rights of children and women in decentralizing environments in the East Asia and Pacific region; and (ii) identify strategic issues and options for UNICEF’s future interventions. The specific objectives of this evaluation are to (i) assess the extent to which UNICEF’s strategies, implicit or explicit, aimed at achieving results for children in decentralizing environments are relevant and effective; (ii) distill good practices and lessons learned from UNICEF’s engagement in decentralizing environments, especially in the areas of capacity development, policy advocacy, and partnerships; and (iii) formulate recommendations on strategic issues and options with a view to improving UNICEF’s strategies and interventions geared towards the realization of “all rights for all children” in decentralizing environments.
This evaluation has many dimensions of a formative evaluation which is also developmental in nature. It attempts to document and capture strategies that are themselves evolving and varied in contexts that are also evolving and varied. Given this, rather than, as evaluators, select apriori and retrospectively a set of theories and test them, the evaluation attempts to elucidate both the strategic thinking guiding work at different levels and the outcomes of that thinking. Put another way, the evaluation uses the data gathered to compare outcomes described with the needs of marginalized children – using multiple experiences as the reference point rather than a reconstructed programme theory. Accordingly, the evaluation adopts a mixed method design that includes case studies, document review, key informant interviews and survey. The design was selected as being able to generate credible, useful, findings within the constraints of time, travel, and other limitations.
The purposes of this regional thematic evaluation are :
• to assess UNICEF’s strategies and interventions geared towards the realization of the rights of children and women in decentralizing environments in the East Asia and Pacific region; and
• to identify strategic issues and options for UNICEF’s future interventions.
The evaluation focuses particularly on the extent to which UNICEF’s strategies in support of and responding to decentralization are aligned towards, and appropriate for, contributing to more equitable results for children. This is intended to help the Regional Management Team to better support responses to decentralization that help countries achieve national goals and priorities in ways that improve the lives of children, and particularly the most disadvantaged children.
This evaluation takes account of and builds upon previous studies on decentralization undertaken by UNICEF in the East Asia and Pacific region. The specific objectives of this evaluation are:
- to assess the extent to which UNICEF’s strategies, implicit or explicit, aimed at achieving results for children in decentralizing environments are relevant and effective;
- to distill good practices and lessons learned from UNICEF’s engagement in decentralizing environments, especially in the areas of capacity development, policy advocacy, and partnerships; and
- to formulate recommendations on strategic issues and options with a view to improving UNICEF’s strategies and interventions geared towards the realization of “all rights for all children” in decentralizing environments
This evaluation considered a theory of change approach. While such an approach was deemed unsuitable as the core basis of this evaluation for a few reasons some of the ideas from theory of change work did inform the evaluation. The term “theory of change” is used in many different ways. Here it is defined as a set of beliefs and assumptions about how and why an intervention will work. This is slightly distinct from the idea of ‘theories of change’ which imply that there can be a range of theories about how and why change occurs. The latter is certainly more useful in the context of this evaluation.
Instead of testing a theory of change, this evaluation uses the data and information gathered to compare outcomes described with the needs of marginalized children. It uses multiple experiences as the reference point rather than a reconstructed program theory. This is in line with feedback on the inception report, where one advisor noted, “an emphasis on policy would be misguided as it will be more interesting to look at success in programs on the ground
The evaluation design is a mixed-method design that includes case studies, document review, key informant interviews and a survey. The following data collection methods have been used:
Document review: This evaluation examines and draws from an extensive list of documentation and materials shared by UNICEF . Several of these studies have covered similar ground to this planned evaluation.
Key Informant Interviews (KIIs): Key informant Interviews were conducted with 12 Country Representatives from Cambodia, China, DPRK, Indonesia, Lao PDR, Malaysia, Mongolia, Myanmar, The Philippines, Thailand, Timor Leste and Viet Nam) ;
Staff Survey: Three survey instruments were administered to the following :
• 12 Deputy Representatives or Focal Persons from Country Offices.
• 8 Section Chiefs in the Country Offices
• 7 Heads of Zonal Offices in countries where applicable
Findings and Conclusions:
Finding 1: UNICEF’s responses in the region appear to be well aligned with and relevant to both the national context as well as government’s priorities of decentralization.
Finding 2: UNICEF has neither laid down an explicit regional strategy nor has it encouraged Country Offices to develop long-term country-specific strategies on decentralization.
Finding 3: The capacity to implement or manage programmes, more than knowledge, is perceived as a constraint in responding to decentralization within UNICEF that has adversely affected effectiveness.
Finding 5: A majority of Zonal Office and Country Office staff rates the technical and strategic support on decentralization provided by the Regional Office as ‘poor’ or ‘very poor.’
Finding 6: Four inherent tensions in UNICEF internal programming and ways of working affect the organization‟s capacity to respond to decentralization, and have limited its effectiveness.
Finding 7: Four factors in particular, internal to UNICEF, appear to account for these tensions that limit the effectiveness of UNICEF’s response. UNICEF does not have a comparative advantage in addressing issues of governance. Decentralization is not a historical or current strength of UNICEF.
Finding 8: Two approaches within UNICEF’s overall programme strategies that are most uniformly regarded as ‘successful’ or ‘appropriate’ in addressing decentralization are (i) strengthening the generation and use of data; and (ii) influencing policies through pilots. Capacity building efforts came out less clearly as being effective.
Finding 9: Deficiencies in the results framework developed and used by UNICEF have been a factor affecting the quality of planning, monitoring and implementation of responses to decentralization.
Finding 10: Inadequate attention paid to design, resourcing and implementation components of decentralization responses have tended to adversely affected sustainability.
Finding 11: UNICEF has built good working relations with a number of national governments on issues of decentralization. However, it has been less successful in establishing strong linkages in this area with other UN agencies, sectoral line ministries and sub-national levels of government.
1) Develop a framework: UNICEF should spell out a regional framework or strategy that recognizes the distinction and interplay between focusing programmatic work ON decentralization and programming IN decentralized or decentralizing contexts, and enables Country offices to make informed choices.
2) Apply an equity focus: An equity lens should be used to inform explicit decision making
3) Leverage opportunities strategically: Encouraging all countries to focus on a response to decentralization may not be the best use of UNICEF resources.
4) Fine-tune country programmes: It is important for Country Offices to fine-tune the country programmes to enhance their strategic and operational effectiveness keeping in mind the context of decentralization and an agreed upon framework. This will, among other things, require:(i) paying more attention to setting clear outcomes and targets, (ii) recalibrating staffing and staff capacities, (iii) addressing shortcomings in hiring, transitions, and leadership, (iv) relying on evidence, (v) addressing various risks, including risks of reform (centre-periphery conflict), fiduciary risks, and risks to quality of services, and (vi) and formulating appropriate roles for zonal offices.
5) Focus in middle income countries
6) Enhance staff knowledge: UNICEF staff should be better equipped to speak to the issues of decentralization, understand implications, advocate effectively using evidence, make strategic interventions where useful, capture funding opportunities where these can contribute to improving the lives of children. Being effective requires more than technical knowledge and sectoral expertise. It requires the ability to more broadly engage in public dialogue and contribute to public reasoning on a wide range of social and economic policies that affect children.
7) Undertake a detailed policy study on piloting: The evaluation suggests a relative lack of sophistication in the analysis of how pilots and ground level experiences actually lead to policy change. UNICEF would benefit by examining in more detail (and with rigour) especially the pilot-to-policy model in order to draw out a more nuanced understanding of these processes. UNICEF should consider doing a detailed policy study to draw on examples of policy influence (or lack of it) from successful and unsuccessful models in order to develop theories of change and a framework of elements that matter for influencing policy and to build deeper understanding and language around policy influence within UNICEF.
Process Oriented recommendations:
8) Involve the staff: UNICEF should revert back to the staff with the findings of this Report, and outline a process for engaging with and spelling out the concrete steps that are needed to address conclusions and findings of this evaluation.
9) Engage with findings: UNICEF management should lay out a process for engaging with and acting on the findings of this evaluation. The findings of this evaluation take UNICEF into deciding on strategy choices and priorities – something squarely in the realm of UNICEF leadership
Full report in PDF
PDF files require Acrobat Reader.