2016 Republic of Kyrgyzstan: Evaluation of UNICEF Country Program (2012-2016) and Strategic Positioning
Author: Tamar Gotsadze, MD., PhD; Kateryna Shalyeva, PhD
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as 'Part 3'.
Kyrgyzstan is a mountainous country located between China, Kazakhstan, Uzbekistan and Tajikistan in Central Asia. Formerly part of the Soviet Union, it became independent in 1991. Since its independence, the Kyrgyz Republic has had a turbulent political history with two presidents overthrown by the public, most recently in April 2010 when a deadly clash broke out. In June 2010, persistent social tensions that had been on the rise in the south of the country, where large ethnic Kyrgyz and Uzbek communities live side by side, and which had been aggravated by the political instability, climaxed in violent inter-ethnic clashes
The Kyrgyz Republic is the only country in central Asia with a parliamentary democracy, albeit not firmly established, and with a burden of problems inherited and emerging issues ranging from respect for human rights to pervasive corruption, leaving the country in an unstable situation. With the gross national income (GNI) per capita for 2013 estimated at US$ 1,200, increasing from US$ 1,040 in 2012, the Kyrgyz Republic has just recently been re-classified as lower middle-income country.
Country Programme of Cooperation between the Government of the Kyrgyz Republic and UNICEF (2012-2016) was signed in January 2012. The Mid-Term Review (MTR) of the Country Programme was carried out between March 2014 – May 2015. The Situation Analysis of the Rights of Children in Kyrgyzstan was commissioned in 2014 in order to bridge information gaps.
The MTR encompassed the analysis of the Country Programme through the prism of the thematic Theories of Change (TOCs), analyses of recent data, reports, studies and evaluations, and sectoral consultations with the engagement of the government counterparts, civil society organisations and international development partners.
The purpose of the evaluation is to assess the key strategies used to implement UNICEF Kyrgyzstan Country Programme of Cooperation 2012-2016, especially approaches recommended during the MTR.
The Country Programme aims to contribute to achieving four Outcomes and sixteen related Outputs focused on programmatic areas.
The evaluation is intended to answer whether and how the key strategies have contributed to better position UNICEF in the national development agenda of Kyrgyzstan, and whether and how these key strategies can accelerate and strengthen the achievement of better results beyond the sum of the sectoral results delivered by the programmes that are supported by UNICEF.
While approaching to the end of the programme cycle and starting the planning phase for the new one, the knowledge generated by the evaluation will be used by the GOK and UNICEF to inform the new Country Programme Document 2018-2022. The findings will be used as a basis for discussions and planning exercises between UNICEF and national actors.
The evaluation assessed selected dimensions of relevance, effectiveness, efficiency and sustainability of the Kyrgyzstan CO’s programme strategies, as put in place since the MTR. It will also identify good practices to be taken forward, and develop recommendations on how to further strengthen the strategic positioning (and performance?) of UNICEF in Kyrgyzstan, with a view to contributing to the achievement of best possible development results for children.
Six evaluation criteria were selected according to the CP evaluation TOR. The first five represent OECD/DAC evaluation criteria, and the last, strategic positioning, was chosen at the request of UNICEF CO. The analysis of UNICEF strategic positioning looked at strategic alignment both corporate and systemic, responsiveness and added value. HRBA analysis examined application of HRBA approach during situation analysis, program design and planning, implementation and monitoring and evaluation.
The ET created an Evaluation Framework (EF) that links the evaluation criteria and questions per each criterion to the chosen indicators, data sources and methodologies.
Apart from the EF the CPAP Results Framework (RF) has been used to demonstrate how programme activities eventually succeeded in achieving its objectives. ET collected and carefully assessed the data provided in the CPAP RF (prior to MTR of the CP and after MTR) and analysed the Programme information database and different monitoring and study reports. Where possible, these data have been used to establish causal relationships between outcomes and interventions and/or unintended consequences.
The evaluation was carried out in three phases by two international consultants between April – August, 2016. Twenty-one days’ field mission to Kyrgyzstan was conducted in May (14 days) and July (7 days) 2016, with visits to Osh and Issyk-Kul oblasts.
Data collection methodology comprised a mix of site visits and observations, face-to-face in-depth interviews, desk-based research and review of existing reports, documents and available secondary data. In addition, the ET used quantitative data collection methodology to collect information on partners’ engagement in the CP/Programme design, implementation and M&E as well as learn about their opinions on the potential future role of UNICEF. Overall 103 Key Informants have been interviewed.
Findings and Conclusions:
Relevance: Overall, the CP is well aligned with international normative frameworks, national policies and priorities. It has proven substantive relevance to right holders’ and duty bearer’s needs. Despite changes of operating context, moving from emergency to post-emergency face, over the course of CP implementation, all strategies applied remained valid and appropriate to attain originally planned outcomes and outputs stipulated in CPAP. Associated CP outcome and output indicators are relevant and aligned with the needs of national stakeholders, especially the most vulnerable groups. The CP is substantively well justified in relation to needs and interests of rights-holders so do interventions on a sectoral level. It also demonstrates relevance in geographical targeting by focusing interventions in the most vulnerable municipalities.
However, operational relevance has been compromised by a lack of clear intervention logic at the onset of the Programme. TOCs developed in the course of MTR, will remedy shortcomings and guides CP implementation till the end of its cycle.
Effectiveness: The majority of outcomes and outputs are most likely to be achieved by the end of the Country Programme. At a time of evaluation Majority of outcome and output indicators were already fully met and some is most likely to be achieved.
Efficiency: The highest proportion of available resources was allocated to ensure equitable access to quality services. The evaluation findings on the effectiveness and efficiency of UNICEF-supported programmes indicate that size of financial allocations are not always a conclusive factor. Instead, targeting and addressing most (“the bulk”) of the identified bottlenecks; using a balanced mix of UNICEF’s core roles.
Sustainability: Sustainability principle appears to have been given scant consideration in CP design. At implementation level, some clear efforts have been made to build common ground for sustainable results.
- Recommendation 1: A well-defined theory of change and framework of results of the new Country Program would inform decisions on which programmes to pursue, prioritize investment of available resources and also how to assess progress and identify needed changes.
- Recommendation 2: The new Country Program has more to focus on setting up the systems that prevent various vulnerabilities, rather to systems generating reactive responses.
- Recommendation 3: In its upcoming programming, the UNICEF CO should increase the focus on sustainability and responsible transition.
- Recommendation 4: New CP should address unfinished agenda to ensure effective coverage of right holders and sustainable results.
- Recommendation 5: Prioritization of interventions directed towards yet unattended and less attended areas of child rights violations with consideration of highest impact possible given UNICEF’s comparative advantage, expertise and resources.
- Recommendation 6: Enhancement of a human rights-based approach to programming, equity and gender mainstreaming.
- Recommendation 7: Further emphasize support for integrated programming.
- Recommendation 8: Continue its long-standing practice of building capacity through partnerships with national and local governments, civil society, academic institutions and the private sector will ensure reducing the dependence of governments and other actors on development assistance over time.
- Recommendation 9: Expanding public advocacy and C4D in the next CP is needed to strengthen UNICEF’s strategic positioning
- Recommendation 10: Strengthening UNICEF CO capacity in results-based management
- Recommendation 11: Enhance CO staff capacity - With the recommended shift of the new CP from service delivery more to policy and systems’ strengthening approach and enhancement of strategic partnerships, the internal capacity building of UNICEF CO staff should be emphasized
Lesson 1: Setting the scene - The CP is a good example of building intervention design on evidence based approaches, addressing the needs of direct beneficiaries, planning for research and practice. Programmatic intervention design, in the second half of CP cycle, was largely informed by the findings of the various needs assessments and research. These findings have been used for advocacy purposes and set the scene for effective design and implementation of the interventions per each KRA.
Lesson 2: Linking national and sub-national goals - In the Equity Programme, there has been persistent advocacy to link municipality-level work and national-level change. This is an important aspect of achieving greater impact and sustainability of results. Nevertheless, linking national, sub-national and local level goals will allow maximizing desired results
Lesson 3: Approach to system-level changes - UNICEF’s skills and its approach to system-level changes in support of access to quality services may have been critical factors for success, albeit approach based on the concept that systems are complex-adaptive systems to which a linear “input - output – outcome” perspective is rarely applicable.
Lesson 4: Complementary activities between programmes - Development gains depend on comprehensive and methodical approaches. While it is rarely possible to capture all aspects of a development challenge in one programme, activities from different programme should be complementary.
Lesson 5: Partnerships are vital in making a difference - UNICEF has high levels of international expertise, but limited resources. Other development partners, including international donors, tend to have greater resources but need specific expertise in shaping their programs to benefit children. UNICEF thus has a crucial contribution to make in providing expert input into shaping national policies.
Full report in PDF
PDF files require Acrobat Reader.