2017 Ukraine: Evaluation of the Country Programme of Cooperation between the Government of Ukraine and UNICEF 2012-2016
Author: Arkadi Toritsyn, David Gzirishvili, Cosette Maiky, Dessislava Ilieva, Natalia Mihailova
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.
The Country Programme (CP) between the Government of Ukraine and UNICEF (2012-2016) was developed in 2011 and had a range of objectives typical for a middle-income country with issues of health, HIV/AIDS, justice, and social exclusion coming to the fore. The interventions in the CP addressed main needs and rights of children and were limited in their scope and nature by focusing on a few specific areas that reflected UNICEF comparative advantages, Government and donors’ priorities.
Ukraine’s context has significantly shaped UNICEF CO work. Despite some progress made in advancing the rights to survival and development expressed in reduced child mortality, the registered HIV prevalence among pregnant women in Ukraine is one of the highest in Europe. Ukraine has experienced outbreaks of vaccine-preventable diseases in recent years as well. There are some regional inequities in health outcomes and neonatal, postneonatal, and infant and under-five mortality rates in rural areas are 50 to 100% higher than in urban areas.
Ukraine has a fairly well developed system of pre-school establishments but the rights to early learning are not fully provided to disadvantaged children who are unable to access and benefit from quality kindergarten programs. Often state institutions are overcrowded.
The significant advancements in the field of right to an inclusive quality education reflected in increased primary and secondary school net enrolment and decrease in the share of primary school age out-of-school children were reversed due to the conflict in the East of the country. Despite the improvements in policy and legislation, preventive social care services are underdeveloped so that high numbers of children are unnecessarily separated from their parents and go to state care. The number of children separated from their parents remains high and was 88,305 in 2015, which is 1.16% of the total child population of Ukraine. A large proportion of them constitute children with disabilities – 62,378. Families with children are susceptible to poverty. Almost every third family with children was poor in 2012.4 The experience of poverty compromises the realization of all children’s rights and limits their access to quality government services due to lack of resources to provide out-of-pocket payments.
Violence against children takes many forms, including domestic violence, sexual assault, and harsh disciplinary practices. It can inflict both physical harm and psychological damage on children. According to UNICEF, the prevalence of violence against children is high. In Ukraine, 61.2% of children aged 2-14 years were subjected to at least one form of psychological or physical punishment by an adult in the household in 2012. One percent of children were subjected to severe physical punishment. Moreover, around 11% households believe that it is fine to apply physical punishment to children.
The conflict in the East has created additional barriers to the fulfillment of children’s rights throughout Ukraine, especially in the conflict-affected areas. Despite the existence of a peace agreement (the Minsk Protocol of September 2014), the likely outcome of this conflict remains uncertain. Conflict-affected resident population and Internally Displaced Persons (IDPs) face shortages in food, health services, basic household items and shelter and suffer from psychological distress. The resilience of the conflict affected populations, including host communities, is steadily depleting. Access to essential vital services is extremely challenging. The humanitarian situation is serious with freedom of movement and humanitarian access restricted due to continued fighting, security measures and Government enforced access and movement measures to and from the Non-Government Controlled Areas (NGCA).
Humanitarian partners estimate that at least 3.7 million people have been affected in Ukraine both directly and indirectly. Of some 5.2 million people in the Donbas (area where the conflict took place), at least 3.9 million have been directly affected by the conflict.7 Out of this estimated number, the humanitarian community is most concerned about the plight of the 0.8 million people living in areas close to the ‘contact line’, and 2.7 million people living in NGCAs.8 As of September 2016, 3.1 million people require humanitarian assistance out of whom 2.5 million have been targeted by various actors.
The conflict has severely weakened the formal and community based child protective environment - most staff have fled from these areas or were overwhelmed by the increased workload. 800,000 people, including 100,000 children living near the frontline are at daily risk of shelling, mines and unexploded ordnances. Due to the interconnection of water supply, central heating systems, electricity supply and sanitation needs, around 2.9 million people are at risk of suffering various levels of disruption of services due to the conflict-related systems damages or breakdown.
The Mid-Term Review of the Country Programme was carried in 2014. The CP was extended at the Government request to 2017. To inform the new Country Programme development (2018-2022), UNICEF commissioned the independent CP evaluation that was completed in May-December 2016. The evaluation covers most of the programme cycle implementation period, from commencement in 2012 to 2016.
The intended audience of this evaluation includes the Government of Ukraine, regional and local authorities, donors, UN sister agencies, development partners, non-governmental organizations, UNICEF Country Office, Regional Office and the Headquarters.
The evaluation’s specific objectives are to:
- Assess the relevance, effectiveness, efficiency, sustainability, coherence, coverage and, to the highest extent possible, impact of the strategies adopted to achieve the Country Programme’s results.
- Identify and document lessons learned in relation to type, combination and way of implementation of the strategies, considering the country context and UNICEF’s comparative advantage.
- Provide recommendations to guide/inform UNICEF strategic planning in Ukraine for the next program cycle, based on the lessons learned and successes of the current CP.
The scope of the evaluation focuses on the following key CP components: social policy and child protection, health, humanitarian response, water, HIV/AIDS and education.
Including the team leader, the evaluation team included five experts in social policy, early intervention and education; health/HIV/immunization; humanitarian action, water in emergencies; and child protection. The evaluation was conducted from May to December 2016, with 18 day mission to Ukraine.
The evaluation is based on CO’s Theories of Change (TOCs) but the ET also examined the CP to assess the results achievement progress. Relevance, effectiveness, efficiency, impact, sustainability, partnership, and coherence of the programme as a whole and within the areas of experts’ focus were examined. Given time and other limitations, in some sectors the ET could not conduct an in-depth evaluation of all interventions that fall under specific areas of expertise of team members. In consultations with the CO, a number of interventions was selectively sampled from each sector/cross-theme to reflect diversity of UNICEF work, with particular focus on diverse vulnerable groups supported.
The ET utilized a diverse range of data collection methods that included stakeholder mapping, systematic documentary review, mapping of available contextual analyses, data analysis, including national statistical data and UNICEF M&E systems data, technical analysis and testing of theories of change/strategies, round tables, and financial analysis. The ET conducted 79 semi-structured interviews with a diverse range of beneficiaries and partners, including Government officials, donors, UN sister agencies and other international organizations, NGOs, etc.
The evaluation followed the UNEG Norms and Standards as well as the UNEG Ethical Guidelines for Evaluation. All the necessary measures were undertaken to ensure objectivity and independence of evaluation.
There is a number of limitations to evaluability that the ET faced and had to address. Limits to the scope of the evaluation include resourcing and timing restrictions, which limited the ability to conduct full assessment across all strategy areas. Another limitation is a high degree of ambiguity in the formulation of the logical connection linking interventions, activities, outputs and outcomes in the CP. Also, as per the TOR, this evaluation did not seek to undertake detailed institutional analysis of the UNICEF Ukraine CO, namely its staffing, management structures, etc. Due to these constraints, this evaluation report is not fully comprehensive. The ET has developed a comprehensive list of mitigation strategies to address these limitations.
Findings and Conclusions:
Overall, the CP is very relevant in relation to the human rights conventions and treaties that have been signed and ratified by the Government of Ukraine. The CP is aligned with national policies and priorities and relevant to right holders’ and duty bearers’ needs. Specific UNICEF interventions were designed and implemented to address the needs of targeted beneficiary groups as identified in the CP and in line with some research and analysis conducted by UNICEF.
UNICEF ensured relevance of its work through continuous alignment of its support with the national development and humanitarian objectives, especially in responding promptly to the conflict in the East of Ukraine. Many stakeholders confirmed relevance of UNICEF support and praised especially its fast and comprehensive support in addressing the needs of children and other vulnerable groups affected by the conflict. UNICEF implemented significant internal management changes to effectively respond to multiple and diverse challenges arising from the conflict and insecurity in the East.
The majority of outcomes and outputs are most likely to be achieved by the end of the CP. Many elements of the CP were verifiably effective in terms of the achievement of most planned outputs, and their plausible contribution to the expected outcomes, and to progress towards country priorities. In some areas UNICEF will not be able to affect national target indicators as the nature and magnitude of interventions target only small groups of beneficiaries in pilot regions.
This evaluation has not found major concerns regarding efficiency. Overall, the ET concludes that UNICEF was able to achieve results in an economic manner and with manageable transaction costs. There was near consensus amongst partners from Government and donors that CP results had been achieved with little waste and duplication. Moreover, partners were keen to emphasize the flexibility and adaptability of UNICEF both during planning and implementation, in particular for the emergency component.
Sustainability objectives were not systematically addressed in the CP design and through its implementation. The bottlenecks analysis helped UNICEF clusters to collect the necessary evidence and conduct the needed analysis to identify priorities for their interventions, but it was not always clear how the pilots and very narrowly targeted interventions addressing diverse bottlenecks can make systemic impact and ensure sustainability.
The CP is under implementation and its impacts cannot be fully measured. In some areas the impact cannot be expected given the scale of UNICEF interventions. In some areas such as standards for social services, it can be expected that they will improve consistency of services in realities of decentralization. Adoption of a new Prevention of mother to child transmission protocol advocated by UNICEF contributed to the substantial reduction in mother to child transmission rate, that eventually decreased HIV related morbidity and mortality in children. The following table presents selected diverse areas of CP impact.
- Re-assess bottlenecks, identify vulnerable groups and improve quality and availability of disaggregated data to support UNICEF program planning and monitoring. Quality and reliable data are critically important to identify vulnerable groups, assess equity aspects and monitor whether the actions implemented improve children and other vulnerable groups’ rights. Ukraine’s statistical system is unable to produce quality and timely data to inform UNICEF work. Data on children must be comprehensive, reliable and sufficiently disaggregated to enable the identification of discrimination and/or disparities in the realization of children’s rights. UNICEF can work closely with the State Statistics Service and development partners to ensure availability of quality, accurate, verifiable, and disaggregated data.
- Ensure that decentralization advances and not harms children’s and vulnerable groups’ rights. UNICEF, together with other partners, should support the Government of Ukraine in building fiscally responsible, responsive, and accountable sub national governments that will, under the oversight of citizens and higher-level government, improve service delivery and governance.
- Shift focus towards policy advocacy, with building broad pro-reform coalitions. The realities of permanent low intensity conflict in the East of the country, political instability, inconsistent and insufficient Government’s commitment and ability to advance systemic reforms, economic hardships and general public support of country’s European choice, create political openings for UNICEF to advance policy reforms advancing rights of children. UNICEF has to identify a few areas of its mandate where success can be realistically achieved within the next five years such as deinstitutionalization and build a strong multi-year policy and advocacy strategy.
The ET identifies the following lessons learned:
- High quality data is essential for UNICEF priority setting, monitoring and achieving results. UNICEF should approach data collection, analysis and its use more systematically to maximize the relevance and impact of its interventions. Continuous monitoring of children’s rights across all areas of UNICEF’s corporate responsibilities ensures timely needs and barriers identification and development of interventions addressing the most urgent bottlenecks. In addition to collecting the data, it is important to create demand for this data and build capacity of UNICEF staff and national partners to use children-specific data in policy/program planning and monitoring. Production of high quality data reports and analysis do not necessarily result in better Government policies and UNICEF interventions. UNICEF staff and Government counterparts should understand why the data is being collected and how it should be used by them in policy and program planning and monitoring.
- When the conflict or any emergency occurs, UNICEF can benefit from the initiation of a stocktaking/reflective exercise for strategic and operational priorities setting. Ukraine CO has undertaken the mid-term CP review that helped it to better strategize, prioritize and properly integrate emergency programming into the CP. The CP was revised and became better targeted and results-focused. Additional areas specific to emergency environment such as support of conflict-affected children in five affected oblasts were added with the corresponding strategies, performance frameworks and indicators.
- UNICEF can achieve systemic changes by focusing on a limited number of priorities. Achieving strategic and long lasting impact is challenging if UNICEF pursues multiple objectives and is spread too thin. In conditions of conflict, political instability and frequent changes of ministers and regional authorities, the demand for UNICEF’s new and long-running activities is coming from national and regional partners, UNICEF HQ and RO. In setting its priorities, UNICEF should keep in mind that systemic changes can only occur if UNICEF focuses on a few areas where it can pursue simultaneously legislative and policy changes, deliver programs on the ground, support a broad pool of champions of reforms and conduct media campaigns to affect broader changes in societal view and behaviours.
- In Middle Income Countries that experienced the conflict or emergency, UNICEF should maintain a proper balance between emergency-specific and regular development interventions. Even when humanitarian needs are significant, UNICEF is not advised to focus solely on emergency supports/commodities delivery. It has to continue providing expertise and policy advice/advocacy so that when the need in humanitarian interventions decreases, UNICEF is better positioned to shift towards more systemic interventions. Emergency response should be as fully mainstreamed into UNICEF country regular operations as possible.
Full report in PDF
PDF files require Acrobat Reader.