Evaluation in UNICEF ECARO
Core to UNICEF's values of accountability, integrity, trust, care and respect
- English
- русский
Evaluations are a key part of UNICEF's core commitments and values. By allowing UNICEF to identify and share findings about our successes, challenges and shortcomings, they ensure transparency, accountability, integrity and trust. And by enabling UNICEF to adjust our processes and programmes accordingly, they help us to improve our performance and results, ensuring the rights of every child and acting on our core values of care and respect for children and families.
Here are some recent evaluations UNICEF has undertaken along with key findings.
| Evaluation | Year | Sector | Country | Short Summary |
|---|---|---|---|---|
Multi-country evaluation of national childcare reforms with a focus on deinstitutionalization (DI) including Children with Disabilities (CwD) and difficult to place | 2024 | Child protection | Armenia Bulgaria Montenegro Georgia Serbia Moldova North Macedonia Tajikistan | UNICEF supports governments in developing systems that provide alternative care and community-based support to families. UNICEF also has consistently emphasized the need to refine deinstitutionalization (DI) reforms with a special focus on children with disabilities (CwD). Although many countries have made significant progress in developing their child protection systems and downsizing large institutions, studies indicate that many large institutions are being replaced with smaller residential units, leading to prolonged placement of children. Data shows a worrying trend of children with disabilities and 'difficult to place' children being 'trans-institutionalized' into small-scale residential care. This evaluation will interrogate DI progress in the region. Specifically, it will analyze any bottlenecks, exploring which potential solutions have been implemented and whether they have worked. |
Legacy Assessment of UNICEF Child Protection Activities within the Ukrainian Refugee Response | 2024 | Child protection | Poland Romania | This exercise identified activities from the outside Ukraine refugee response. It showed that response-building in a humanitarian context requires a targeted effort and that a preexisting presence in the country makes a difference, allowing UNICEF to use existing partnerships in building reputation and partnerships. While teams can leverage humanitarian work to expose the systemic bottlenecks, build partnerships and reputation, and advocate for sustainable change, meeting these goals requires a strategic perspective. |
UNICEF's Positioning at the Country Office Level | 2024 | Social policy | Armenia | The exercise provides important insights on how decision makers (around children policy) interact with each other in Georgia, Armenia, Kazakhstan, North Macedonia, and Bosnia-Herzegovina. The most vivid and noticeable finding of the exercise is that UNICEF was strongly positioned in all five countries, is an integral member of each national network, and plays significant roles in the flow of information in terms of child-related policy advocacy, policy design, and implementation on all positioning issues. |
Multi-Country Evaluation of Immunization Programming in the ECA Region | 2024 | Health | Kosovo (UN SC Resolution 1244) | The region is seeing decreasing rates of immunization coverage, noted disparities within and across countries, and an increase in vaccine hesitancy. This evaluation will inform the region’s strategic approach to addressing immunization needs at the national and sub-national levels. It also will interrogate the progress and any outcomes for each case-study country. Specifically, it will examine common bottlenecks, exploring potential solutions that have been implemented by national governments, including those conducted with UNICEF support, and analyzing why they were or were not successful. |
Real-time Evaluation of the UNICEF Response to Refugees from Ukraine
| 2023 | Emergency | Bulgaria Belarus Poland Moldova Romania Czech Republic Slovakia Hungary | The evaluation addressed UNICEF's response to the outflow of Ukrainian refugees. It revealed that UNICEF swiftly deployed its existing assets, professional expertise, and experience. Overall, UNICEF’s response largely met the Core Commitments to Children in Humanitarian Action. Children and their families were provided access to safe and appropriate reception, accommodation, and care. Additionally, children had timely access to child-friendly information on their rights, available services, and legal and administrative processes and solutions. However, shortcomings emerged related to the lack of due diligence, the fast turnover in surge deployments, the limited overview by the Regional Office of programme development in some countries, and the sustainability of programmes. |
Evaluation of Child Guarantee, Preparatory Stage, Phase III
| 2023 | Social Protection | Bulgaria Croatia Italy Greece | The European Child Guarantee (ECG) is a European Union initiative, endorsed by a Council Recommendation, which was developed to break the cycle of poverty and social exclusion experienced by disadvantaged children. The evaluation underlined the fact that the successful implementation of the ECG requires political engagement and willpower. UNICEF has the potential to ensure continued political commitment and coherence between the ECG recommendation and its implementation. |
Impact Evaluation of UNICEF’s Parenting App “BEBBO” | 2023 | Early Childhood Development | Serbia Bulgaria | BEBBO is the mobile application developed by UNICEF ECARO to support parents with timely guidance when direct contact with service providers is not possible or localized solutions are not easily available. The application aims to support the most vulnerable parents/caregivers. To evaluate this application, a randomized controlled trial was designed. While it did not find any impact, there are indications that it could have impact if targeted at specific populations. |
An Experimental Implementation Evaluation of U-Report on the Move | 2023 | Health | Italy | U-Report on the Move utilizes messaging platforms like Facebook Messenger and Telegram for anonymous, free support targeting young refugees and migrants. Users of U-Report also can seek support from the Here4U service, remote legal and psychosocial service for young migrants and refugees in Italy. The evaluation focused on the potential impact of U-Report on the stigma associated with seeking mental health and psychosocial support seeking behaviour amongst U-Reporters. Low survey completion rate limited the statistical power of the evaluation. However, there is considerable evidence showing that UReport on the Move's impact is limited by low engagement and retention rates from its users. |
Exploration of Social Media Listening Application in Real Time Assessment of UNICEF Response to COVID-19
| 2022 | Education, Social Protection | Albania Azerbaijan Bosnia & Herzegovina Montenegro North Macedonia Serbia Tajikistan Türkiye Uzbekistan | This report contributed to the ongoing Real-Time Assessment (RTA) of UNICEF's response to the COVID-19 pandemic in Eastern Europe and Central Asia (ECAR). Social media posts related to education and COVID-19 expanded over time and spiked with the waves of the pandemic. The analysis found that social policy discussions expanded in terms of the overall volume of comments. It also showed that Social Media Listening (SML) in the early stages of evaluations could be utilized as a formative or exploratory tool for informing the design and direction of the evaluation. |
Developmental Evaluation of LearnIn Initiative in ECAR
| 2022 | Education | Albania Montenegro Tajikistan | LearnIn is a regional digital learning initiative launched in April 2020 by the UNICEF Europe and Central Asia (ECA) Regional Office in response to the immediate pandemic-induced needs of ECA countries for remote learning. Evaluation results indicated that even though the regional conceptualization of LearnIn was largely adaptable, countries need more support in building national visions for using digital technologies to improve the quality of learning for all children. Also, the support for platform development was not well adapted to different countries' needs. The evaluation also underlined the importance of co-creation and a shared understanding. |
Evaluation of the UNICEF Project – Social Inclusion of Roma Children and Children with Disabilities in the Western Balkans and Moldova
| 2022 | Social Protection | Albania Bosnia and Herzegovina Kosovo (UN SC Resolution 1244) Moldova Montenegro North Macedonia Serbia | This project aimed to promote social inclusion in the context of early child development (ECD) policies and services, especially for boys and girls from ethnic minority groups such as Roma. The evaluation results concluded that the project addresses the evolving needs and aspirations of right holders and duty bearers. In particular, by creating and expanding networks of partners and helping with building skills of its counterparts, it contributed to making change. The modelling of improved ECD services and the preparation of policy and legal documents provided platforms the ability to concretely test the common vision of UNICEF and ECD stakeholders. |
Evaluation of the UNICEF Migrant and Refugee Child Health Project
| 2022 | Health, Emergency | Bosnia and Herzegovina Bulgaria Greece Italy Serbia | The 'Strengthening refugee and migrant children’s health status in Southern and South-Eastern Europe' (RMChild-Health Project) project focused on strengthening the health status of refugee and migrant children. The evaluation showed that while the project delivered some planned outputs, its impact was constrained by the short timelines. The project’s multisectoral approach and strong linkages, in particular mental/psychosocial protection and physical health services by building trust, was its key strength. The project also helped with the capacity building of frontline workers, which led to more resilient networks. |
Real Time Assessment of UNICEF education response to COVID-19 in selected countries | 2022 | Education | Azerbaijan Bosnia and Herzegovina Serbia Türkiye | This evaluation examines the education response of UNICEF to COVID-19 from the onset and through the crisis in order to inform future education planning and implementation. In the context of uncertainty caused by the pandemic, UNICEF COs successfully utilized pre-existing knowledge, funds, expertise and partnerships to contribute to the functioning and strengthening of the education sector. The evaluation underscored that pre-existing inequities and disparities in vulnerable populations must be taken into account during crisis planning. |
Multi-country evaluation of the UNICEF Early Childhood Development response to COVID-19 in Europe and Central Asia region | 2022 | Early Childhood Development | Croatia Georgia Moldova Ukraine | This evaluation was conducted to assess the key adaptations made in UNICEF’s Early Childhood Development programmes in the region during COVID-19. UNICEF-supported ECD-related services were well-suited to respond to COVID-19, while new modalities for service provision required by the pandemic were not equally relevant for all rightsholders. Challenges emerged regarding outreach to disadvantaged beneficiary groups and addressing their specific needs. Also, increased digitization of capacity-building activities and service provision were not equally relevant and effective for all groups, types of services and problems. |
ECAR Multi-Country Programme Evaluation Synthesis Report | 2022 | Cross-sectoral | Bulgaria Kyrgyzstan Moldova Ukraine | This evaluation focused on the interventions implemented in Bulgaria, Kyrgyzstan, Moldova and Ukraine between 2018 and 2021 under the Country Programme Documents. The evaluation found that Country Programmes were developed in line with UNICEF guidelines, based on in-depth research on the situation of children in each country, while putting great effort in evidence generation in regards to child rights. It also found that COs adjusted their work by focusing on the pressing needs of children in the COVID-19 crisis. Finally, the evaluation documented successful examples of scaled-up models. |
Real Time Assessment of UNICEF social protection response to COVID-19 in selected countries
| 2021 | Social Protection | Albania Montenegro North Macedonia Tajikistan Uzbekistan | This report provides a snapshot of UNICEF’s social protection responses during the pandemic. While UNICEF’s contribution to national social protection programming and implementation has been valuable, more could have been done if the funding targets had been met. Gender awareness, such as a focus on single-parent households, were visible, but largely overlooked in components focusing on children. UNICEF programmes and initiatives successfully included emergency cash transfers to the existing social protection systems. |
Real Time Assessment of UNICEF response to COVID-19 at the country level
| 2021 | Emergency | Croatia Georgia Kyrgyzstan Moldova Tajikistan Ukraine | This evaluation focused on UNICEF’s response to COVID-19, UNICEF’s capacity to adapt its response to the evolving context, lessons learnt and the way forward. The evaluation revealed that UNICEF COs could adapt their response to the countries’ dynamic needs. However, limited budgets inhibited their capacity to keep up with expanding needs. |
Methodological resources
The UNICEF ECARO Evaluation section engages in methodological methods to support evidence generation as well as to conduct innovative evaluation designs. Some of the methodological work that UNICEF ECARO has developed to guide evidence generation includes:
Using Experimental Evaluation Design of Digital Child-Related Solutions in ECA Region for UNICEF | This framework addresses the gaps in evidence for digital solutions in the international development area. It aims to provide a comprehensive conceptual and practical framework to use evaluations for the R&D of digital solutions. |
Introducing Evaluative Thinking into the Development of Digital Interventions | This guidance note supports programme and monitoring and evaluation practitioners who are working on ideation, development or replication of digital interventions in incorporating evaluative thinking to their intervention. |
Using MICS, MICS Plus and other Household Survey Data in Impact Evaluations of National/Subnational Programs and Models-to-Scale | These documents provide guidance with assessing when and how existing MICS, MICS Plus and other household survey data can be used in impact evaluations; and how MICS Plus can benefit impact evaluations as a cost-saving strategy. |
Methodology for Using Machine Learning Techniques to Follow Online Mental Health and Well-Being Discourse among Young People and Adolescents | This transferable process model will including participatory data gathering and technical tool selection and adjustments allowing the building of an analytical tool that will provide timely insights into adolescent mental health at a national scale. |
Technical Note on the inclusion of hard-to-reach populations and children in evaluations | Midway through implementation of Agenda 2030, there is an urgent need for evaluative tools and guidance that explicitly include hard-to-reach populations. Doing so is essential to ensure equity-focused policymaking, and to make sure that evaluations truly reflect the realities of all children – not just those who are easiest to reach. |
| Methodology to assess national preparedness to humanitarian shocks (ongoing) | This methodology will assess the likely demands on national systems due to expected humanitarian shocks and their capacity and willingness to absorb said demands by forecasting needs, simulating the government response and highlighting the areas of strategic investment. |
| Use of AI in Policy Design and Evaluations (ongoing) | This work aims to answer the “which policies works for whom?” question using Machine Learning. |
National Evaluation Capacity Development
In the context of rapid changes impacting the whole world, insights into what is working and what is not is crucial to deliver results for children. UNICEF’s mandate and the decentralized nature of its evaluation function necessitate strong field capacities which enable effective learning from programmes and using evaluations to continuously enhance outcomes for children. For this reason, UNICEF contributes to the National Evaluation Capacity Development to strengthen the capacity of key actors to carry out equity focused evaluation of national development policies and programmes.
UNICEF ECARO has facilitated the participation of 15 countries from the Europe and Central Asia region to Executive Course for Evaluation Leaders (ExCEL) with partnership of the National University of Singapore. While enhancing the skills and expertise of participants to manage and lead more robust evaluations and related initiatives, the course also cultivates an evidence-based culture and a greater appreciation of the role of evaluation to achieve sustainable results. Also, the results of the study conducted to understand the country led evaluations in the Central Asian context are being utilized to lead the national evaluation capacity development efforts in the region.
Intermediate Moderated Programme for Evaluation Systems’ Strengthening (IMPrESS), a 6-week facilitated online training designed for those commissioning, conducting and utilizing evaluation results for development, is also being conducted in the region to support National Evaluation Capacity Development efforts.
Comparative cross-country analysis of evaluation policies and systems in five Central Asian countries | This report presents the findings of a comparative cross-country analysis of evaluation policies and systems in five Central Asian (CA) countries. This is the first comprehensive assessment of national evaluation policies and systems for all five Central Asian countries: Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan and Uzbekistan. |
Partnerships for Evaluation Capacity Building
The UNICEF Europe and Central Asia Regional Office (ECARO) Evaluation Section recognizes the pivotal role of strategic partnerships in enhancing the capacities to deliver impactful outcomes for children and advancing sustainable development goals. By collaborating with leading institutions, we aim to promote rigorous evaluation methodologies, facilitate knowledge exchange, and build local capacities to ensure that evaluation findings effectively inform policy and practice for the betterment of children's lives across the region. Those efforts have been institutionalized with Memorandums of Understanding (MoU) with Koç University (Türkiye) and J-Pal Europe which are meant to use evaluations as research and development tools as well as looking at the impact of our work.