We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 CEE/CIS: Progress in Reducing Health System Bottlenecks Towards Achieving the MDG 4: Evaluation of UNICEF's Contribution in Five CEE/CIS Countries



Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

Advancing health gains and reducing child mortality in the Central and Eastern Europe/Commonwealth of Independent States (CEE/CIS) countries require better functioning health systems capable of delivering equitable and quality health services for mothers and children.  The identification of barriers and bottlenecks arising at the level of the enabling enviornment, or with respect to supply, quality  and/or demand, and their timely removal or reduction, are pre-requisites for implementing effective public health interventions. Since the 1990s UNICEF has been assisting CEE/CIS countries to overcome these barriers, to improve the performance of health systems, and assure effective coverage with Maternal, Newborn and Child Health (MNCH) services towards the achievement of the Millennium Development Goals (MDGs).  At the beginning of new millennium, in light of the changing social-economic situation in these countries, UNICEF programming in the CEE/CIS changed from an emergency service delivery mode to a more up-stream health systems approach, aligning itself with global efforts to modernize health systems and improve the quality of service delivery.

Purpose/Objective:

The evaluation objectives[1] are as follows:

  • To document results in terms of changes in access to MNCH services (enhanced coverage of children with proven health services packages and interventions) and reduction of equity gaps; 
  • To assess how system-level changes (enabling environment, supply and demand, and quality of MNCH services) led to these results; and
  • To document the contribution of UNICEF in addressing health system level bottlenecks.

[1]     Following consultations with the UNICEF RO, the evaluation objectives were slightly modified from those presented in the MCE ToR during the inception phase of the evaluation.

Methodology:

Chapter 3 – Evaluation Methodology summarizes the final methodological approach used in the evaluation, including data collection methods, sources, and the data triangulation and analysis process. It also presents the quality assurance process, evaluation limitations, stakeholder involvement and participation in the evaluation, as well as ethical considerations.

Findings and Conclusions:

Impact: There has been a positive change in the reduction of infant and under-5 mortality and morbidity over the period 2000 to 2012 in the countries evaluated. However, the trends in key child health indicators across geographical, ethnic, gender and other socio-economic stratifiers in the CEE/CIS and the evaluation countries were uneven, and some groups remain outliers.

Relevance: UNICEF-supported programmes were well aligned with national development and sectoral priorities in all the evaluation countries, including those upholding fundamental human rights and tackling inequities. UNICEF-supported programmes invariably addressed the most important causes of infant and under-5 morbidity and mortality in all the evaluation countries, with the exception of causes related to the preconception period. Mortality and morbidity causes originating in the antenatal period were also less addressed. UNICEF identified and attempted to address all of the most important bottlenecks in effective coverage by MNCH services. UNICEF-supported programmes were mostly successful in identifying and applying the right interventions (activities), with the appropriate scope, target groups and scale to address these health system bottlenecks.

Effectiveness and Efficiency: Although it was not always the highest contributor in monetary terms, UNICEF invariably played an active, if not a lead, role in all the evaluation countries within most of the partnerships forged to promote MNCH issues and define wider health sector policies. UNICEF applied considerable efforts to involve relevant partners in programme design, implementation and evaluation. However, the representation of beneficiaries in this process was relatively small, limiting the potential for realising the human rights-based approach through the participation of rights holders.

UNICEF-supported programmes most likely made a significant contribution to achieving the required system and community level changes in the countries evaluated. Modelling, enabling knowledge exchange, child rights monitoring & evaluation and policy advice & technical assistance were the core roles most frequently used and/or the most resourced and carried out.

Equity: Results were mixed in the case of reducing the equity gaps in MNCH service coverage. Some gaps narrowed but others widened. Subsequently, there were significant inter- and intra-country disparities, and some marginalised groups remained outside of effective coverage by critical MNCH interventions. With a few exceptions (e.g. Roma in Serbia and Moldova), MNCH programmes supported by UNICEF largely failed to define marginalised, vulnerable and hard-to-reach groups clearly for programmatic purposes, or to focus interventions towards these groups. Gender equality was not mainstreamed in UNICEF programming up until the very end of the evaluation period, however UNICEF did monitor the equity effects of its interventions using population-based surveys and routine statistics.

Sustainability: Most UNICEF-supported programmes were integrated into national policies and budgets, however UNICEF was somewhat more successful in assuring programme integration into national policies than into national budgets. They succeeded in assuring the scale-up of pilots and their inclusion in national policies and/or systems by progressively and systematically applying UNICEF core roles from modelling/piloting to knowledge exchange and to policy advice and technical assistance. 

The ToC proved to be a useful tool both for creating and evaluating UNICEF-supported programmes. However, when analysing the data on programme interventions provided by the UNICEF Country Offices (CO) from the evaluation countries, the ET found a wide variation among COs in in how they attributed UNICEF-supported programme interventions to the relevant core roles (for example, capacity building) and the MoRES determinants they target.

Recommendations:

The evaluation recommendations draw on the findings, conclusions and lessons learned, were developed in a participatory manner and were validated with the RKLA Reference Group. The recommendations are grouped in five major recommendations with a number of concrete suggesions that can be found in the full evaluation report:

  1. Sharpen equity-focus of UNICEF’s programming
  2. Consolidate and advance the gains in child health
  3. Where possible, consider not sufficiently addressed underlying causes of child mortality and morbidity
  4. Address persisting bottlenecks at health system and community levels
  5. General Recommendations on the TOC, country office guidance needed and planning and financing systems.

Lessons Learned:

Factors explaining success:

Evaluation findings on the effectiveness and efficiency of the UNICEF-supported programmes indicate that while the size of financial allocations for UNICEF-supported programmes may influence the achievement of results, they are not always a determining factor. These results were largely obtained by enlisting the sustained engagement of governments and development partners who brought substantial financial resources to the table.

UNICEF’s skills and its approach to health system-level changes may have been critical factors for success as well. Their approach included the concept that health systems are complex-adaptive systems to which a linear “input - output – outcome” perspective is rarely applicable.

What did not work well:

Despite the existence of toolkits for health system performance and for strengthening assessments, they were not consistently applied and/or institutionalised to rigorously monitor system-level changes in any of the evaluation countries.  Systematic efforts should be applied to develop UNICEF and partner governments’ capacities to assess health system performance.

UNICEF was not always successful when addressing health financing and health service delivery system bottlenecks independently or--in certain cases-- even with their development partners who are traditionally active in health reforms (World Bank, USAID, EU). This was mainly true for the lower income countries evaluated--Uzbekistan and Kyrgyzstan. The case studies show that in these countries several factors negatively affect the flexibility of health systems and their capacity to change, for example limited government spending on health, low staff motivation and frequent turnover of public health managers. Global evidence suggests that these challenges are pervasive in many LMICs and that more sustainable approaches towards HSS must be created.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2015

Region:
CEE/CIS

Theme:
Health

Type:
Evaluation

Language:
English

New enhanced search