2015 Uzbekistan: Evaluation of “Improvement of Mother and Child Health Services” Phase 2
Author: Ivdity Chikovani, MD; Maya Kherkheulidze MD.,PhD; Konstantin Osipov
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.
The process of health care reforms in Uzbekistan largely benefited from the external technical and financial assistance received from multi and bilateral donors. In particular, from 2008-present, EU funding was made available in support of MCH sector reforms in two phases. To address MCH service bottlenecks, the Ministry of Health (MoH), UNICEF and the EU initiated the “Improvement of Mother and Child Health Service - Phase 1” (IMCHS) project, which was funded by the EU from 2008 to 2011. The IMCHS project Phase 1 mostly concentrated on piloting the application of the WHO Live Birth Definition, together with strengthening newborn care and improving the quality of maternal and child health care by developing the skills and capacity in 8 selected regions of the country. The second phase of the IMCHS project aimed at supporting the MoH in capitalizing on the achievements of Phase 1, enhancing and expanding the coverage and quality of MCH services nationwide, filling systematic gaps related to the MCH health sector and the low level of population knowledge about responsive parenting.
The IMCHS Phase 2 project was planned for the period of 25 July 2012 – 25 January 2016. The specific objective of the project is to support the MoH to increase the quality of mother and child health care services, and to increase the capacity of families to make informed choices about health and nutrition. The project plans to achieve two key results: 1) Institutional strengthening and 2) Capacity development and empowerment. In pursuit of the first result, the skills and operational capacity of the Ministry of Health and its three tiers (primary, secondary, and tertiary) have been enhanced to effectively support the health reform process concerning MCH services, in accordance with international standards. To achieve the second result, a comprehensive education-training process dealing with MCH was developed and put in place at both a medical-institutional and family level.
Evaluation Rationale: This summative evaluation comes at a time when the IMCHS project phase II is concluding its activities, and the Country Program of Cooperation between the Government of Uzbekistan (GoU) and UNICEF is finalizing the 2010-2015 cycle and is embarking on implementation of a new 2016 – 2020 Country Program (CP). In this context, the evaluation offers the opportunity to critically assess the IMCHS project phase II as a stand-alone project as well as in the broader context of scale up cost-effective interventions related to maternal, newborn and child health care and their contribution to maternal and child health and well-being.
Evaluation Objectives: The main objectives of the evaluation are: i) Assess the relevance, efficiency, effectiveness, sustainability, coherence and to the extent possible, impact of the project results; ii) Identify and document success, challenges and lessons learned; iii) Provide recommendations to guide implementation of the next programme cycle and policy level decision making by relevant stakeholders and international development agencies.
Evaluation Scope: The evaluation covers most of the 2nd phase implementation period from July 2012 to December 2015 and covers six regions of the country where the ''Improvement of Mother and Child Health Services, Phase II'' (IMCHS Phase II) programme is implemented.
Evaluation Target Audience & Benefits: The findings of the evaluation will be used by the Government of Uzbekistan (GoU) for further improvement of maternal and child health services. The Government of Uzbekistan and UNICEF will utilize the knowledge generated by the evaluation to inform the strategies to be applied in the new Country Program 2016-2020. The findings and recommendations will primarily be addressed with policy makers and project managers, both internally in UNICEF and externally in government, partner organizations and academia.
The evaluation is summative in nature and considers the five standard OECD criteria (relevance, effectiveness, efficiency, sustainability and impact) and adds the criteria related to human rights based approach and coherence. The overall evaluation approach is based on the project Theory of Change (TOC). To achieve evaluation objectives, the evaluation framework (EF) has been developed. The EF identifies indicators for each question that has been measured and/or assessed during the evaluation. It also identifies the sources of information, evaluation methods applied, the range of documents reviewed and key informants being interviewed for each question. Apart from the EF the Project Results Framework (RF) has been used to demonstrate how programme activities eventually succeeded in achieving its objectives. The evaluation methodology comprised a mix of site visits and observations, face-to-face semi structured interviews, group interviews, desk-based review of existing reports, documents and available secondary data. All data collected during the evaluation were analyzed using NVivo 10™ software. The evaluation was carried out by an ET comprised of two international evaluators and one national consultant. During the evaluation process the ET ensured the impartiality and independence at all stages of the evaluation process, which contributes to the credibility of the evaluation and the avoidance of bias in findings, analyses and conclusions. Statement of facts is methodically clearly distinguished from opinions; different perspectives were taken into account, as well as strengths and weaknesses; and outcomes/outputs, conclusions and recommendations are supported by evidence and are comprehensible. The ET also considered ethical issues and in compliance with UNICEF guidelines, the ET submitted the evaluation protocol to the UNICEF CO established “Ethics Review Panel” and obtained clearance prior to the initiation of the work.
Findings and Conclusions:
Summary of key findings:
- The IMCHS Phase 2 Project demonstrates high relevance to the country’s MCH context.
- The project was effective in reaching most of the stated outcomes, and it is likely that the project contributed towards the decline in maternal and child mortality rates in the country.
- The project interventions have been effective in facilitating MCH sector reforms with respect to policy change and quality improvement.
- MCH Coordination Council is the first such kind of coordination mechanism in the MCH sector of Republic of Uzbekistan, created by the project, which made an essential contribution to improving coordination between the MoH, other government agencies and international partners in the area of MCH.
- The human resource capacity built by the project, together with international knowledge sharing, contributed to improve service providers’ knowledge and practices in Newborn and Child Survival Packages.
- Furthermore, the project has placed a special emphasis on empowering families to take better care of their children, including providing them with a better understanding of the importance of breastfeeding, nutrition, vaccination, hygiene and child development stimulation.
- The robust M&E system (including supportive supervision) established and applied by the project was effective in reinforcing skills application and tracking progress at the national, sub-national and facility levels.
- The high degree of MCH reform coordination attained at national level through the establishment of MCH Coordination Council as a platform for policy discussions, coordination of the intervention and experience exchange, along with the coherence of strategies and interventions of key partners, ensured the achievement of project outcomes
And other reported strengths and weaknesses.
This Section provides key recommendations based on the findings of the evaluation, highlights priority issues affecting effectiveness of quality MCH services and health status of children and women in the country, and suggests possible strategic interventions for the Government and its development partners.
Recommendation #1: Continue further enhancement of the Government’s stewardship role
Recommendation #2: Enhance managerial capacity, especially at subnational level on evidence-based planning, budgeting and monitoring.
Recommendation #3: Allocation of adequate funding for implementation of modern maternal and child health approaches.
Recommendation #4: Ensure an adequate supply of required medical equipment, medicines and consumables.
Recommendation #5: Continue the institutionalization of modern medical MCH technologies and approaches in under-graduate and postgraduate education countrywide.
Recommendation #6: Ensure effective operation of new elements of the information systems
Recommendation #7: Ensure the institutionalization and sustainment of community behavioral change interventions.
Recommendation #8: Focus on targeting the most marginalized and vulnerable.
Recommendation #9: Continue support to GoU’s MCH reform Implementation
Recommendation #10: Strengthen inter-agency collaboration
Recommendation #11: Enhance advocacy
Lesson 1: Setting the scene - The IMCHS Phase 2 project is a good example of building project design on evidence based approaches, addressing the needs of direct beneficiaries, planning for research and practice. The project/programme, which is tailored to local needs and is based on international best practices, is a prerequisite for obtaining the buy in of governmental and nongovernmental stakeholders in any country
Lesson 2: Partnerships are vital in making a difference - UNICEF has high levels of international expertise, but limited resources. Other developing partners, including international donors, tend to have greater resources but need specific expertise in shaping their programs to benefit children. Coordination with the government and with development partners is essential in ensuring common messages are given out and resources are not wasted in overlapping efforts.
Lesson 3: Participatory monitoring - A participatory monitoring approach (JMT) was deployed in the project by involving national and local experts, professionals and development partners. These experts have been involved in monitoring missions to the health facilities by the project. This approach to M&E resulted in a process of individual and collective learning and capacity development, among others.
Lesson 4: Supportive supervision vs. one time training - The series of evaluations of health human resource capacity building interventions conducted in the CEE/CIS region revealed that one time training is insufficient to achieve the desired capacity in health facilities, which instead requires periodic refresher trainings, although that is constrained by scarcity of funds for such activities. The IMCHS Phase 2 project was innovative in addressing this challenge. Specifically, instead of funding refresher trainings for a couple of days, the project shifted from training towards a supportive supervision approach.
Terms of Reference
Full report in PDF
PDF files require Acrobat Reader.