2013 Kyrgyzstan: Evaluation of UNICEF Programme on Perinatal Care for the period 2010 – 2013
Author: Mrs. Tamar Gotsadze, MD, PHD and Mrs. Chiara Zanetti, MD, MPH
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.
This report is the product of a summative evaluation of the Dao Project Ensuring Access to Affordable Health Services for Women of Reproductive Age and Newborns in the Affected Areas of the Kyrgyzstan Republic. The external evaluation has been commissioned by UNICEF Kyrgyzstan Country Office and was a joint and participatory process involving key stakeholders working within the MCH sector.
Since 2012, Kyrgyzstan is on-track to achieve Millennium Development Goal (MDG) 4 on Reducing Child Mortality as infant and under-five mortality rates have steadily decreased. According to 2012 data, under-five mortality rate is at 27 and infant mortality at 24 per 1,000 live births. Neonatal deaths comprise 52% of all under-5 deaths at the rate of 14 per 1,000 live births. Though 97% of pregnant women receive antenatal care and give birth under attendance of skilled health care professional, high maternal mortality rates (MMR) have been observed in Kyrgyzstan for many years.
In 2010, MMR was at 49.7 per 100,000 live births indicating that reaching the target of MDG 5 seems very unlikely. Therefore, the country has focused on maternal mortality as part of the MDG Acceleration Framework. In addition, looking beyond the MDGs and 2015, the Government of the Kyrgyz Republic has pledged A Promise Renewed initiative, a global movement to accelerate efforts to improve maternal, newborn and child survival, and has committed to work towards a new target -decreasing under five mortality rate to 20 or fewer deaths per 1,000 live births by 2035.
The maternal and newborn health care system in the country is developed with adequate services in terms of the quantity. However, the quality of care is a serious concern; a majority of neonatal deaths occur in the first seven days of infants’ life, and of these the largest share during the first 24 hours after childbirth meaning that the majority of neonatal death takes place under supervision of the health professionals.
The UNICEF commissioned the summative evaluation of DaO project as UNICEF Country Programme (2012-2016) is reaching the midpoint. The main purpose of the evaluation was to document and increase the knowledge of results, good practices and lessons learnt in perinatal care with specific recommendations.
It also specifies UNICEF’s contribution to enhancing maternal and child health care system, quality of care and maternal and child survival in the country as well as determines whether UNICEF pilot projects have been effective and should be scaled up countrywide. In addition to UNICEF Country Office, other UN Agencies and development partners as well as the Ministry of Health will benefit from the evaluation in planning, implementing and coordinating perinatal care.
The evaluation focused on assessing the project’s current and potential contribution to the improvement of MCH health indicators in Kyrgyzstan. Each evaluation criterion was analyzed from the perspective of assessing the implications of project activities’ on: i) final beneficiaries - women and children; ii) service providers - health care professionals whose capacity has been built (including doctors, midwives, and health facility managers); iii) sub-national decision-making level - Regional health authorities and (local governments; and iv) national decision-making level - national health authorities and key stakeholders (Ministry of Health and Department heads, Education institutions, Health Statistics Department, International Development and Implementing Partners, etc.).
The evaluation examined the relevance, effectiveness, efficiency and sustainability of the UNICEF’s contribution for which the OECD DAC8 evaluation approach has been applied. To translate the questions for the evaluation and the contextual issues, the Evaluation Framework (EF) was developed which structures the issues and questions as indicators that enabled to measure or assess them during the evaluation. Furthermore the EF also identifies the sources of information and the methods the evaluation team applied, the range of documents reviewed and key informants interviewed for each question.
The evaluation methodology comprised a mix of six major sources of data were used during the Evaluation:
People - Individuals were consulted through individual (semi-structured) interviews and focus groups;
Site visits: Data collected during the visits to sampled project supported sites;
Documents: All project and thematic area related documents were reviewed;
Observations: Qualitative data collected during observation of service delivery at the visited facilities were carried out;
Criterion based audit: data was collected through clinical audit of the patient medical cards.
Quantitative analysis: The ET utilized quantitative analysis to examine changes in selected but comparable indicators from available data.
For key informants interviews the topic guides were developed based on the Evaluation Framework to help ensure systematic coverage of questions and issues. The Focus Group Discussions (FGDs) were carried out for a) physicians, b) midwives and nurses, and d) beneficiaries in all facilities visited. In total the team carried out 15 FDG in two Project target areas (Osh and Batken). The evaluation was a joint and participatory process.
Findings and Conclusions:
The Relevance of UNICEF project to national priorities was evaluated based on following two criteria: i) Relevance of project-specific objectives to the priorities declared in the national policy documents; and ii) Relevance of project-specific objectives to the priority health problems of the population as evidenced by the available epidemiological data. Based on the analysis it is obvious that the relevance of the project is high through clear alignment with national reform and national perinatal care policies and strategies. The current UNICEF’s assistance contributes to the achievement of objectives of National Perinatal Care Project and aims to strengthen the capacity of maternity hospitals with the weakest capacity of medical workers, infrastructure, compliance to standards and with the highest rate of delivery.
In terms of efficiency the governance and management mechanisms set up proved to work well. So did the coordination system. Efficiency was ensured by adequate resource allocation, selection of most efficient funding modalities, especially for training and monitoring and supervision components, timely implementation of the planned activities and budget adjustments.
Training funding modality proved to efficient. UNICEF contracted Kyrgyz Association of Obstetricians, Gynecologists and Neonatologists (KAON) for the organization and delivery of trainings. While the given approach was beneficial for the empowerment of the professional association, the project staff observed the lack of ownership on part of the health facility management.
Efficiency of investments in infrastructure and medical equipment is acknowledged. 19% of funds were devoted for provision of critical life saving medical equipment and improvement of infrastructure.
UNICEF planted seeds for Perinatal Care sustainability in Kyrgyzstan. UNICEF has become a leader and reliable partner in the MCH sector and PC in particular.
Continued support to GOK’s
Perinatal Care Strategy Implementation - In MCH sector, UNICEF is recognized as one of the leading agencies.
Strengthen inter-agency collaboration- Continue and further improve inter-agency collaboration where all partners, building on their comparative advantage, will have a role to play in supporting the MoH in the implementation and update of programmes in the area of perinatal health, including further investment in strengthening the main health system components, and policies to reduce inequities in access to care, as well as in the quality of services provided, all of which will further improve the health status of mothers and children in Kyrgyzstan.
Enhance advocacy - The new challenges identified, in the section below, will require promotion of greater linkage and partnership through strengthening of the UNICEF Country Office (CO) technical capacity in the health policy advice.
Improve project design - Comprehensive Project design, addressing all health system blocks, with clear distribution of functions (activities and geographical and or thematic areas between key UN agencies), coordination mechanism between UN and other developing partners, as well as well formulated Results framework with annual targets, should be given a priority.
Though UNICEF is well positioned to influence the PC policy in the country, success of the perinatal services will very much depend on going beyond the perinatal care sector and targeting other health sector policy areas.
The GoK’s commitment for free perinatal care services are challenged by inadequate service case rates and weight resulting in underfinancing of health providers. Deficient reimbursement became more problematic with the introduction of the regionalization of perinatal services where the same case rates are applied for perinatal services regardless of the level of service provider. The situation is further complicated with absence of standardized claims management methodology, mostly based on previous service provision practices and poor understanding and knowledge of claims managers in new perinatal service provision technologies and referral guidelines, thus resulting in introduction of financial penalties to the health providers. Filling these gaps is a high priority for the MoH.
Assurance of the population access to quality PC service is important for attainment of the MDGs. The MoH already established the QoC and Pharmaceutical Policy Department at the MoH. Apart from this, the QoC committees are operational at the health facility level though with limited national guidance. Whereas availability of these structures demonstrates government’s ownership of the service quality improvement, much remains to be done to ensure well functioning service quality assurance system in the country.
Full report in PDF
PDF files require Acrobat Reader.