2010 Kazakhstan: Final Evaluation: UNICEF component aimed at Strengthening Health and Social Services. Within the 2008-2009 Joint UNFPA/UNICEF/UNDP/UNV Project on Enhancing Human Security in the Former Nuclear Test Site of Semipalatinsk in Kazakhstan
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
The Joint UNFPA/UNICEF/UNDP/UNV three-year programme on "Enhancing Human Security in the Former Nuclear Test Site of Semipalatinsk" (2008-2010) was launched in March 2008 to support the Government of Kazakhstan in its recovery and development of the Semipalatinsk region. The programme was funded by the Government of Japan through the UN Human Security Trust Fund (UNTFHS) in the amount of USD 1,978,698. The Joint Programme aimed at improving the wellbeing of vulnerable populations in the Semipalatinsk region and developing economic, ecological, social and health conditions so as to enhance social security and prevent social conflict. The programme will be completed in December 2010.
By examining information and data available on the project structure, design and implementation, products and results, the evaluation aims to assess the project in terms of its effectiveness, impact, and replication. More specifically, the evaluation examines the project in terms of: Relevance in light of national priorities and needs, particularly in EKO; Effectiveness of its partnership strategy, component activities; Efficiency in design, structure and utilization of resources, Impact in terms of tangible development results; and, Sustainability in terms of the potential for national ownership, integration of results and replication in the future. Findings are presented and analyzed to describe opportunities and challenges and suggest recommendations for consideration in the second phase of this project.
Specific objectives of the Evaluation are:
1. To evaluate the relevance, effectiveness, efficiency, sustainability and impact of the project in overall and, specifically of the UNICEF component;
2. To assess the relevance of objectives of the UNICEF-assisted component of the project and to what extent they have been achieved;
3. To assess the relevance of the UNICEF-assisted interventions vis-à-vis national priorities and existing state programmes and strategies;
4. To identify gaps and weaknesses in the project’s design and implementation and offer recommendations regarding the achievement of the envisaged outcomes;
5. To identify lessons learned from the project interventions with a view to assessing opportunities to continue, refine or change interventions in the future;
6. To examine partnerships between the main stakeholders implementing the project;
7. To contribute to the overall evaluation of the project.
The following methodology and approach guide this evaluation:
• Literature review: Documents reviewed include the original project document, Annual Reports, Minutes of Project Board, Mid-term evaluation report, relevant government documents on health care, education and social protection reform, development vision and strategies for Kazakhstan, SITANs, UNICEF CP and CPAP reports, UNDAF report, training session report, independent evaluation reports, and other key project documents and evaluations and related material. See Annex 2 for a full list of documents reviewed for this assessment.
• Interviews with Programme and Project staff: Interviews conducted in Astana with relevant programme and project staff at UNICEF-Kazakhstan, UN implementing agencies (especially UNFPA) to assess partnership strategy, programme design and implementation, achievement of objectives (including specific UNICEF component), and progress toward goals. Structured and unstructured interviews aim at assessing relevance, effectiveness, efficiency, sustainability and impact of programme and component projects. (see Annex 5 for interview/discussion guide)
• In country interviews with non-UN implementing partner agencies and staff: Interviews conducted with local authorities and other national partner and implementing organizations and relevant stakeholders in Semey and Ust-Kamenogorsk (including SSMU, SMC, PI, YFS and other relevant health, education, child protection and social services departments, local and regional governments) (See Annex 5 for interview/discussion guide)
Using structured and unstructured interviews, the goal is to assess overall project achievements, component-specific activities and progress in realizing outcomes and objectives as stated in the original project document. Questions are also formulated to assess project design and implementation and assess results in terms of relevance, effectiveness, efficiency, impact and potential for replication and in order to identify challenges and opportunities in promoting human security and long term development in the Semipalatinsk region.
• Interviews with Project Beneficiaries: As feasible and as time allows, visits to project sites in Semey and Ust-Kamenogorsk to talk to direct beneficiaries of project activities, including counseling staff at YFS centers, FSC health providers, mothers and young people so as to obtain their direct feedback on the value of the project, its component activities and interventions in addressing the health and social needs of the community and effecting real change and improvement in health and development conditions.
• Presentation of Findings: Findings of the evaluation (PowerPoint presentation) shared with UNICEF management and staff at conclusion of in-country visit.
• Draft and Final reports: A draft report synthesizing findings of the evaluation in reference to the TOR prepared and submitted to UNICEF-Kazakhstan. Comments and responses received incorporated as needed into the final report.
Findings and Conclusions:
The evaluation analyzes achievements of the UNICEF project component by highlighting successfully achieved outputs and corresponding promising changes and trends regarding perinatal health conditions and health care in the Semipalatinsk region. Observations and conclusions about project design, relevance and effectiveness, efficiency, sustainability and outcomes are examined in light of the findings and with reference to the questions posed in the TORs
The evaluation uncovered progress in key areas:
• Effectiveness of trainings: There appears demonstrable evidence of increased knowledge, skills and capacity imparted through this project in critical areas of maternal and child health and wellbeing among those targeted for training in this project (pediatric physicians, Ob/Gyn, neonatal health care professionals and nurses, midwives, managers, etc). In light of long-term changes sought, it is still too early to ascertain how much of a shift in attitudes and practice have been institutionalized into the provision of health care at health facilities in the EKO.
• Relevance: In light of the ongoing national commitment to reform and development in Kazakhstan, the UNICEF component is very much in line with national priorities and needs and in line with the GoK vision for long term development. The project revealed growing consensus among government and non-government counterparts about the importance and value of addressing critical health and social needs of mothers and children in the vicinity of the former nuclear site in the Semipalatinsk region and the surrounding area.
• Efficiency and sustainability: The project succeeded in creating synergy – involving participation of a range of stakeholders who utilize project results and outcomes for their own work. Cascading and ripple effects expanded the pool of beneficiaries and entrenched the sense of national ownership and commitment to the effort. Institutionalizing tools and methodologies from this project and adapting them to the needs of the local community also help ensure sustainability beyond the life of the project (for example, training Master Trainers, incorporating early child care into the curricula at SSMU, SMC and the Medical College in Ust-Kamenogorsk, among others).
The effectiveness and efficiency of these and other outcomes of the project are mitigated by the lack of systemic monitoring systems. While UNICEF monitoring mechanisms were in place to monitor implementation of activities, mechanisms for monitoring practice and performance at the local government and partner levels have yet to be incorporated and strengthened. These are indispensible to assessing outcomes, quality assurance and quality control. Without such systems it is difficult to measure practice, performance, feedback, management, filling gaps, etc – all the elements of an efficient and effective use of project resources at any level (also see Challenges and Recommendations sections below).
• Impact of the Multi-sectoral approach: Both UNICEF and its national partners recognize that a multi-sectoral approach involving all key players is central to effective project interventions and successful outcomes in this region. Collaboration by local authorities and institutions in this project fosters national ownership, greater effectiveness, efficiency and sustainability and the potential for long-term development results (for example, Youth Friendly Services (YFS), Family Support Centers (FSC), and especially the Child Friendly City Initiative (CFCI)).
• Outcomes and Results: Partners state there is discernible reduction in both infant mortality and maternal mortality as a result of this project’s focus on training and capacity building among relevant health professionals. (As explained below, evidence is mixed on maternal mortality rates)
• Lessons learned: The project was originally to be implemented similar to other UNICEF interventions (e.g. technical assistance, training health workers and other immediate stakeholders). But UNICEF soon discovered that buy-in and collaboration with local government entities, particularly the local health department in Semey was indispensible to achieving project goals. Outreach and collaboration with local and regional authorities was subsequently incorporated into the ongoing project. In response to partner requests, the project also later incorporated trainings for relevant health workers in Ust-Kamenogorsk.
• Evidence of scaling up and upstream efforts: This project demonstrates successful piloting and evidence of scaling up; achievements that enhance the potential of long term impact on improving maternal and perinatal health and wellbeing. Of particular significance to UNICEF is evidence that the project contributes to and reinforces upstream efforts – to ensure national legislation and documents reflect international human rights and development standards. One of the major achievements in this regard is government passage of Order 691 that mandates all outpatient clinics in the country to establish a Healthy Baby Room for the care of infants 0-3 years. Partners report that access to government agencies and departments and the opening of channels between them and government has been one of the major achievements of the project.
The main recommendations include the following:
• Project Design: Project inputs and outputs and are not clearly linked to achievement of project objectives. This may be due to the fact that UNICEF concentrates on monitoring implementation of activities while local government and partners (who may lack sufficient capacity in this regard) are largely responsible for monitoring outcomes. Project reports concentrate on documenting “progress” and outputs, which in the absence of consistent outcome reporting do not automatically demonstrate or correlate with achievement of objectives Consider incorporating an outcome monitoring mechanism directly in UNICEF’s design and implementation plan and formulating a more logical loop and linkage between objectives on the one hand, and inputs/activities, outputs and outcomes intended to achieve them on the other (see sections below for more detail).
• Linkage to enhance effectiveness: Continue to make linkages between training for skills, knowledge, capacity-building with opportunities to put the latter into practice. This entrenches knowledge, enhances skills, reveals gaps and improves performance. Consider including training for other specialists – such as Family Doctors and General Practitioners that were not targeted in this project.
• Results Based Management (RBM) and Budget: Quality Assurance (QA) and Quality Control (QC) incorporate child responsive budgets and results based management. Progress in performance/results-based management (RBM) and budgeting (RBB) initiated under earlier UNICEF projects (such as the 2007 Public Expenditure Review (PER) in the social sector can be built upon to strengthen capacity of local government and partners in planning, management, and monitoring and evaluation of social programmes and budgets in upcoming phases of the project.
• Gender Responsive Budget (GRB): Alongside child responsive management and budgeting, QA and QC should immediately incorporate GRB. This is especially critical before the CFCI gets underway and should be incorporated into current plans of action for long term development. Where UNICEF and other UN agencies are able to integrate this they can also model GRB for other national and local partners and organizations in their own development plans.
• Partnerships: As UNICEF is aware, national ownership is greatly enhanced when proposals and implementation plans are developed in partnership and collaboration with national partners - especially at the regional level and in light of upcoming 2020 national development plans. These provide and an opportunity to incorporate child health and wellbeing (and means of addressing these) directly into plans formulated at the local level. (Up-scaling to the national level is highly anticipated as interest and motivation around these interventions spread)
• Synergy with projects of other UN agencies: Results are enhanced and entrenched, and outcomes are most visible and effective when partner UN agencies are complimenting UNICEF work toward the same long-term development goals: Ongoing collaboration is a logical outcome of common priorities and mission; such as UNDP (in light of its areas of expertise in poverty reduction, small business capacity building, governance etc)
• Private sector: Consider futhering efforts in support of national partners in their efforts to engage and involve private sector enterprises. This is pertinent to long-term development, CFCI and the overall goal of enhancing human security in the Semipalatinsk region.
• National Partners: Consider opportunities to map other local NGOs and institutions (for example, of the 100 organizations in Semey, 30 reportedly have the capacity to participate). Expanding the pool of national organizations will foster more national ownership– especially as the CFCI gets underway
• Local Government: Continuing work local government departments to streamline and add coherence to activities and standards may yield promising outcomes and opportunities to translate children’s rights under the CRC into concrete development projects, especially as the CFCI gets underway.
• Monitoring and follow up: Follow up with those trained is critical, especially with those trained as “master trainers” and responsible for training other health workers. Related is the importance of putting in place and using systematic monitoring mechanisms to assess and improve performance, quality assurance and control (major areas of deficiency across the board in this project)
• Rural areas: Some partners are conducting trainings or other activities in rural areas but this project initially concentrated on cities (Semey, later Ust-Kamenogorsk) and several rural districts that suffered most from the nuclear tests (Abay, Beskaragay, Kurchatov and Shulbinsk). Expanding to rural areas so as to tackle the centers of poverty and vulnerability at their roots is critical and will be addressed in upcoming phases.
• Upstream efforts: UNICEF has demonstrated strategies and successes at the local level (in these projects) for use in advocacy at the national policy level. Continuing these efforts is important as the GoK undertakes reform - in health care, child protection, education and special services, etc. The same strategy is useful with respect to local and regional authorities, so as to incorporate international human rights standards directly into government policies and initiatives.
• Public awareness campaigns: Consider support to public awareness campaigns coinciding with the unfolding and implementation of CFCI and to raise public awareness on perinatal health issues and the benefits of new techniques and approaches to improving mother and infant care
• Ongoing Child Rights Focus: Partners report UNICEF’s unique and valuable contribution in modeling international standards and encouraging government and nongovernment partners to adhere to provisions of the CRC and international standards for incorporating child rights directly into the mandate and operation of relevant sectors.
Full report in PDF
PDF files require Acrobat Reader.