We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2017 Zambia: Mid-Term Evaluation of the Millennium Development Goal Initiative Accelerating Progress towards maternal, neonatal and child mortality reduction in Zambia



Author: Christine Thayer; Maria Paalman; Beyant Kabwe Nkenda Sachingongu

Executive summary

 With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’

Background:

The Zambian health sector has recorded significant progress in reducing maternal, child, infant and neonatal mortality rates, as well as HIV prevalence and the incidence of TB and malaria. Nevertheless, it still has to face a high disease burden and suffers from severe shortages of medical and nursing staff, poor management of drug and medical equipment supply and the maldistribution of health infrastructure and facilities. Inadequate financing, under-developed information systems and weaknesses in governance offer further challenges.

The MDGi programme is aligned to Zambia’s development agenda, in particular its Revised Sixth National Development Plan 2013-2016, which focuses on increased rural investment, including investments in health. The latter is the subject of the National Health Strategic Plan 2011-2015, designed to reduce the disease burden and improve health outcomes. It sets out the country’s health priorities with an emphasis on (a) the human resources crisis (b) health care infrastructure (c) multi-sectoral responses in key areas such as nutrition, HIV/AIDS, control of epidemics and health education, and (d) access to basic environmental health facilities such as water and sanitation, electricity, and telecommunications.

Purpose/Objective:

The main purpose of the midterm evaluation was to assess progress against attainment of the programme Expected Results, specific objectives included;
• Assess the design (relevance) of the MDGi programme and measure performance (efficiency and effectiveness) at activity and output level against the key indicators contained in the programme logical framework.
• Assess the adequacy of the M&E framework, including the overall theory of change (ToC) of the MDGi programme, and make recommendations, where appropriate, for its improvement.
• Assess the extent to which the programme is on schedule and including the quality and coherence of the MDGi programme linkages and synergies with the government's existing investments and health activities as well as those of other development partners in the field of reproductive, maternal, newborn, child health and nutrition (RNMCH&N).

Methodology:

The MTE applied a mixed-method approach combining both qualitative and quantitative methods. The MTE was primarily based on qualitative methods with the only primary quantitative data collection exercise being the client satisfaction survey conducted at sampled health facilities. Secondary quantitative data collected by the MDGi programme itself were used to analyze progress on key MDGi mid-term evaluation Final report.
Key data collection methods included: a) Comprehensive desk review of MDGi programme level documents, national health policies and strategies, and documents on other RMNCH&N programmes in Zambia b) Key informant interviews (KIIs) with stakeholders at national, provincial, district and health facility levels c) Group interviews with community-based volunteers (CBVs) and peer educators (PEs) working with the 15 sampled health facilities d) Focus group discussions (FGDs) with community members (women, adolescents and men) from the catchment area of the 15 sampled health facilities e) Exit interviews with clients having received RMNCH&N services at the 15 sampled health facilities and f) Collection of quantitative secondary data from existing sources, including baseline surveys, ZDHS (2013-2014) and HMIS on key indicators .
To measure progress towards targets for key programme indicators, the MTE analyzed the data presented in the baseline studies and annual reports of the MDGi programme. That is, baselines were compared with data from the Lot Quality Assurance Sampling (LQAS) and Zambia Demographic Health Survey (ZDHS) wherever available. Trendlines were developed to illustrate progress over time.

Findings and Conclusions:

Overall, the MTE concluded that the programme was well designed. Though with a slow start, the MTE revealed that the programme, has now attained its cruising speed and appears to be on course to achieve most of its targets.
In terms of assessing programme relevance, the MTE revealed that the programme design is highly relevant in the Zambian context and clearly aligned to national health priorities for reproductive, maternal, newborn, child health and nutrition (RMNCH&N). It addresses some of the most important health issues and gaps in the Zambian health system, namely a high maternal mortality rate, high fertility rate and early childbearing, a low contraceptive prevalence, poor knowledge of HIV and AIDS especially in rural areas, and nationally, 40% of children under age 5 stunted, with 17% severely stunted.
The effectiveness aspect of the MTE measured the extent to which the Expected Results(ER) have been achieved. Overall, the MDGi programme is on track of meeting most of its targets, under the nutrition component, the proportion of facilities delivering outpatient treatment for SAM has increased, but is still 24% short of the target (60%). The programme also scored significant attainment on the RMNCH training under its Maternal, neonatal and Child health.
At the time of the MTE, it was difficult to arrive at a satisfactory estimation of impact as the implementation has only been underway for 18 months. Many inputs still have to be fully rolled out and completed and work to improve the data captured by the HMIS is a critical element in determining overall impact. The end line surveys will thus provide possibilities of the MDGi programme to assess the contribution of the MDI programme to positive impact against RMNCH indicators.
In terms of assessing its sustainability, the MTE reveals that, it is likely that programme training activities and improvements to infrastructure and equipment will have long-term benefits.

Recommendations:

High Priority:
(1) Develop a practical and budgeted exit and sustainability action plan: The management team should give urgent attention to the drafting of a highly practical, budgeted exit and sustainability action plan for discussion with the MoH and the Ministry of Finance.
(2) Improve high-level coordination of the MDGi programme: Steps should be taken to convene the Steering Committee at least on a quarterly basis and to ensure that it receives a much more detailed financial report.
(3) Extend the programme until May 2019: Taking into account that no money can be spent during the last months of programme implementation and the large number of activities still to be undertaken or finalized, the evaluation team recommends an extension of one year until May 2019.

Medium priority:
(1) Integrate the MDGi programme’s log frame into the Medium-Term Expenditure Framework, including its goals, objectives and main strategies, indicators relevant to each of these, sources of verification and assumptions, and the resources/ inputs required.
(2) Improving the organization of birth registration services: The management team should carefully document the lessons learned in relation to this experiment and pass their report to the relevant government departments and other key stakeholders to inform further scale-up.
(3) Production and distribution of additional teaching aids, flyers and brochures on RMNCH&N topics for health facilities, CBVs and peer educators, with a view to supporting service delivery and demand creation



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2017

Country:
Zambia

Region:
ESAR

Type:
Evaluation

Theme:
Health

Partners:
Ministry of Health

Language:
English

Sequence:
2017/002

New enhanced search