2015 Malawi: Summative Evaluation of the CI IHSS Programme in Malawi
Author: Zembe W, Doherty T, Daviaud E, Besada D, Ngandu N, Kinney M, Daniels K, Jackson D, Sanders D for the IHSS Evaluation study group*
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 4’ of the report.
The Catalytic Initiative to Save a Million Lives was an international multi-donor partnership designed to accelerate progress on the health-related Millennium Development Goals (MDGs). As part of the Catalytic Initiative, from 2007 to 2013 the Canadian Department of Foreign Affairs, Trade and Development (DFATD) supported UNICEF’s Integrated Health Systems Strengthening (IHSS) programme in Ethiopia, Ghana, Malawi, Mali, Mozambique and Niger.
The aim of the IHSS was to reduce maternal and child mortality by strengthening the health system’s capacity to deliver high-impact interventions at the community level. During the first two years of implementation in Malawi, the IHSS supported a range of preventive interventions, including vaccinations, vitamin A supplementation and the distribution of insecticide-treated nets (ITNs). From 2009 to 2013, the programme supported the training and equipping of Health Surveillance Assistants (HSAs) to deliver integrated community case management (iCCM) of diarrhoea, malaria and pneumonia.
The IHSS was implemented in 10 districts in Malawi, which together contain 48 per cent of the population of the country.
In 2014 DFATD and UNICEF contracted the Medical Research Council (MRC), South Africa, to conduct an external evaluation of the IHSS. The purpose of the evaluation, which was conducted in partnership with the University of the Western Cape and Save the Children, was to evaluate the effect of the IHSS on coverage of a package of maternal and child health interventions in Malawi and to inform future programme and policy decisions in Malawi and regionally.
The objectives of the evaluation were to assess the effect of the IHSS on the following:
Relevance: Alignment with national priorities and plans, enhanced policy environment and promotion of gender equity.
Effectiveness: Effect on strengthening the health system and the capacity of government and/or civil society organizations to train, equip, deploy and supervise front-line health workers to deliver a limited package of high-impact health interventions.
Impact: Effect on coverage of health and nutrition interventions supported by the IHSS; as well as the effect on the number of additional lives saved calculated using the Lives Saved Tool (LiST).
Sustainability: The cost of implementing iCCM and the organizational and financial sustainability of the programme.
The scope of the external evaluation was limited to assessing the plausible contribution of the IHSS intervention to the above areas due to the infeasibility of establishing a true counterfactual.
The intended audience of the external evaluation includes the Ministries of Health in the six programme countries, DFATD, UNICEF, other UN agencies, and governmental and civil society partners at national, regional, and global levels.
A mixed method approach with quantitative, qualitative and economic evaluation methods was used. Data sources for coverage and LiST analyses included the Malawi Demographic and Health Survey (DHS) 2000, the Malawi Multiple Indicator Cluster Survey (MICS) 2006, the DHS 2010 and the 2012/13 Lot Quality Assurance Survey (LQAS). Trend analysis was performed using a non-parametric test of trend across years and wealth quintiles for all available household surveys.
Data to assess implementation strength, utilisation and quality of care were taken from routine programme data collected by UNICEF, a HSA survey in Balaka district and a quality of care survey in 6 of the 10 IHSS programme districts. The indicators reported are aligned with the global iCCM indicators of the Expanded Results Framework.
We used LiST to investigate the extent to which changes in child mortality could be attributed to increases in intervention coverage. On the basis of measured baseline mortality values and changes in coverage, we forecasted child mortality over three time periods: pre-IHSS, and phase I and phase II of the IHSS programme.
The costing component assessed the additional costs to the health sector due to the introduction of curative interventions at community level by HSAs for the treatment of malaria, diarrhoea and pneumonia in children under 5. It also assessed the financial sustainability of the programme in relation to current utilisation and anticipated increased future levels of utilisation. Costs are expressed in US$ 2012.
The effect of contextual factors were analysed using data from document reviews, extraction from relevant databases, key informant interviews and focus group discussions with national stakeholders, district personnel, HSAs, HSA supervisors, and beneficiaries. The team visited two districts for field work, Kasungu and Mzimba, in addition to spending three days in Lilongwe.
Findings and Conclusions:
Key conclusion 1: The IHSS was well aligned with the policies of the Government of Malawi.
Key conclusion 2: By training more than 1,000 community health workers, the IHSS strengthened Malawi’s health system.
Key conclusion 3: The IHSS supported the procurement and distribution of a range of essential supplies. However, stock outs were a major challenge.
Key conclusion 4: The practice of having HSAs work in areas they do not reside in hindered utilization.
Key conclusion 5: The IHSS contributed to improvements in coverage of a number of high-impact, low-cost interventions.
Key conclusion 6: During IHSS implementation, equity improved around postnatal care, IPTp coverage and rates of early breastfeeding.
Key conclusion 7: The IHSS contributed to a significant number of deaths averted.
Key conclusion 8: The additional cost of an iCCM treatment was relatively low, at an average of $1.44 per treatment.
Key conclusion 9: Gender equality remains a challenge in Malawi’s iCCM programme. An overwhelming number of HSAs are men.
- Increase coordination and tracking of partners working on iCCM.
- Increase harmonization of activities around the supply chain, encouraging all partners to work through government systems.
- Consider decreasing the educational requirements for HSAs and training illiterate women to be HSAs.
- Decentralize the recruitment of HSAs to communities.
- Increase the frequency of one-on-one supervision, including case management observation.
- Include a line item for the maintenance of HSA bicycles in future iCCM budgets.
- Further investigate the underlying reasons for the decline in coverage of important preventive interventions, including vitamin A supplementation.
- Undertake a new sustainability study once the programme has reached higher maturity to better evaluate the costs of the programme.
- Formulate a long-term financial plan for the iCCM programme that recognizes that external support will be needed for the foreseeable future.
The Government of Malawi has demonstrated commitment to, and ownership of, iCCM, and the approach has been embraced and integrated into the national health system. The evaluation concluded that the programme runs efficiently. However, given the difficult financial situation in the country, the government is unlikely to be able to fund a programme of this magnitude on its own, and therefore some form of continued external support will be needed. The evaluation team suggested that the Global Fund’s New Funding Model, in which iCCM programmes are eligible for funding, could help ensure financial sustainability.
Full report in PDF
PDF files require Acrobat Reader.