2015 Rep. of Mozambique: Summative Evaluation of the CI IHSS in Mozambique
Author: Rohde S, Rohde J, Daviaud E, Besada D, Kinney M for the IHSS Evaluation study group*
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 4’ of the report.
The Catalytic Initiative to Save a Million Lives was an international multi-donor partnership designed to accelerate progress on the health-related Millennium Development Goals (MDGs). As part of the Catalytic Initiative, from 2007 to 2013 the Canadian Department of Foreign Affairs, Trade and Development (DFATD) supported UNICEF’s Integrated Health Systems Strengthening (IHSS) programme in Ethiopia, Ghana, Malawi, Mali, Mozambique and Niger.
The aim of the IHSS was to reduce maternal and child mortality by strengthening the health system’s capacity to deliver high-impact interventions at the community level. The programme was implemented in two phases in Mozambique. During Phase I (2007-2010), the focus was on the adoption of the Reach Every District (RED) approach, organization of child health weeks and procurement and distribution of insecticide-treated nets (ITNs). Phase II (2010-2013) focused on policy dialogue and planning that resulted in the training and deployment of Agente Polivalente Elementares (APEs) to deliver integrated community case management (iCCM) of diarrhoea, malaria and pneumonia. The revitalization of the APE cadre began as a pilot in eight districts in 2010 and was scaled up to reach 120 districts in 2013.
In 2014 DFATD and UNICEF contracted the Medical Research Council (MRC), South Africa, to conduct an external evaluation of the IHSS. The purpose of the evaluation, which was conducted in partnership with the University of the Western Cape and Save the Children, was to evaluate the effect of the IHSS on coverage of a package of maternal and child health interventions in Mozambique and to inform future programme and policy decisions in Mozambique and regionally.
The objectives of the evaluation were to assess the effect of the IHSS on the following:
- Relevance: Alignment with national priorities and plans, enhanced policy environment and promotion of gender equity.
- Effectiveness: Effect on strengthening the health system and the capacity of government and/or civil society organizations to train, equip, deploy and supervise front-line health workers to deliver a limited package of high-impact health interventions.
- Impact: Effect on coverage of health and nutrition interventions supported by the IHSS; as well as the effect on the number of additional lives saved calculated using the Lives Saved Tool (LiST).
- Sustainability: The cost of implementing iCCM and the organizational and financial sustainability of the programme.
At the time of the evaluation, the IHSS in Mozambique was still in the process of going to scale. Therefore, the evaluation served as a mid-term assessment and is not intended as an endline evaluation.
The scope of the external evaluation was limited to assessing the plausible contribution of the IHSS intervention to the above areas due to the infeasibility of establishing a true counterfactual.
The intended audience of the external evaluation includes the Ministries of Health in the six programme countries, DFATD, UNICEF, other UN agencies, and governmental and civil society partners at national, regional, and global levels.
A mixed method approach to this evaluation was used, involving quantitative, qualitative and economic evaluation methods. For analysis of coverage, trend analysis was performed using a non-parametric test of trend across years and wealth quintiles for the Demographic Health Survey (DHS) and the Multiple Indicator Cluster Survey (MICS) surveys. Data to assess implementation activities, utilisation and quality of care were taken from routine programme data collected by UNICEF from APE reports. The indicators reported are aligned with the global iCCM indicators of the Expanded Results Framework.
Using household survey data, we used LiST to investigate the extent to which changes in child mortality could be attributed to increases in intervention coverage. On the basis of measured baseline mortality values and changes in coverage, we forecasted child mortality by cause over three time periods (2000-07, Phase I 2007-10 and, Phase II 2010-13).
The costing component assessed the additional costs to the health sector including due to the introduction of curative interventions at community level by APEs for the treatment of malaria, diarrhoea and pneumonia in children under 5. It also assessed the financial sustainability of the programme in relation to current utilisation and anticipated increased future levels of utilisation.
The effect of contextual factors (including socioeconomic progress, policy changes, epidemiological changes and complementary and competing interventions by other donors and government) were described using data from document reviews and relevant databases and from extensive interviews with MISAU and other key informants nationally, in health districts and communities served by APEs. The evaluation team visited two provinces for field work (Gaza and Inhambane) in addition to three days spent in Maputo. A full list of individuals interviewed is provided in Appendix D.
Findings and Conclusions:
Key conclusion 1: The IHSS programme was well aligned with the policies of the Government of Mozambique.
Key conclusion 2: By training more than 900 community health workers, the IHSS strengthened Mozambique’s health system. However, supervision remains a challenge.
Key conclusion 3: The IHSS supported the procurement and distribution of a range of essential supplies. However, improved monitoring is needed to ensure accountability.
Key conclusion 4: The IHSS catalysed increased utilization of community health services.
Key conclusion 5: Improvements in coverage of a number of focus interventions were realized during the IHSS.
Key conclusion 6: The IHSS contributed to improved equity in access to health services.
Key conclusion 7: While attributing deaths averted to the programme would be premature, the IHSS delivered interventions that have the potential to save lives.
Key conclusion 8: The additional cost of an iCCM treatment was relatively high, at an average of $7.98 per treatment.
Key conclusion 9: Gender equality remains a challenge in Mozambique’s iCCM programme, with females making up only 30 per cent of APEs trained by the IHSS.
- Explore various mechanisms to ensure donor and NGO coordination and adherence to programme norms, while allowing flexibility to explore innovative approaches.
- Carefully pilot and evaluate the possibility of expanding the services offered by APEs, including around family planning and tuberculosis treatment.
- Revise human resource requirements to regularize the APE workforce.
- Conduct trials on cost recovery mechanisms.
- Design and implement activities to generate increased demand for the services of APEs.
- Improve and regularize the supervision of APEs, including training on the proper use of registers and reports.
- Compare the current four-month continuous training model with episodic training of one month every quarter for four rounds.
- Promote experienced and well-functioning APEs to a supervisory or mentoring role.
- Conduct periodic lot quality assurance sampling (LQAS) surveys to validate routine data and assist with planning.
- Develop a set of indicators to measure the effectiveness of APEs.
- Encourage government commitment to take on the financial burden of APE subsidies.
- Maintain short-term donor funding for the programme while alternative sources are explored.
- Encourage the World Bank and the Global Fund to support APEs as a mechanism to deliver integrated primary health care in hard-to-reach and vulnerable communities.
The evaluation found that the APE programme in Mozambique is clearly part of national policy and is fully owned by the Ministry of Health. The main concern about the sustainability of the programme, consistently reiterated in interviews with key stakeholders, is the capacity of the government to take on the responsibility for APE subsidies. As an interim measure, some donors have indicated their willingness to continue to pay the cost of training and supplies if the government commits to paying the APE subsidies. However, the government’s clear support of the programme has not yet translated into a commitment to pay for subsidies.
Full report in PDF
PDF files require Acrobat Reader.