We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 Niger: Summative Evaluation of the CI IHSS Programme in Niger



Author: Leon N, Besada D, Sanders D, Daviaud E, Kerber K, Rohde S, Doherty T for the IHSS Evaluation study group*

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 4’ of the report.

Background:

The Catalytic Initiative to Save a Million Lives was an international multi-donor partnership designed to accelerate progress on the health-related Millennium Development Goals (MDGs). As part of the Catalytic Initiative, from 2003 to 2013 the Canadian Department of Foreign Affairs, Trade and Development (DFATD) supported UNICEF’s Integrated Health Systems Strengthening (IHSS) programme in Ethiopia, Ghana, Malawi, Mali, Mozambique and Niger.

The IHSS aimed to contribute to the ultimate outcome of reducing maternal and child mortality. In alignment with government policies and plans, the IHSS sought to strengthen the health system’s capacity to deliver high-impact interventions at community level, thereby contributing to increased intervention coverage and lives saved. The programme was implemented in two phases. In most countries, Phase I (2007-2010) focused on the delivery of a number of preventive interventions, including immunizations, vitamin A supplementation, promotion of infant and young child feeding (IYCF) practices and the distribution of insecticide-treated nets (ITNs). During Phase I, the Lives Saved Tool (LiST) was applied across the six focus countries, showing that the greatest reductions in under-five mortality could be realized by strengthening the case management of childhood illnesses. Phase II (2010-2013) focused on:

  1. Formation of policy reforms to allow community health workers (CHWs) to provide treatment for diarrhoea, malaria and pneumonia, and alignment of iCCM within existing policies and strategies;
  2. Training on iCCM for CHWs and CHW supervisors at health facilities; and
  3. Establishment of systems for supplying CHWs with essential commodities, and supervising and monitoring their work.

Purpose/Objective:

In 2014 DFATD and UNICEF contracted the Medical Research Council (MRC), South Africa to conduct a summative external evaluation of the IHSS. The purpose of the evaluation, which was conducted in partnership with the University of the Western Cape and Save the Children, was to evaluate the effect of the IHSS on coverage of a package of maternal and child health interventions in the focus countries and to inform future programme and policy decisions in those countries.

The objectives of the evaluation were to assess the effect of the IHSS on the following:

Relevance: Alignment with national priorities and plans, enhanced policy environment and promotion of gender equity.

Effectiveness: Effect on strengthening the health system and the capacity of government and/or civil society organizations to train, equip, deploy and supervise front-line health workers to deliver a limited package of high-impact health interventions.

Impact: Effect on coverage of health and nutrition interventions supported by the IHSS; as well as the effect on the number of additional lives saved calculated using LiST.

Sustainability: The cost of implementing iCCM and the organizational and financial sustainability of the programme.

The scope of the external evaluation was limited to assessing the plausible contribution of the IHSS intervention to the above areas due to the infeasibility of establishing a true counterfactual.

The intended audience of the external evaluation includes the Ministries of Health in the six programme countries, DFATD, UNICEF, other UN agencies, and governmental and civil society partners at national, regional, and global levels.

Methodology:

A mixed methods approach -- including quantitative, qualitative and economic evaluation methods --  was used. Data sources for the coverage included the Niger Demographic and Health Survey (DHS) 1998, 2006, 2012, Multiple Indicator Cluster Survey (MICS) 2000, and the Niger Survival and Mortality Report 2010. Coverage trend analysis was performed using a non-parametric test of trend across years and wealth quintiles for all available surveys. Data to assess implementation strength, utilisation and quality of care were taken from routine data collected by UNICEF as well as the 2013 Census of Case de Santé and Centres de Santé Intégrés. The indicators reported are aligned with the global iCCM indicators of the Expanded Results Framework.

We used LiST to investigate the extent to which changes in child mortality could be attributed to changes in intervention coverage prior to IHSS (2000-2007) and during the period of IHSS programme implementation from 2007-2012, using baseline mortality values and changes in coverage of newborn and child health interventions from DHS, together with the LiST default input data.

The costing component assessed the additional costs to the health sector due to the introduction of curative interventions at community level by Agents de Sante Communautaires (ASCs) for the treatment of malaria, diarrhoea and pneumonia in children under 5, assuming delivery according to protocols. The costing component also assessed the financial sustainability in relation to current utilisation and anticipated increased future levels of utilisation.

The effect of contextual factors was described using data from document reviews, extraction from relevant databases, key informant interviews and focus group discussions -- the latter two conducted during an 11 day field visit in April 2013. The full list of interviewees is provided in Appendix D. The field visit was limited to two regions, due to limited time available and due to security concerns.

Findings and Conclusions:

Key conclusion 1: The IHSS was well aligned with the policies of the Government of Niger.

Key conclusion 2: By training more than 2,500 community health workers, the IHSS strengthened Niger’s health system.

Key conclusion 3: The IHSS supported the procurement and distribution of a range of essential supplies. However, the use of a parallel system was contrary to the programme’s health systems strengthening approach.

Key conclusion 4: The IHSS catalysed increased utilization of community health services.

Key conclusion 5: The IHSS contributed to improvements in coverage of a number of high-impact, low-cost interventions.

Key conclusion 6: During IHSS implementation, equity improved around a number of indicators, including ORS coverage and tetanus toxoid vaccination.

Key conclusion 7: The IHSS contributed to a significant number of deaths averted.

Key conclusion 8: The additional cost of an iCCM treatment was relatively low, at an average of $3.32 per treatment.

Key conclusion 9: A majority of ASCs trained in iCCM were female. However, female attrition was a challenge.

Recommendations:

  • Investigate the high attrition rate among female ASCs.
  • Improve the supply chain for iCCM medicines, including strengthening stock management at facility level to address the problem of expired drugs.
  • Strengthen routine health information systems at all levels of the health system.
  • Monitor the quality of iCCM care on an ongoing basis.
  • Further investigate the underlying reasons for the stagnation or decline in coverage of interventions.
  • Develop a formal system to facilitate efficient validation and payment of claims at health depots under the free health care programme.
  • Closely monitor changes to primary health care delivery (such as the delivery of obstetric and neonatal care at health depots), including cost implications and the impact on ASCs.

Lessons Learned:

The evaluation found that as a result of high utilization, the iCCM programme in Niger is cost effective. However, with competing demands on the government budget from increased security concerns, there is a real possibility that the government’s expenditure on health may decrease, and support from donors will be needed to ensure the sustainability of the iCCM programme. In this context the Global Fund’s new funding model, which can support the cost of iCCM programmes, could help ensure the continuity of the programme.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2015

Country:
Niger

Region:
WCARO

Theme:
Health

Type:
Evaluation

Partners:
Department of Foreign Affairs, Trade and Development Canada; Ministry of Health in Niger; UNICEF Niger

Language:
English

New enhanced search