We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 Ethiopia: Summative Evaluation of the CI IHSS Programme in Ethiopia



Author: Doherty T, Loveday M, Nsibande D, Daniels K, Daviaud E, Kerber K, Zembe W, Leon N, Kinney M for the IHSS Evaluation study group*

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 4’ of the report.

Background:

The Catalytic Initiative to Save a Million Lives was an international multi-donor partnership designed to accelerate progress on the health-related Millennium Development Goals (MDGs). As part of the Catalytic Initiative, from 2007 to 2013 the Canadian Department of Foreign Affairs, Trade and Development (DFATD) supported UNICEF’s Integrated Health Systems Strengthening (IHSS) programme in Ethiopia, Ghana, Malawi, Mali, Mozambique and Niger.

The IHSS aimed to contribute to reducing maternal and child mortality. In alignment with government policies and plans, the IHSS sought to strengthen the health system’s capacity to deliver high-impact interventions at the community level, thereby increasing coverage and saving lives. The programme was implemented in two phases in Ethiopia. During Phase I (2007-2010), health extension workers (HEWs)1 were trained to provide integrated community case management (iCCM) of diarrhoea and malaria. Other supported interventions included vitamin A supplementation, immunization and the promotion of infant and young child feeding (IYCF) practices. Rollout of the full iCCM package was undertaken during Phase II (2010-2013), after a government policy change allowed HEWs to treat pneumonia with antibiotics and HEWs were trained to manage acute malnutrition.

Through the programme UNICEF provided full support – including the provision of training materials, commodities and technical support – in 239 rural woredas (districts) in Ethiopia’s Amhara, Benishangul-Gumuz, Oromia, Southern Nations, Nationalities and Peoples’ (SNNP) and Tigray regions. Partial support was provided in an additional 340 woredas. Approximately 4.6 million children under 5 years of age live in the districts covered by full IHSS programme support and 5.6 million live in districts that benefited from partial IHSS programme support.


Purpose/Objective:

In 2014 DFATD and UNICEF contracted the Medical Research Council (MRC), South Africa to conduct a summative external evaluation of the IHSS. The purpose of the evaluation, which was conducted in partnership with the University of the Western Cape and Save the Children, was to evaluate the effect of the IHSS on coverage of a package of maternal and child health interventions in the focus countries and to inform future programme and policy decisions in those countries.
The objectives of the evaluation were to assess the effect of the IHSS on the following:

Relevance: Alignment with national priorities and plans, enhanced policy environment and promotion of gender equity.

Effectiveness: Effect on strengthening the health system and the capacity of government and/or civil society organizations to train, equip, deploy and supervise front-line health workers to deliver a limited package of high-impact health interventions.

Impact: Effect on coverage of health and nutrition interventions supported by the IHSS; as well as the effect on the number of additional lives saved calculated using LiST.

Sustainability: The cost of implementing iCCM and the organizational and financial sustainability of the programme.

The scope of the external evaluation was limited to assessing the plausible contribution of the IHSS intervention to the above areas due to the infeasibility of establishing a true counterfactual.

The intended audience of the external evaluation includes the Ministries of Health in the six programme countries, DFATD, UNICEF, other UN agencies, and governmental and civil society partners at national, regional, and global levels.

Methodology:

A mixed methods approach (including quantitative, qualitative and economic evaluation methods) was utilised. For analysis of coverage, trend analysis was performed using a non-parametric tests across years and wealth quintiles for all available surveys. Data to assess implementation strength, utilisation and quality of care were taken from routine programme data as well as the Johns Hopkins Evaluation in Oromia. The indicators reported are aligned with the global iCCM indicators of the Expanded Results Framework.

Subnational population and regional household survey data were used in LiST to investigate the extent to which changes in child mortality could be attributed to changes in intervention coverage. On the basis of measured baseline mortality values and changes in coverage of newborn and child health interventions, we forecasted child mortality over three time periods (pre-IHSS, phase I and phase II).

The costing component assessed the additional costs to the health sector due to the introduction of curative interventions at community level by HEWs for the treatment of malaria, diarrhoea and pneumonia in children under 5. It also assessed the financial sustainability of the programme in relation to current utilisation and anticipated increased future levels of utilisation.

The effect of contextual factors (including rapid socioeconomic progress, policy changes, epidemiological changes and complementary and competing interventions by other donors and government) were described using data from document reviews and relevant databases. Contextual data to support the quantitative coverage data were collected during key informant interviews and focus group discussions with national stakeholders, key district personnel, HEWs, their supervisors, and beneficiaries.

The team visited Gojam and Gondor districts in the Amhara region and Shewa district in Addis Ababa in addition to three days spent in Addis Ababa.

Findings and Conclusions:

  • Key conclusion 1: The IHSS resulted in the introduction of iCCM in a policy environment that was not wholly receptive at the outset of implementation.
  • Key conclusion 2: By training more than 30,000 front-line health workers, the IHSS strengthened Ethiopia’s health system. However, supervision remains a challenge.
  • Key conclusion 3: The IHSS supported the procurement and distribution of a range of essential supplies. However, both stock outs and the under-utilization of supplies were experienced.
  • Key conclusion 4: A range of factors hindered the utilization of HEWs.
  • Key conclusion 5: The IHSS contributed to improvements in coverage of a number of high-impact, low-cost interventions.
  • Key conclusion 6: The programme’s effect on equitable access to health services was mixed.
  • Key conclusion 7: The IHSS contributed to a significant number of deaths averted.
  • Key conclusion 8: The additional cost of an iCCM treatment was relatively high, at an average of $7.33 per treatment.
  • Key conclusion 9: All HEWs trained through the IHSS were women. However, further efforts are needed to improve gender equity in the health workforce.

Recommendations:

The main recommendations of the report are:

  1. Consider utilizing HEWs to improve maternal health, particularly around postnatal care.
  2. Continue the promotion of key family practices (especially infant and young child feeding (IYCF) and WASH) through HEWs.
  3. Conduct further research on the barriers to utilization of HEWs and demand for their services; particularly whether health posts are optimally located in terms of distance from health facilities and distance from each other.
  4. Increase utilization of health posts for iCCM through demand creation activities.
  5. Assess the adequacy of the current HEW training and equipment to support maternal and newborn health
  6. Improve supply chain management, with a focus on getting supplies from medical stores to health posts.
  7. Improve and routinize the supervision of HEWs, including observation of case management.
  8. Increase efforts around postnatal care, exclusive breastfeeding and vitamin A supplementation.
  9. Improve the cost-efficiency of the programme by integrating iCCM training into basic training courses.

Lessons Learned:

The evaluation found that the IHSS acted as a catalyst in Ethiopia, strengthening the community platform from which essential maternal and child health interventions can be implemented. iCCM currently covers 10 million of the country’s 12 million children under 5 years of age (83 per cent of the total). The evaluation found that if iCCM were to be scaled up throughout the country to cover 100 per cent of children, the total cost would be 0.05 per cent of Ethiopia’s public health expenditure and 0.16 per cent of the government’s own health expenditure. In this context, the evaluation concluded that the Government of Ethiopia has the capacity to finance the recurring costs of the iCCM programme from its own funding and suggests that the programme is financially sustainable.

 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2015

Country:
Ethiopia

Region:
ESARO

Theme:
Health

Type:
Evaluation

Partners:
Department of Foreign Affairs, Trade and Development Canada; Ministry of Health; UNICEF Ethiopia

Language:
English

New enhanced search