2015 Ghana: Summative Evaluation of the CI IHSS Programme in Ghana
Author: Doherty T, Daniels K, Daviaud E, Kinney M, Ngandu N for the IHSS Evaluation study group*
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 4’ of the report.
The Catalytic Initiative to Save a Million Lives was an international multi-donor partnership designed to accelerate progress on the health-related Millennium Development Goals (MDGs). As part of the Catalytic Initiative, from 2007 to 2013 the Canadian Department of Foreign Affairs, Trade and Development (DFATD) supported UNICEF’s Integrated Health Systems Strengthening (IHSS) programme in Ethiopia, Ghana, Malawi, Mali, Mozambique and Niger.
The aim of the IHSS was to reduce maternal and child mortality by strengthening the health system’s capacity to deliver high-impact interventions at the community level. During the first two years of implementation in Malawi, the IHSS supported a range of preventive interventions, including vaccinations, vitamin A supplementation and the distribution of insecticide-treated nets (ITNs). From 2009 to 2013, the programme supported the training and equipping of Health Surveillance Assistants (HSAs) to deliver integrated community case management (iCCM) of diarrhoea, malaria and pneumonia.
The IHSS was implemented in 10 districts in Malawi, which together contain 48 per cent of the population of the country.
In 2014 DFATD and UNICEF contracted the Medical Research Council (MRC), South Africa to conduct a summative external evaluation of the IHSS. The purpose of the evaluation, which was conducted in partnership with the University of the Western Cape and Save the Children, was to evaluate the effect of the IHSS on coverage of a package of maternal and child health interventions in the focus countries and to inform future programme and policy decisions in those countries.
The objectives of the evaluation were to assess the effect of the IHSS on the following:
Relevance: Alignment with national priorities and plans, enhanced policy environment and promotion of gender equity.
Effectiveness: Effect on strengthening the health system and the capacity of government and/or civil society organizations to train, equip, deploy and supervise front-line health workers to deliver a limited package of high-impact health interventions.
Impact: Effect on coverage of health and nutrition interventions supported by the IHSS; as well as the effect on the number of additional lives saved calculated using LiST.
Sustainability: The cost of implementing iCCM and the organizational and financial sustainability of the programme.
The scope of the external evaluation was limited to assessing the plausible contribution of the IHSS intervention to the above areas due to the infeasibility of establishing a true counterfactual.
The intended audience of the external evaluation includes the Ministries of Health in the six programme countries, DFATD, UNICEF, other UN agencies, and governmental and civil society partners at national, regional, and global levels.
A mixed method approach, including quantitative, qualitative and economic evaluation methods, was utilised. Coverage trend analysis was performed using a non-parametric test of trend across years and wealth quintiles for all available surveys. Data to assess implementation strength, utilisation, and quality of care were taken from routine programme data collected by UNICEF as well as the 2012 CHO/CHN survey. The indicators reported are aligned with the global iCCM indicators of the Expanded Results Framework.
We used LiST to investigate the extent to which changes in child mortality could be attributed to increases in intervention coverage. On the basis of measured baseline mortality values and changes in coverage, we forecasted child mortality over three time periods (pre-IHSS, phase I and phase II) and compared cumulative lives saved and annual rates of reduction for each period.
The costing component assessed the additional costs to the health sector including due to the introduction of curative interventions at community level by CBAs for the treatment of malaria, diarrhoea and pneumonia in children under 5. It also assessed the financial sustainability of the programme in relation to current utilisation and anticipated increased future levels of utilisation.
The effect of contextual factors, including rapid socioeconomic progress, policy changes, epidemiological changes and complementary and competing interventions by other donors and government, were described using data from document reviews and relevant databases, key informant interviews and focus group discussions with key stakeholders at all levels of the health system and at in communities.
The team visited three districts in the Northern region for field work (Saboba, Tolon and Savelugu) in addition to three days spent in Accra. A full list of individuals interviewed is provided in Appendix D.
Findings and Conclusions:
Key conclusion 1: The IHSS was well aligned with the policies of the Government of Ghana. However, lack of government ownership is a concern.
Key conclusion 2: By training more than 17,000 front-line health workers, the IHSS strengthened Ghana’s health system.
Key conclusion 3: The IHSS supported the procurement and distribution of a range of essential supplies. However, stock outs were a significant challenge.
Key conclusion 4: Exclusion of CBAs from the National Health Insurance Scheme hindered utilization.
Key conclusion 5: Improvements in coverage of a number of focus interventions were realized during the IHSS. However, low utilization of CBAs suggests that their contribution to these increases was relatively small.
Key conclusion 6: The programme’s effect on equitable access to health services was mixed.
Key conclusion 7: The IHSS contributed to a significant number of deaths averted.
Key conclusion 8: The additional cost of an iCCM treatment was very high, at an average of $13.70 per treatment.
Key conclusion 9: The CBAs trained were evenly split between men and women. This was widely seen as appropriate.
- Develop and implement strategies to increase demand for CBA services, including the possibility of including CBAs within the NHIS.
- Consider using CBAs to implement community-led total sanitation (CLTS) and community-based nutrition as part of the SUN (Scaling Up Nutrition) Initiative.
- Improve monitoring of the utilization of iCCM through community-based systems and periodic LQAS surveys.
- Document and disseminate lessons learnt around the increases in early initiation and maintenance of exclusive breastfeeding.
- Conduct further research to determine whether NHIS requirements are detrimentally affecting care-seeking.
- Improve the distribution and monitoring of drugs and other health supplies, possibly through the use of information technology.
- To reduce treatment costs and improve programme sustainability, undertake one of the following:
- Reduce the total number of CBAs, increasing the number of treatments provided per CBA; or
- Reduce the number of CBAs who provide iCCM. The other CBAs would provide only preventive and promotional interventions.
Ghana’s iCCM programme was set-up to respond to a lack of access to health services, in particular for the country’s poorest and most vulnerable people. The evaluation concluded that the cost of the programme does not appear to be a significant barrier to iCCM sustainability in Ghana. However, programme impact is being hindered by low utilization of CBAs. The evaluation suggested several possible reasons for this low utilization, including the exclusion of CBAs from the NHIS, improvements in access to facility-based care, lack of attention and funding to demand creation strategies, drug stock outs, and the fact that CBAs are voluntary and spend little of their time on iCCM.
Full report in PDF
PDF files require Acrobat Reader.