We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Rwanda: Comprehensive Evaluation of The Community Health Program in Rwanda



Author: Luigi D’Aquino, Audrey Mahieu

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as ‘Part 4’.

Background:

Rwanda started its community health program (CHP) in 1995, soon after the genocide against the Tutsi. The number of community health workers (CHWs) was about 12,000. At that time, there was no policy, strategy or operational guidelines on how the CHP should be implemented. Initially, CHWs focused on health education and facilitated health campaigns; with time, their role evolved into a more comprehensive community-led initiative.

From 2005, after the decentralisation policy was introduced countrywide, there was sustained capacity building of the CHWs through training and supply of materials. By 2011, the number of CHWs had grown to 60,000. In May 2012, the Ministry of Health and Ministry of Local Government decided to remove the CHWs in-charge of Social Affairs in all the villages. The number of CHWs was therefore reduced from 60,000 to approximately 45,000. Each village is meant to have 3 CHWs. One CHW, named Assistante Maternelle de Santé (ASM), is in charge of maternal and newborn health and the other two CHWs constists in a Binôme. The Binôme is a male and female pair of CHWs who are multi-disciplinary, polivalent health agents. The range of services offered at community level by CHWs has evolved over time and so has its underlying policy, plans and implementation strategies.

Purpose/Objective:

The objective of the evaluation is to understand whether the CHW program has achieved its intended objectives, thus contributing to the overarching objectives defined in the HSSP III of improving the health status of the population by “Ensuring universal accessibility of quality health services for all Rwandans”.

This evaluation has focused on CHWs, who are selected, trained and deployed by the MoH to deliver a defined set of tasks at community level. CHWs are the central element of the Community Health Policy and of the community health strategy plan (CHSP) of the MoH. The evaluation has been designed to assess the program nationwide; therefore, the evaluation questions have been addressed at a national level.

In summary, the evaluation relied primarily on the design of a ToC, and on the exercise of testing key assumptions of the ToC via various methods of data collection and analysis. The study provides an independent assessment of the CHP in Rwanda against the agreed criteria: relevance, impact, effectiveness, efficiency and sustainability.

Methodology:

The evaluation used mixed-methods both formative and summative aspects evaluation in line with the Organisation for Economic Cooperation and Development (OECD)/ Development Assistance Committee (DAC) criteria for international development evaluations . The main focus of the evaluation is on learning. To that effect, a participatory approach to the evaluation is embedded in all the stages of the exercise, from design to implementation, analysis and validation of findings.

A set of evaluation questions were proposed by the MoH and UNICEF via initial ToRs, against these criteria. In light of the findings and recommendations of the Evaluability Assessment performed at inception, a theory-based approach was envisaged as the most suitable method to address the evaluation questions. Rather than addressing the traditional question ‘To what extent can a specific net impact be attributed to the intervention?’ a theory based evaluation aims at addressing the question of whether the intervention has made a difference.

Findings and Conclusions:

Relevance: The CHP is highly consistent with national policies and plans.
The evaluation notes that there is a very high relevance of the CHP approach to national development policies and plans, including Vision 2020, the manifesto for Rwanda transformation, and the Health Sector Strategic Plan III. In addition, the CHP was openly aligned with the Millennium Development Goals and its design was linked to achieving MGDs targets.
Effectiveness: The CHP is successful in delivering a set of essential interventions at scale.
There is evidence that the CHP delivers a minimum defined package of interventions at scale; and that service delivery is ensured steadily over time and consistently across different locations.
Efficiency: Costs of the CHP are in line with evidence available from other countries and from benchmarks.
At the current levels, the CHP would cost on average 3 USD per inhabitant per year in Rwanda; this would be more than 10% of the total Government Expenditure on Health, should the entire program cost be met via domestic resources.
Impact: Reconstructing and testing the theory of change has helped to unveil a number of both design issues (the model) and operational issues (its functioning) that may undermine the full potential of the CHP. Despite of such findings, which will be important to define a response plan, overall the evaluation concludes that the program relies on a solid and consistent Theory of Change, and that most activities are realized as intended. Per se, this is sufficient to conclude that the CHP contributes to enhanced health outcomes in Rwanda.

Recommendations:

  1. MoH and RBC should rapidly redesign the training model for the CHP, considering an integrated training approach as the way forward, in line with many other countries implementing CHPs; in doing so, aspects of health promotion and of respectful care should be included in the training package. Alternative forms of training (e-platforms) may be considered as options, especially for refresher trainings.
  2. Evidence from the assessment of the Rapid SMS system in Rwanda should be used by the MoH as a starting point to engage stakeholders in a comprehensive discussion on the possibility of merging the reporting systems of CHWs into a single, unified and simplified one.
  3. The Community Health Unit should move from a largely unused ‘CHP logframe’ to a lean, real time dashboard which makes use of routine data to manage the program against selected KPIs at all levels of the system.
  4. With support from partners if relevant, the MoH should assess which are the systemic bottlenecks determining an inefficient and ineffective distribution of medicines at the last mile, i.e. from facilities to CHWs.
  5. Given the short term funding constraints facing the CHP, the RBC and MoH should immediately produce a ‘business case’ for the CHP. This will be used to engage the Ministry of Finance and Economic Planning and key partners and donors in an informed dialogue aimed to explore short term financing options for the program.

Lessons Learned:

A common issue that evaluators face in assessing health programs in low and middle income countries is the lack of sufficient evidence from literature on the object under evaluation. In the case of Rwanda, LSTM could access, classify and use a considerable amount of published and unpublished papers, whilst also utilising internal reports. This leads to two considerations:

  • In future, it will be important to define a medium term research agenda for the CHP, and that the key studies performed in country by researchers address the questions defined by such agenda. This will lead to evidence-based decision making, as well as to the optimisation of resources allocated for research in Rwanda by domestic and international partners.
  • The evaluators could not find any evidence of a systematic approach to the collection, management and use of literature for decision making. In future, it will be important that the monitoring and evaluation function of the CHP takes an active role in the systematic collection and dissemination of research on the program. A database should be set in place and made openly accessible to all stakeholders, to facilitate the use of existing evidence for policy or programmatic changes.

Annexes to the Report found below:

  1. Annex 1 Terms of Reference
  2. Annex 2 Inception Report
  3. Annex 3 Ethical Approvals
  4. Annex 4 Theory of Change
  5. Annex 5 Survey Report
  6. Annex 6 Key Findings of Qualitative Research
  7. Annex 7 List of Secondary Data


 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
Rwanda

Region:
ESAR

Type:
Evaluation

Theme:
Health

Partners:
MoH, RBC

Language:
English

Sequence #:
2016/004

New enhanced search