We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Rwanda: Rwanda RapidSMS Impact Evaluation



Author: Hinda Ruton, Angele Musabyimana

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

In line with its socio economic development, Rwanda has achieved remarkable progress in improving maternal and child health outcomes. According to recent (RDHS 2014-15), the under-5 mortality rate has declined from 152 per 1,000 live births in 2005 (Rwanda Demographic Health Survey (RDHS) 2005) to 50 per 1,000 live births in 2014; the neonatal mortality rate has also reduced from 37 per 1,000 live births to 20 per 1,000 live births. The maternal mortality ratio (MMR) has also successfully reduced: MMR was estimated at 750 per 100,000 live births in 2005, and at 210 per 1,000 live births in 2014-15 (RDHS 2014-15).
As mobile technologies have become widespread in developing countries, they present an opportunity for new methods of communication. RapidSMS is a free and open source platform for mobile system built in Python and Django, which was customized for application to maternal and child health in Rwanda.
In Rwanda, CHWs are equipped with mobile phones to enable them to collect and use real-time data on key maternal, neonatal and child health indicators. This is collected during the first 1000 days of life (pregnancy until birth, and newborn until 2 years), and includes a broad range of areas: antenatal care, delivery, post-natal care, growth monitoring, as well as death indicators such as maternal and neonatal and child mortality. The real-time indicators are recorded using RapidSMS. The system then generates automatic reminders for clinical appointments, delivery, and post-natal care visits, with the intent of increasing attendance at antenatal care and postnatal care visits. Additional intended outcomes include the provision of a quick link to emergency obstetric care through so-called Red Alerts, and the creation of a database of clinical records on maternal care delivery.

Purpose/Objective:

This evaluation was commissioned by UNICEF and the Ministry of Health to assess the progress and impact of the RapidSMS system in Rwanda over the first three years after it had been nationally scaled-up. The overall objective of this evaluation is thus to understand whether the intended objectives of the project have been achieved, in line with the plan. Basically to document, analyze and disseminate the latest RapidSMS project experience for future use in programming. The evaluation looked at programmatic achievements and constraints, to explore and understand the reasons behind those. The findings and recommendations of this formative evaluation will be used for improving design of the program, planning, technical guidance and to enhancing implementation of innovative RapidSMS implementation.

Methodology:

The study used a mixed methods approach to provide key information on the RapidSMS program.  The quantitative analysis used a rigorous quasi-experimental approach using program data from the RapidSMS database, and administrative data from the Rwandan Health Management Information System (HMIS). The qualitative study leveraged key informant interviews from stakeholders and program recipients along with focus groups of CHWs, mothers, and fathers. Using comprehensive databases and both quantitative and qualitative research techniques, it was conducted an integrated set of analyses to respond to the main objectives of this evaluation. Drawing from the Theory of Change, the policy analysis focused on the following OECD/DAC evaluation criteria of relevance, effectiveness, impact, efficiency & Coordination and equity.

Findings and Conclusions:

As is clear from the documents we reviewed and the views of interview participants, the intent of the RapidSMS system was to improve health outcomes for mothers and children in Rwanda. Overall, the evaluation found the program was relevant to national priorities and well used. The quantitative results, however, found that RapidSMS only contributed to some changes in the use of particular health care services in supported Districts.

Recommendations:

General

  1. Focus on specific priority outcomes
  2. Develop a sustainability plan           
  3. Consider the use of other UNICEF management support programs

Technical Aspects

  1. Improve data validation
  2. Study direct messaging
  3. Ensure that CHWs can access functional phone and charging equipment
  4. Refine the red alert system

Program Operation

  1. Improve CHW training programs
  2. Consider reimbursement for messaging performance          
  3. Ensure CHWs have the necessary equipment to effectively use RapidSMS   
  4. Improve the sharing of RapidSMS data with health providers and district level staff

Lessons Learned:

  1. The most important lessons learned are likely related to our finding that the impact of RapidSMS was concentrated in those Districts for which UNICEF provided additional support. This suggests two key lessons. First, it suggests that mHealth initiatives such as RapidSMS will only function in jurisdictions that have the necessary infrastructure and equipment in place. The key example in this instance was malnutrition screening, where the supported districts received the necessary equipment and saw large increases in their use.
  2. Second, our finding of an effect concentrated in supported Districts suggests that simply enacting mHealth system is not sufficient to assure its use. As the UNICEF support included embedding an individual in each District and conducting regular trainings and meetings, it seems likely that this activity was necessary in order to see the benefits of the program. This was clearly evidenced by the much higher rate of pregnancy registrations and use of RED Alert messages in the supported Districts, for example.
  3. We heard from several CHWs and their supervisors that the mobile phones provided at the start of the program have been lost or broken. In terms of long-term planning to support a technology-focused program such as RapidSMS, turnover and replacement of devices should be planned from the outset.
  4. The low use of the RED Alert system suggests that any technology platform intended for use during emergencies needs to be simple and fast to use. The fact that CHWs had to use code cards and input long National ID numbers were likely key factors in the very low use of this notification system. In addition, the reported lack of response by emergency services for many RED Alerts likely acted to prevent the use of the system as well.

              


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
Rwanda

Region:
ESAR

Type:
Evaluation

Theme:
Health

Partners:
MoH, RBC and KOICA

Language:
English

Sequence #:
2016/003

New enhanced search