We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2010 Rwanda: Post-Introduction Evaluation of 7-valent Conjugate Pneumococcal

Executive summary


“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Created in 1978, the Expanded Programme on Immunization (EPI/Rwanda) developed rapidly, and, in 1980, became operational with six routine disease vaccinations: BCG, OPV, DPT, TT and measles. Since 1999, DPT 3 coverage has reached and exceeded 80%, until the present day. Since 2003, the proportion of health districts with a DPT3 coverage of at least 80% has only increased. The DPT1-DPT3 dropout rates are always below 10 percent.

The country eliminated neonatal tetanus by 20041. In 2002, the programme introduced the pentavalent vaccine, including DPT, hep B and Hib, thanks to GAVI Alliance support. Pneumococcal infections are an important cause of morbidity and mortality in all regions of the world. In Rwanda, a review of the fragmentary studies done at the university sites in Kigali and Butare showed that pneumococcal conditions are frequent in the country. Among children under five years of age suspected of meningitis, examination of cerebrospinal fluid showed more Streptococcus pneumoniae than N. meningitidis or Salmonella.

Given the extent and seriousness of pneumococcal infections in the country and the WHO recommendation encouraging all poor countries with an under-five mortality linked to these infections of 50 deaths per 1000 live births to introduce the pneumococcal vaccine (PCV-7), the government of Rwanda took the decision to introduce this vaccine in its national Expanded Programme on Immunization (EPI).

In February 2008, Rwanda EPI submitted a proposal to the GAVI Alliance to support pneumococcal vaccine. In May 2008, the GAVI Alliance approved Rwanda’s proposal, making Rwanda the first country in sub-Saharan Africa to obtain approval for this vaccine. From then, preparations began, with introduction of the vaccine on 25 April 2009, with a province by province approach. This introduction, starting in April 2009, was completed in
August 2009.

The objectives of this evaluation were as follows:
– Evaluate the process of introduction of the pneumococcal vaccine through the collection of qualitative and quantitative data from the central, district and health center levels;
– Summarize the strong points and the areas for improvement during the three phases (before, during and after implementation);
– Present the observations to the Ministry of Health and to partners;
– Formulate recommendations for strengthening of the programme and for improving future new vaccine introductions.

1. Selection of facilities to visit: The guide used for the evaluation proposes selection of at least three provinces, with two health districts per province (six in total) and three health centers in each district (18 health centers) in addition to the central level, which corresponds to the EPI coordination. The selection criteria also took into account the chronology of introduction (early or late), urban/rural locations, population size, EPI performance and the geographic location of the sites.

2. Data collection tools and methodology: A generic tool from the evaluation guide was adapted to the context of Rwanda and the conjugate pneumococcal vaccine in prefilled glass containing syringes (PCV-7), before the visits to the facilities selected. Thus, in addition to documentary review at all levels, the following tools have been used in the evaluation in Rwanda: (i) Standardized questionnaires have been used for interviews of health personnel at central, district hospital and health center levels; (ii) Checklists for observation of vaccination practices and practices of vaccine storage; (iii) Another questionnaire served for interviews of mothers after vaccination sessions in the health centers evaluated.

3. Areas covered by the evaluation: All EPI components were reviewed in the course of the evaluation. These are: Planning and the process of introduction; Vaccination coverage and reports; Cold chain management; Vaccine management, storage and logistics; Waste management and injection safety; Monitoring and supervision; Training and knowledge of health personnel; Vaccine wastage; Adverse events following immunization (AEFI); Advocacy, communication and acceptance; Sustainability; Surveillance.

Findings and Conclusions:
I. Pre-introduction phase; Planning, Training, Cold Chain, Waste Management, and Key messages for mothers

– Revised introduction plan available
– Most health personnel trained by provinces before vaccine introduction;
– Reference documents for training available
– Cold chain equipment obtained and distributed to health facilities before introduction
– Installation of a high temperature incinerator for the destruction of glass containing syringes before introduction of PCV-7
– Key messages developed and tested to address the concerns of mothers

Areas for Improvement
– Absence of activity timelines for introduction of PCV-7 in certain health facilities
– Feedback from certain staffers showed that training was of short duration and that certain practical aspects were not included in the training.
– In one health center visited, personnel did not get training.

II. Introductory phase: Acceptance of the new vaccine, injection safety and waste management, vaccination coverage and dropout rates, data quality, management of vaccines and cold chain equipment, monitoring and supervision, knowledge of health professionals

– Most staff interviewed found the introduction of PCV-7 to be a smooth process with good preparation.
– Vaccine well accepted by parents because of key messages developed to address mothers’ concerns
– The process of collection and elimination of PCV-7 prefilled, glass containing syringes was well understood by health personnel.
– In 35% of the health centers visited, the coverage for PCV-7 was higher than Penta coverage (one year before), and in 41 % of the centers, the first to third dose dropout rate for PCV-7 was lower than that for Penta 1 – 3.
– Supervisory visits were organized in December 2009 and March 2010 in the framework of preparation for the evaluation. Feedback was given to health centers (each health center was visited at least once in the last six months).
– No vaccine stockouts, no VVM at Stage 3 or 4, and no vaccines expired in the last six months

Areas for Improvement
– The EPI director mentioned that communications between the international community and the country on new vaccines could be improved. In particular, notification of the first delivery of PCV-7 was suboptimal preventing sufficient planning by the EPI staff.
– Weak quality of vaccination data (coverage above 100% and negative dropout rates).
– Poor knowledge of the target population (use of different sources of demographic data).
– In most health centers, data are not analyzed
– Interpersonal communication at the immunization session between vaccinators and mothers did not take place in all facilities.
– Feedback not documented in some health centers Moreover, recommendations not implemented because the written feedback was in a language unknown to those supervised
– Vaccine wastage not calculated in some health centers
– Absence of vaccine wastage reports and/or incomplete reports
– Placement of vaccines which cannot be frozen (PCV-7, Pentavalent, TT) at the bottom of top opening refrigerators
– Use of frozen icepacks for vaccine transport
– For lack of training, not all health professionals are familiar with this tool.
– A few unsafe injections were observed during

Post-introduction phase: Surveillance and monitoring of AEFI
– AEFI surveillance protocol described in the fiches techniques of EPI.
– Investigation report forms available at the health center level.

Areas for Improvement
– No case of AEFI reported. There is no system of zero reporting if no AEFI cases are seen during a reporting period.
– Weak syndromic surveillance system in place for pneumococcal related sicknesses with standard case definitions

Vaccine coverage and management
– Provide updated vaccine coverage wall charts
– Explore ways to provide a more accurate target population for coverage calculation
– Analyze data for vaccine coverage, drop-out rates, wastage; implement self-evaluation of data quality
– Assess proper documentation for PCV receipt

Training, supervision
– Provide supportive supervision or retraining on AEFI procedures and forms, safe practices for mixing vaccine components, use of Fridge Tags®, vaccine wastage reporting, use of vaccine management tools, shake test, open vial policy, conditioning of ice packs for vaccine transport, data management, and interpersonal communication of key messages at immunization table

– For communication, explore opportunities to incorporate corresponding messages on disease prevention, recognition of warning signs, improved care seeking and home treatment into vaccination activities and tools

– Conduct an assessment of surveillance for vaccine preventable diseases including pneumonia, meningitis, and diarrhea in preparation for new vaccine introduction.
– Consider impact study of PCV7 or next generation PCV to document the effect on the vaccine on disease for decision making for vaccine sustainability.

Cold chain
– Conduct a cold chain assessment to determine current capacity and plan for introduction of rotavirus (RV) vaccine and higher valency PCV formulation in multi-dose vial

Lessons Learned (Optional):
1. New vaccine introduction requires a lead time of at least 12 months, a decentralized system of primary health care, and staff trained and responsible at the operational level.
2. New vaccine introduction can require an increase in vaccine storage capacity. To do this, it is indispensable to organize an evaluation of the cold chain storage capacity, to identify the real needs and to determine the gaps which must be closed to accommodate the new vaccine.
3. New vaccine introduction requires prior training of all personnel participating in vaccination activities, not only for the vaccine in question but also for all technical areas of vaccination to insure quality services.
4. To assure parental acceptance, new vaccine introduction requires the development of key message which take into account parental concerns.
5. Taking account of the high costs of new vaccines, advocacy with the political authorities is indispensable to securing new vaccines (co financing).

Full report in PDF

PDF files require Acrobat Reader.



Report information

New enhanced search