We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2013 Botswana: Post Introduction Evaluation of Pneumococcal and Rotarix Vaccines in Botswana


Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report."


An estimated 8.8 million global annual deaths occurred amongst children under 5 years of age in 2008. Out of the deaths, WHO estimated that 476 000 (333 000–529 000) were caused by pneumococcal infections. Disease rates and mortality are higher in developing than in industrialized settings, with the majority of deaths occurring in Africa and Asia.
According to WHO estimates of severe illness cases and deaths in children from one month to less than five years of age due to Streptococcus pneumoniae, 10,540 cases and 894 deaths were reported in Botswana in 2000.

WHO estimates that in 2008, approximately 453 000 (420 000–494 000) rotavirus gastroenteritis (RVGE)-associated child deaths occurred worldwide (updated WHO estimates on global mortality due to RVGE are soon to be published). These fatalities accounted for about 5% of all child deaths and a cause-specific mortality rate of 86 deaths per 100 000 population aged < 5 years. In 2011, WHO estimated that, between 57 and 78 (working estimate of 67) child deaths due to rotavirus infection occurred in Botswana in 2008.

Routine immunisation coverage has remained relatively high in Botswana; For instance, WHO and UNICEF estimate that Pentavalent-3 coverage for 2012 was 96%. However, the country has been facing denominator problems for some time now and this makes it difficult to ascertain the true coverage estimates and monitor progress. Rectification of this long standing problem is even more pertinent now as expensive life-saving vaccines are added to the immunisation programme.

The country introduced Monovalent Hepatitis B (HepB) in 1995, followed by pentavalent DTP-HepB-Hib in a fully liquid formulation in 1-dose vial presentation in November 2010, while Pneumococcal 13-valent (PCV13) and Rotarix vaccines, in fully liquid formulations in 1-dose presentations were introduced in July 2012.


The post Introduction Evaluation (PIE) was conducted in accordance with the World health Organisation (WHO) recommendation that all countries that have introduced a new vaccine should conduct a PIE, 6-12 months post-introduction, to assess the overall impact of the introduction of new vaccine(s) on a country national immunisation programme. Botswana was therefore due to conduct a PIE for the two vaccines (pneumococcal and rotavirus) which were introduced in July 2012.

The main objective of the evaluation was to assess the overall impact of the new vaccine introduction on the national immunisation programme.

The PIE specific objectives were:
1. To identify, document and rectify any programmatic and logistical difficulties relating to the introduction of pneumococcal and rotavirus vaccines.
2. To evaluate the incremental costs of introducing new vaccines
3. To provide lessons learned for similar experiences in future and for future countries that may introduce new vaccines.


The WHO guidelines for performing a PIE were used to assess the overall impact of the introduction of a new vaccine on the national immunization programme in Botswana.
The key elements of the evaluation included:
• Pre-implementation Planning and Vaccine Introduction Process
• Training
• Vaccine Coverage
• Cold Chain Management
• Vaccine Management, Transport and Logistics
• Waste Management and Injection Safety
• Monitoring and Supervision
• Adverse Events Following Immunisation
• Advocacy and Communication
• Surveillance
• Sustainability (programme financing)

The evaluation exercise consisted of:
• Orientation workshop for national evaluators and adaptation of tools
• Interviews with national, district and health facility level staff using a standardized questionnaire
• Exit interviews were conducted for mothers and observation on immunisation practices and cold stores done at the sampled health facilities
• Visits to the national and district vaccine stores
• Visits to 18 health facilities across the country for observation and record review using standardized questionnaires and checklists. Six districts were visited including 3 health facilities in each district. The choice of the health facilities took into account the following criteria: high performance; medium performance; low performance; and hard to reach.
• Collected data was compiled and analysed
• Debriefing of the Inter agency Coordinating Committee

The evaluators were drawn from:
1. AFRO/Inter-country Support Team for East and South Africa (IST/ESA)
2. WHO and UNICEF Country Office staff in Botswana
3. Ministry of Health officials from Botswana

The PIE was co-sponsored by WHO/IST/ESA and UNICEF ESARO

Findings and Conclusions:

The PIE identified numerous strengths that the EPI Programme should maintain and improve upon as well some weaknesses that should be addressed in order to strengthen programme performance and immunisation service delivery in the country. The key issues to be addressed included the following: target population and coverage figures; vaccine stock outs; communication and social mobilisation; and programme financing. The other issues were lack of introduction plans at sub-national levels and delayed introduction in two districts; short duration for training and general absence of training materials in districts; lack of evidence of using data for action; frequent interruption of power supply in the country and lack of temperature monitoring in the districts; lack of vaccine management guidelines at all levels; non-use of auto-disable (AD) syringes for immunisation services in the country; inadequate support supervision; lack of a written protocol for adverse events following immunisation (AEFI) and inadequate reporting; failure to conduct launch ceremonies at national level and in many districts; and cessation of PBM sentinel surveillance.

The PIE identified numerous strengths that the EPI Programme should maintain and improve upon as well as some weaknesses that should be addressed in order to strengthen programme performance and immunisation service delivery in the country. A number of recommendations were made in order to address the identified weak areas

Detailed findings of the evaluation providing strengths and weaknesses for each are assessed and for each of the district can be found in the findings section of the full report.


-Pre-implementation Planning and Vaccine Introduction: Provide adequate reference materials to districts for guidance; Update recording and reporting tools; Develop introduction plan based on the national plan to guide district implementation.
-Training: Provide regular refresher training and increase duration of training at sub-national levels; Provide practical demonstrations during training.
-Vaccine Coverage: Provide updated target population figures for the subnational levels; Retrain districts on RED and DQS and support implementation.
-Cold Chain Management: Monitor vaccine fridge temperatures in districts and HFs twice daily including weekends; Use freeze watch/freeze tags during vaccine transportation.
-Vaccine Management, Transport and Logistics: Conduct refresher training on vaccine management at subnational levels; Ensure that all expired vaccines are handed over to the relevant authorities for destruction.
-Waste Management and Injection Safety: Provide adequate logistics for safe disposal of injection waste and train staff on injection safety and safe waste disposal including providing AD syringes for immunisation services
-Monitoring and Supervision: Ensure that all planned supportive supervision visits are conducted and documented and follow up on identified weaknesses for corrective actions.
-Adverse Events Following Immunisation: Develop and provide written AEFI protocol to all HFs and monitor AEFI surveillance.
-Advocacy and Communication: Conduct national and district launches for new vaccines and supply adequate IEC materials to sub-national levels; Train health workers on interpersonal communication.
-Surveillance: Find long term solutions to revamp Pneumonia & Bacterial Meningitis sentinel surveillance and establish Rotavirus sentinel surveillance.
-Sustainability: Track information on programme to meet GVAP reporting requirements to the World Health Assembly

Complete recommendations available in the full report attached.

Lessons Learned:

1.Availability of local evidence of disease burden, strong political will and effective stakeholder engagement are necessary requisites for introduction of new vaccines
2. Adequate planning, mobilisation of sufficient resources and close monitoring are critical for smooth implementation of new vaccines particularly multiple vaccines
3. Communication and social mobilisation are essential so that health workers and communities are well informed and communities mobilised for immunisation services and new vaccines in particular
4. Use of Balanced Score Card in MOH at national level to monitor immunisation performance at sub-national levels on a quarterly basis is a good practice.


Full report in PDF

PDF files require Acrobat Reader.



Report information







New enhanced search