We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 Zambia: Measuring the Impact of SMS-Based Interventions on VMMC Uptake in Lusaka Province, Zambia

Author: Kevin Leiby1, Alison Connor*1, Landry Tsague2, Crispin Sapele3, Albert Kaonga4, Joshua Kakaire2, Paul Wang1

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.


In 2012, there were 2.3 million new HIV infections, adding to the 35.3 million people living with HIV around the globe (UNAIDS, 2013). Effective prevention strategies remain paramount to achieving Millennium Development Goal 6 (MDG-6), which, in part, calls for halting the spread of HIV/AIDS (United Nations, 2000). Voluntary medical male circumcision (VMMC) has been shown to reduce the risk of heterosexual HIV transmission from women to men by around 60%.  In response to these findings, the World Health Organization (WHO) and the Joint United Nations Programme on HIV/AIDS (UNAIDS) recommended including VMMC as a component of a comprehensive HIV prevention package (WHO and UNAIDS, 2007).  National governments and international partners have since worked together to increase access to quality VMMC services around the world.
Zambia is one of 14 countries in East and Southern Africa with high HIV prevalence and low circumcision rates prioritized to expand VMMC services and increase uptake.  Around 12.7% of the adult Zambian population is HIV-positive, with over five new infections every hour. However, in 2007, only 13% of males aged 15-49 years were circumcised (CSO et al., 2009). Recognizing the promise of VMMC to prevent HIV, the Government of Zambia set national targets to circumcise 80% of its HIV-negative male population between the ages of 15-49 by performing 1,864,396 circumcisions by 2015 (Ministry of Health, 2012). Achieving this ambitious target could potentially avert 339,632 new HIV infections and save the government 2.4 billion US dollars (USD) by 2025
However, efforts to promote VMMC in Zambia have produced only modest results.  Only 340,992 VMMCs had been performed from 2008-2012, representing 18% of the 2015 target (WHO, 2013). Having greatly increased the availability of VMMC services, Zambia is shifting focus to stimulating demand since demand, not supply, is now the major bottleneck hindering Zambia’s VMMC efforts


This RCT had one primary aim and four secondary aims.
The primary aim was to:Measure the impact of the two campaign strategies on VMMC uptake.   

Secondary aims were to measure intermediate outcomes and understand campaign effectiveness:

  1. Measure the impact of the two campaign strategies on self-reported intention to receive VMMC among Zambia U-Report participants in Lusaka Province. 
  2. Measure the impact of campaign messages on engagement with U-Report counselors.  
  3. Obtain qualitative insights on SMS-based VMMC behavior change to guide potential at-scale operations or program modification. 
  4. Map intervention costs and cost-effectiveness.

This evaluation was designed to answer the question: What are the impacts of two different SMS-based campaign strategies on VMMC uptake?


The study was designed as a three-armed randomized trial, as shown in Figure 5. The final sample size was ultimately limited by availability of eligible participants and responsiveness to the baseline screening survey. Sample size calculations to measure a 3% effect size at 80% power in the case of 2% control group uptake determined a target sample of 2,550 participants (accounting for projected 20% attrition). However, only 2,312 participants met the original eligibility criteria and were enrolled in the study, and 1,652 remained after attrition and revealed ineligibility. At 80% power, the available sample size enabled us to detect a similar effect size of slightly less than 3% at the level of control arm uptake.

Data were collected from five sources: 1) existing U-Report platform data, 2) SMS surveys, 3) facility records, 4) semi-structured qualitative interviews, and 5) UNICEF budgets and cost estimates.

The study’s research protocol was approved by the ERES Converge IRB Board in Lusaka, Zambia. The approval permitted a sample with participants younger than 18 years of age. Government authorization to implement the study and to collect limited client data was received from the Zambia Ministry of Health, the Zambia Ministry of Community Development, the Lusaka Provincial Medical Office, the Lusaka District Health Office, and the Chongwe District Health Office.


The evaluation results provide numerous takeaways for policymakers, program implementers, and researchers. 

Because of the study’s null results, we do not recommend that the U-Report platform be used to scale-up either of the campaigns evaluated. Nevertheless, U-Report’s administrators as well as other parties may still be interested in continuing to explore SMSbased tools for VMMC demand generation. 

  1. Low-cost implementation of SMS campaigns promoting VMMC should be prioritized 
    The five-month campaigns evaluated did not have detectable effects on VMMC uptake, and if SMS campaigns nevertheless have potential to move participants to uptake, any impact would probably be small.
  2. There appears to be value in simple prompting  Overall, interviews and messages to counselors suggested that participants were aware of VMMC and already had basic information.  
  3. SMS tools may have more potential when linked with counseling services and other interventions 
    The value of the SMS campaigns appeared to be largely mediated through counselor interactions. Indeed, campaigns spurred high levels of counselor engagement, thoughthat engagement and demand for information was not translated into VMMC uptake.
  4. Policymakers should consider potential value in structured, SMS-based postoperative engagement 
    Many participants engaged U-Report only after they went for circumcision and revealed demand for post-operative information and counseling.  

This evaluation provided the first narrow examination of a focused, sustained campaign on Zambia U-Report. As U-Report policymakers consider the platform’s broader value and cost-effectiveness, this research should serve as a starting point. Future research should look at the platform’s impact in other service areas and produce evidence that can be used to constantly improve its offerings.

Lessons Learned:

Several lessons from campaign and research implementation could be useful for others: 

  • Staggered roll-out of campaigns and surveys could alleviate pressure on server and counselor capacity. Although the study was implemented without major problems, the large baseline survey began to run up against system capacities. Simple measures would circumvent this challenge. 
  • Airtime incentives can boost response rates for SMS research surveys. Fears around negative effects on future engagement have not been realized at notable levels, but effects of incentives can continue to be monitored.  
  • Verifiability was essential to the study’s design. Self-reported data was even more unreliable than expected and would be hard to interpret without some degree of verifiability.  
  • Recruitment efforts should be directed where they have greatest returns. The potential of SMS-based behavior change rests in part on low-cost scalability, and recruitment is a major fixed cost. From this project’s experience, rural recruitment may only be cost-effective if efforts leverage large gatherings of youth (e.g. community events). 
  • VMMC referral should be focused on facilities providing the most reliable services. Many sites in the study area provided variable services over the course of the evaluation. For urban areas with many sites offering VMMC, referral can be prioritized to sites that provide consistent services on a weekly basis over time. Counselors can still provide information on other sites as requested.

Full report in PDF

PDF files require Acrobat Reader.



Report information




HIV/AIDS - Young People


3CHAMP, 4Ministry of Community Development, Mother and Child Health

1IDinsight, 2UNICEF, 3CHAMP, 4Ministry of Community Development, Mother and Child Health


New enhanced search