We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Bangladesh: Impact evaluation of the WASH SHEWA-B programme in Bangladesh



Author: Dr. Stephen P Luby, Dr. Leanne Unicomb, Dr. Amal K Halder and all.

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

The Sanitation, Hygiene Education and Water supply in Bangladesh (SHEWA-B), funded by DFID, was among the largest intensive hygiene, sanitation and water quality improvement programmes ever attempted in a low income country. The goal of SHEWA-B was for Bangladesh to achieve the Millennium Development Goals (MDGs) relating to water and sanitation; and make a significant contribution to the MDGs relating to under-five child mortality and gender disparities in primary schools. Approximately 21.4 million people were reached through hygiene promotion in 60 upazilas (rural sub-districts) in 16 districts of plain land Bangladesh, and in 600 Para-centres from 16 sub-districts in three Chittagong Hill Tract (CHT) including 2.4 million children in 8,800 primary schools and 1 million urban poor in 18 municipalities, between 2007 and 2012.

Purpose/Objective:

The purpose of the study is to measure programme results in terms of outputs and outcomes, and to evaluate its impact on beneficiaries' health. The primary method used was the Health Impact Study, a unique set of surveys and health surveillance among intervention and matched control areas that enabled detailed monitoring of patterns of water, sanitation and hygiene practices in the target communities.
During the first phase of the HIS, the programme focus was on urban households alone. The 2009 midline evaluation suggested that there had been much less impact on targeted behaviors and health than planned by the designers of the programme. In response, SHEWA-B implemented revamped hygiene strategies focusing on few key hygiene messages and launched a country wide mass media (television and radio) campaign starting mid-2011. The population groups expanded to include urban households and schools. The rural household assessment expanded to include investigation of the SHEWA-B intervention implementation across sub-districts and assessment of the mass media campaign launched in 2011.

Methodology:

Baseline 2007, interim 2008, midline 2009, interim 2010, endline 2012. Assessments conducted among rural households (hh), urban slums located in rural districts, and schools, with comparison to non-intervention communities or schools. 
Rural hh: Baseline, midline & endline data collected from a different set of 50 intervention & 50 control (non-intervention) randomly selected clusters at each time point. From each cluster, 17 hh with children <5 years of age enrolled. From each of the 58 intervention sub-districts, 20 starting points sampled per sub-district and 28 hh with children<5 per starting point. For qualitative studies, participants purposively selected among intervention communities.
Urban hh: At the interim assessment, 50 slums selected from 18 intervention municipalities and 50 from control slums from adjacent matching municipalities. From each slum, 10 hh selected with children<5.
Schools: In 2011: rural schools selected using the geographic starting point used for the rural midline hh survey and in 2012: schools from 50 new starting points. At both time points, 800 intervention & 600 control schools, interviewing 4 students from each school.
Structured observation to assess key behaviors and a cross sectional survey related to hygiene knowledge & practices among 41,427 hh. Anthropometry measured among children <5 years of age from cross-sectional survey hh.
Qualitative studies in intervention communities to identify the strengths & weaknesses of hygiene message delivery, to identify the primary barriers to achieving the behavior changes and to explore reasons for differential outcomes.
Surveillance: Among rural and urban hh, monthly disease burden data collected for two recall times + quarterly test of water quality (E. coli + arsenic).
In schools: cross-sectional surveys related to WASH knowledge & practices with the headmaster and students.

Findings and Conclusions:

SHEWA-B exposed a large population to many recommendations on WASH behaviors. At the end of the intervention hh were more likely to recall key messages, but very little improvements in behaviors (handwashing practices, use of latrines, observed child feces disposal, use of improved water, ownership of an improved water source, safe water storage) were observed after 5 years compared with controls. There was no difference in awareness regarding arsenic between intervention & controls, and no improvement in the proportion of hh meeting the arsenic standard. There was little measurable impact of the community hygiene promotion on behaviors. There were several changes observed in the intervention communities over time, but similar changes were consistently observed in control communities. This suggests that the observed changes were part of broader changes occurring across rural Bangladesh and were not a result of SHEWA-B. Findings suggest that mass media likely contributed to increased knowledge among the population and in turn to increased practices of target behaviors.
In contrast to the absence of discernible effect of the intervention on child diarrhea & respiratory disease in rural communities during its initial 24 months, the urban assessment after 4 years of intervention found less diarrhea & less acute respiratory infections among beneficiary children compared to the control.
Compared to control schools, intervention schools had quite small improvements in facilities over the 1 year period between 2011 & 2012. HW practice among students receiving the hardware + behavior change communication improved significantly compared to control schools, but there were no changes in practices among the schools with the behavior change communication only intervention compared to control. The impact of the programme for the entire implementation period could not be assessed due to lack of data prior to 2011.

Recommendations:

  • Design future behavior change programmes that consider the primary barriers to targeted behavior and to ensure that the intervention addresses the barriers.
  • In evaluating programmes, we recommend that UNICEF avoid over-reliance on self-reported behaviors, and consistently enrol control groups.
  • UNICEF should continue to embrace rigorous external review to determine whether changes are attributable to the intervention. UNICEF's willingness to invest in a control group prevented drawing a fundamentally flawed conclusion from the evaluation. More detailed assessment of the source of knowledge, specifically from the intervention or other sources should be included.
  • Although the results of this programme evaluation are disappointing, there was some good news. The inclusion of mass media showed promise as a means of improving behaviour and could be consider for future interventions and rigorous assessment. There were important insights gained on promoter management difficulties and the need to include intervention fidelity assessments.
  • Include baseline assessment among all intervention populations. This shortcoming was particularly evident from the assessment of WinS. In schools, the 2011 comparison of those from intervention and control areas suggested that the intervention may have had impact; however, there were minimal significant changes from 2011 to 2012, later in the intervention period.


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Bangladesh

Region:
ROSA

Theme:
WASH

Partners:
University of California, Berkeley (UCB), DFID, and other partners

Institution:
Center for Communicable Diseases (CCD), icddr,b Water, Sanitation and Hygiene Research Group (Bangladesh)

Type:
Evaluation

Language:
English

New enhanced search