We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 India: Evaluation of Pilot DHaAL Interventions for Group Handwashing with Soap Before Mid-Day Meals in Rural Elementary Schools of Assam 2013-14



Author: NEERMAN

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as 'Part 4'.

Background:

Handwashing with soap is considered one of the most cost-effective interventions to reduce the burden of waterborne diseases (Fewtrell et al. 2005; Waddington et al. 2009). Under its Water-Sanitation-Hygiene (WASH) in School (WinS) portfolio, UNICEF Assam implemented a pilot group handwashing programme called Daily Handwashing for an Ailment-Free Life (DHaAL), in collaboration with Sarva Shiksha Abhiyan (SSA), Government of Assam, and a local implementation partner, the Centre for Environmental Education (CEE), North East Cell. DHaAL sought to institutionalise handwashing in school settings by advocating with the state WinS Task Force and SSA, creating an enabling environment, installing group handwashing stations, making provisions for soap, and sensitising and training SSA block and cluster staff, teachers, School Management Committee (SMC) members, Mothers Group members, and students. From July 2013 to July 2014, the DHaAL pilot was implemented in 100 rural elementary public schools from Rani and Rampur blocks in Kamrup district with funding of ₹2.75 million from UNICEF.

Purpose/Objective:

The primary objectives of this evaluation were to:

  • Assess the impact, effectiveness, efficiency and sustainability of the DHaAL pilot, based on OECD-DAC1 criteria 
  • Generate relevant lessons and recommendations for sustaining and scaling up DHaAL elsewhere in Assam and across India

Findings from this evaluation now will be used to advocate for scaling up of DHaAL with national and state governments alike; to contribute to the thin sectoral knowledge base with regard to the efficacy of handwashing interventions; and to justify and inform an eventual, more rigorous evaluation of health and education outcomes of group handwashing with soap initiatives elsewhere in India.

Methodology:

The evaluation questions and methods were extensively discussed and finalised with an Evaluation Reference Group comprising UNICEF, SSA, Government of Assam and CEE members. In addition, this evaluation follows United Nations Evaluation Group (UNEG) Norms and Standards as well as the UNEG Ethical Guidelines for Evaluation.2

A mixed-method approach was used to evaluate the DHaAL pilot, based on OECD-DAC criteria. Qualitative research tools include document and literature reviews, in-depth interviews and focus group discussions, and school observations. In particular, this evaluation included 32 in-depth interviews and 10 focus group discussions with teachers, SMC members, Mothers Group members, and SSA and CEE staff, along with informal discussions with children.

Impacts on handwashing behaviours and other intermediate outcomes attributed to DHaAL were evaluated using a quantitative, counterfactual analysis. Two variants of DHaAL intervention were assessed: (1) DHaAL implemented along with the Child Friendly School System (CFSS) programme, which aims at the holistic improvement of the school environment in terms of teachers’ capability, school infrastructure, physical education and a comprehensive focus on WinS components, of which DHaAL is a part (DHaAL+CFSS); and (2) DHaAL only, with other "business as usual" school activities and schemes (DHaAL-only). For each intervention school, a control school from adjacent blocks was chosen using regression based Propensity Score Matching (PSM) methods. The impacts or the change in an outcome attributed to the intervention were estimated by statistically comparing the outcome means between the intervention and control groups using regression models. The sample size for impact evaluation consisted of 15 DHaAL+CFSS schools, 15 DHaAL-only schools, 30 control schools, 823 children from Classes 3-5 and their parents for interviews, and 2,554 children from Classes 1-5 for handwashing observations. Nevertheless, several limitations were encountered, including (1) lack of a data-driven, and thus verifiable, logframe for the DHaAL pilot and (2) lack of adequate monitoring and evaluation (M&E) data to serve the objectives of the evaluation.

Recommendations:

Recommendations

The following recommendations, which have been discussed with the Evaluation Reference Group in person, are ordered based on the evaluation team’s perception of their urgency and importance, but UNICEF and SSA are encouraged to further determine the prioritisation of these recommendations based on their needs and circumstances.

Recommendations to UNICEF

  1. Provide technical assistance, documentation and advocacy support to SSA to implement the recommendations made to SSA
  2. 2. Consider during further programme design improvements on how schools will transfer from simple to more robust handwashing units once behaviour change is demonstrated and how this infrastructure will be maintained
  3. Make the M&E system a learning systems instead of only tracking activities and outputs and make it more data-driven system for future pilot projects
  4. Include, document and monitor gender, social inclusion and equity dimensions in programme design and monitoring, in line with UNICEF’s country priorities and commitment to gender- and equity-sensitive programming. Particularly, efforts are needed on better documentation and presentation of the evidence of the efforts actually taken on gender, social inclusion and equity dimensions
  5. Employ a rigorous experimental impact evaluation, combined with qualitative methods, to generate robust evidence on whether handwashing programs can indeed improve health and educational outcomes and how these impacts can be moderated by fidelity of program implementation and local institutional-geographical factors, and whether the impacts are indeed sustained over long term
  6. Advocate in other states the establishment of structures similar to the Assam WinS Task Force, which played an important and effective role in coordinating various Government line departments and external agencies

Lessons Learned:

Evaluation of the DHaAL pilot has generated several key lessons that also can be applied to broader WASH programmes. Most importantly, learning should be an important objective of pilot programmes, which necessitates a robust and data-driven M&E system, as well as the integration of rigorous evaluation methods at the design stage of the project. Such integration also requires systematic collaboration between programme implementation and evaluation staff or external agencies.

Moreover, when a programme such as DHaAL is implemented by using another programme as a platform, such as the MDM scheme, success of the former depends critically on success of the latter, as noted above. Such dependencies need to be considered and managed during the project design stage.

Lastly, it is appropriate that the local community and institutions such as SMCs are involved in school-based programme, but the role should be commensurate with communities’ capacities. Reliance on the community for contribution of funds or provision of technical and managerial skills for O&M of infrastructure, for example, may not be viable at scale and are better suited functions for the government machinery.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
India

Region:
ROSA

Theme:
WASH

Type:
Evaluation

Language:
English

Sequence #:
2016/002

New enhanced search