2013 Burundi: Evaluation of the WaSH project in the integrated rural villages of Rutana
Author: Lucie Goulet and Julie Rodier
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.
This report presents the findings of an end of term evaluation of the UNICEF WaSH initiative entitled “Appui à la réinsertion des populations vulnérables par l’accès aux services sociaux de base, notamment l’eau, l’assainissement et l’hygiène”, implemented in 2 provinces in Burundi, between 2010 and 2013 under funding from the European Union. This project was one of three components of a larger reintegration project for former refugees done in collaboration with UNDP and FAO as UN agencies responsible for livelihoods and food security respectively.
This evaluation was limited to 3 main aspects of the achievements of the water, sanitation and hygiene project in 2 of the Rural Integrated Villages (IRV), namely effectiveness, impact, and sustainability. The project evaluation was mandated by UNICEF which entered into agreement with the School of Social Policy and Program Administration of Carleton University, Ottawa, Canada for the execution of the evaluation. The preliminary research and design work on the evaluation was done between November 2012 and June 2013, while the evaluation mission took place in Burundi over 18 days in July 2013 with a team of 2 evaluators supported by UNICEF Burundi staff from the Social Policy, Advocacy and Evaluation Section and the WaSH programme.
The overall purpose of this evaluation was to assess 3 aspects of the project: effectiveness, impact and sustainability. Effectiveness looked at the extent to which the objectives were achieved, impact looked at the most significant changes due to the project as identified by beneficiaries, and sustainability assessment was focused on the likelihood of long term functionality of systems and of functionality of operating and managerial structures in view of future needs. The objectives were to assess the level of achievement of expected outcomes as well as the results attained in relation to access to clean water and improved hygiene practices in two IRVs in the Commune of Giharo in the province of Rutana in Burundi, as well as the long term viability of the results. The evaluation also aimed to indicate the perceptions of beneficiaries on the contribution of the WaSH project to social cohesion, a cross-cutting theme of the larger UN initiative.
The evaluation of the WaSH project used a non-experimental design (i.e. no comparison or control group). This design was fit due to the lack of baseline information but also meets appropriately the objectives of UNICEF for this evaluation, which are to get a measure of the effectiveness of the project implementation based on planned activities, an assessment of the impact of the WaSH project on a population that had previously no access to clean water on site, and an assessment of the likely sustainability of the project’s benefits based on findings overall. Since all IRVs in the reachable geographic area were included in the UNICEF programme, the evaluation chose to focus on the changes before and after the project within the same population. Therefore, qualitative methods were adopted in carrying out this evaluation to gather an in-depth understanding of the perspectives of various project’s primary targets (direct beneficiaries, members of water management and school hygiene committees, program staff, and staff from the implementing partner agency) and secondary stakeholders (such as representatives from the other UN agencies involved in the project and government staff at the national, district and communal levels).
Since this evaluation looked at a limited number of cases (2 IRVs) and concerned a limited percentage of the overall target population of the WaSH project, a case study design has been chosen to analyse and present findings associated with sustainability. Evaluation methods included in-depth interviews, semi-structured focus group discussions, and a conversational approach to gather and rate what beneficiaries identified as the “most significant changes” due to the project. Evaluators also used observation, informal discussions, documentary review as well as literature review in order to complement the collection of data.
Findings and Conclusions:
Based on the findings of the evaluation of effectiveness, we conclude that the overall objective of improving sustainable access to quality water, basic adequate sanitation infrastructure, and hygiene for people especially children and women was mostly attained. There was a clear improvement in the access to clean water at both the household and school levels. Since sanitation facilities were not constructed in IRVs as part of this project, the access to basic adequate sanitation infrastructure has not changed for the communities with the exception of children attending the schools in which sanitation facilities were built.
The evaluation concludes that the WaSH project reached some of its short-term outcomes and all of its medium-term outcomes. Although increase knowledge about clean water, sanitation, and hygiene was not verifiable nor identified as a significant impact, there is evidence that some knowledge transfer enabled communities to reach the medium-term outcomes.
Furthermore, the project had some important unexpected impacts on targeted IRVs and those living nearby such as time gained from closer access to water, food security and diversity, and contribution to social cohesion. This may be explained by the combined effects of the FAO and UNICEF projects impacting beneficiaries beyond the planned results of the project.
Although the project impacted on everyone, there seems to be a consensus that women were impacted the most from the changes as they often are the ones who are responsible for water, sanitation, and hygiene in the household.
The evaluation concludes that without a range of additional support by UNICEF, government partners and other skilled partners, there is a high risk that many of the benefits of the project will not be sustainable over the long term.
- Undertake some repairs and re-conditioning of water and sanitation structures: UNICEF should do a proper diagnostic of major structural risks to water and sanitation structures so as to address issues beyond community’s capacity that put the system’s functionality at risk, and jeopardize the investment made.
- Curtail the density of water points’ use: Insofar as possible, water management committees and/or the Communal water board should find ways to reduce density of use on water points.
- Ensure water points outside of the IRV are functional: UNICEF and the Régie communale de l’Eau should ensure that all water points built or rehabilitated within the context of this project are functional so as to alleviate demand on a fewer number of functioning water points.
- Clarify usage of water infrastructures at schools: UNICEF should clarify the authorization of community use of school water points with school authorities. Prolonged open use of water points will put pressure on the system and could threaten its viability for school children unless co-management responsibility is addressed.
- Add wash basins for laundry close to water points should be considered if resources allow: they would be advantageous for water use efficiency, convenience and socialization for women, and improvements of use of water points.
- Extend training and awareness raising in communities: in the near future and regularly, UNICEF in conjunction with partners should offer refresher trainings that would include providing members with some visual material they can use to sensitize the community on proper use of the water points, sanitation and hygiene’s benefits.
- Do training in schools: reiterating the rules for proper use of the water point and behavioral aspects of control, especially focusing on training young children about the value of clean water and use of sanitary facilities, is of vital importance to support early life habits.
Coordination between UN agencies is key in the design and implementation of a multi-component project to ensure the integration of its components to create a more cohesive program and to better support the objectives of the project. However, roles and responsibilities of all parties must be clear and understood from the beginning to ensure a strong governing structure and the continuity of the results.
The physical environment for settlement and infrastructures is a foundation for long-term viability. Taking the particular environment of the IRVs in consideration would greatly increase the viability of infrastructures.
Construction of water and sanitation infrastructures does not ensure a shift in habits and mentality in communities. Training and awareness raising is as crucial in enabling change.
Full consideration needs to be given to women and children needs in training and awareness raising campaigns since women have a pivotal role in use, maintenance and proper care of water facilities and in hygiene and sanitation habits while children are young and adopt behavioural changes more easily than grown-ups (less ingrained habits).
Mobility of stakeholders in the project, especially at community and communal levels, has affected knowledge retention on different important aspects of the project’s implementation and sustainability.
A lack of program planning and monitoring expertise, financial, technical and logistical means as well as well human resources make it challenging for the government to assume its role fully, especially at local levels.
Delays in implementation of the WaSH project impacted the quality and intensity of post-construction activities. The UNICEF project had not in earnest started yet at the November 2011 mid-term evaluation and despite a year extension, had little time for proper training and awareness building activities following infrastructure building.
Full report in PDF
PDF files require Acrobat Reader.