We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Nigeria: Impact Evaluation of Water, Sanitation, and Hygiene (WASH) within the UNICEF Country Programme of Cooperation, Government of Nigeria and UNICEF, 2009-2013



Author: Jurrien Toonen, Ngozi Akwataghibe, Liezel Wolmarans and Madeleen Wegelin

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

The Nigerian situation mirrors the Sub Saharan African situation where out of an estimated population of 160 million, approximately 63 million still lack access to safe water supply and 113 million people lack access to basic sanitation facilities. Wide disparities between the six geopolitical zones and within states also worsen the situation, with the poor and more vulnerable populations at greater risk of WASH related health and non-health (such as time saving, productivity, school enrolment and attendance, gender inequality) human development problems.

The WASH programme within the UNICEF Country Programme (2009-2013) has increasing access to safe water sources, hygienic practices and improved sanitation especially in the rural areas and among vulnerable populations as its key objective. According to the Terms of Reference (TOR), the main results planned include an increase of 5% of proportion of population in focus States with access and use of improved water sources (2007 baseline); an increase of 3.5% of proportion of population in focus States with access to and use of improved sanitary and hygiene facilities (2007 baseline); an additional 800 schools have safe water sources and sanitation facilities; adoption of sustained   behaviours for water, hygiene and sanitation adopted in 2000 communities; and Nigeria certified dracunculiasis-free.

The programme has been funded with UNICEF regular resources and donor funds from the European Commission (EC), United Kingdom Agency for International Development, and a few others. The programme has provided services in  the 36 states and the Federal Capital Territory (FCT), Abuja. To date, 4,000,000 people have been documented as having gained access to improved water supply and 3,000,000 people to sanitation. More than 400 schools and 25 health centres had been provided with water supply while about 1,000 schools and more than 270 health centres have functional sanitation facilities.

Purpose/Objective:

There are two main objectives:
• To determine the effectiveness of the WASH intervention and to identify what worked well and mechanisms that made it work, in order to learn and improve effectiveness for scaling up.
• To assess the impact of the WASH interventions in order to identify opportunities to improve impact.

Methodology:

During the inception phase, an evaluability assessment was carried out by the Impact Evaluation team to determine  the extent to which the Federal Government/ UNICEF Country WASH programme (2009 -2013) could  be evaluated in a reliable and credible fashion. The main findings of the assessment are included in the report attached.

The goal of the evaluation was not only to appreciate if the intervention worked, but also how it worked: why, where and for whom. The best way to respond to this was to use a “mixed methods approach”: combining quantitative and qualitative methods.

As there was no information from a baseline study available, the evaluation  needed to beretrospective in nature – still it was asked by the Terms of Reference to identify trends in time, place and person. Quasi-experimental design was developed to simulate a ‘before and after’ approach.

For qualitative methodology, a desk study, Most Significant Change (MSC) and the Participatory Assessment of   Development (PADev) Tool were utilized. The qualitative techniques for data collection are essentially In-Depth Interviews (IDI) with purposively selected key persons and Focus Group Discussions (FGDs) with groups of different stakeholders.

For qualitative sampling, a two-stage cluster design to select a representative sample of around 1100 households (refer to the Annex 2 in the report for sampling considerations) across the six purposively selected LGAs was followed. For the qualitative side of the evaluation, purposive sampling was employed to select appropriate respondents for IDIs and FGDs. For the IDIs, a snowballing technique was used: One or two key informants from the implementing agencies were identified for interviews, and then, they guided the researcher to the other  informants. The state and local government stakeholders played a key role in this process. A total of 22 key informant interviews were carried out.

Findings and Conclusions:

Open defecation was significantly lower in intervention areas. Still, there was no significant difference between intervention and counterfactual in diarrhoeal incidence in the household – but there was a difference in under-five children having diarrhoea. Differences in diarrhea incidence were better explained by geographical area, and wealth:  the poor were more affected.

Interviewees perceived that there is less diarrhoea since WASH conditions were introduced as an important outcome, too – even if this is not underpinned by  statistical analysis, as this determines their change of behavior: an important outcome.

The same counts for time allocation: in the perception of the interviewees less time was lost because of the WASH interventions, so more time became available for e.g. schooling. This was not always confirmed by our statistical analysis. Indeed missing school was lower in counterfactual areas.

Empowering women and younger people was explained by the interviewees as an important outcome. They participated more in development activities, which they attributed to the WASH intervention.

Read more about findings in the attached report.

Recommendations:

Recommendations have been categorized into; building an evidence based; Approach and; financial sustainability.

Building an evidence base
•In order to measure impact, it is important to do a “before-after” study; as such a baseline study is essential. Relying on a “with and without” study only has its limitations because identifying counterfactuals for intervention areas is difficult and may be hampered by sources of bias.
•Adapt the indicators of the program’s M&E tool to those of the baseline, in order to have management information at the project’s disposal at all times
•More systematic approach to monitoring done by WASHCOMs, including training on analysis and consequent actions - so they know why they are monitoring and what happens with the results.
•Create accountability structures between providers and clients involved in WASH services

Approach
•School sanitation to be taken as a first action of the village so that youth becomes a driver of change and  they also get proper latrines and water in schools - explore links with education sector for hardware funding
•Establish more clear distribution of roles and responsibilities between community representatives,WASHCOM and LGA – eventually through a contracting approach
•Develop a strategy to involve the private sector more in maintenance and repair of WASH infrastructure

Financial sustainability
•Not all who have entered an ODF process will receive support from the WASH program, because of a lack  of     funding. It is better not to start a triggering process if support (financial, technical, logistical) support is not guaranteed. Ensure funding for the total WASH program (including water) when starting CLTS
•Ensure co-payment and/or revolving fund system to sustain the existing WASH infrastructure.
•Discuss options for regular user fees for water as this system is not sustainable if pumps are really going to break down.

See report for more recommendations.

Lessons Learned:

The WASH programme has put several structures in place to strengthen systems for hygiene and sanitation promotion. One of them is the Hygiene Improvement Framework (HIF) which is a set of rational and systematic procedure used to achieve behavioural change motivation within communities and schools. Using this framework the programme    identified that the majority of issues that motivate communities toward change in poor hygiene behaviours were more social than health related. This informed the development of tools and strategies in the programme targeting   social angle of change. The HIF was used to achieve hygiene and sanitation transformation within schools and communities. The lessons learned on the field using this framework motivated a move to wider implementation leading to the   initiation of National Level dialogue on HIF – a national meeting of WASH practitioners was held for 3 days in October 2012.

A lesson stated as having been learned from the EU funded SRIP II in their 2013 report is the high level of resourcefulness at the community level that can be targeted towards more sustainable programme delivery. This is based on the vital roles natural (community) leaders have played in the implementation of Community led total sanitation (CLTS) both within their communities and in triggering other communities to change. Case studies of the CLTS process have shown the importance and effectiveness of the roles of traditional leaders as well as religious leaders in sensitizing their peers and the communities and thereby triggering change. The programme has found working with community- based organizations as more result oriented.

A good lesson is that, having women represented in the WASHCOMs resulted in more women being involved in hygiene promotion as community hygiene promoters.

Several lessons have been learnt from the evaluation. See more in the report.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Nigeria

Region:
WCARO

Theme:
Water, Sanitation, and Hygiene (WASH)

Institution:
The Royal Tropical Institute

Partners:
Federal Ministry of Water Resources and the European Union, UKAid UNICEF, LGAs and the community leaders and WASHCOM members and Rural Water Supply and Sanitation Agency.

Type:
Evaluation

Language:
English

New enhanced search