Evaluation database

Evaluation report

2011 Global: Impact Evaluation of Drinking Water Supply and Sanitation Interventions in Mozambique



Author: UNICEF – Government of The Netherlands Partnership for Water Supply, Sanitation and Hygiene

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports.  Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it.  You will find the link to the quality rating below, labeled as ‘Part 2’ of the report.

Background:

The focus of the assessment is on the impact of interventions on the welfare of the final beneficiaries.  The ultimate purpose of the support provided to water supply, sanitation and hygiene goes beyond coverage: the support is intended to improve health, reduce the time used for collecting water -- particularly for women and girls, raise school enrolment, retention and performance of children and enable communities to improve their livelihoods.

There is consensus on the importance of such ultimate impacts on human welfare but conventional evaluation studies do not usually quantify them.
Quantification is a key characteristic of the impact study.  The study further includes an assessment of sustainability of the programme benefits.  Sustainability is defined as the (probability of) contrinuation of benefits after major development assistance has been completed.  The assessment addresses relevant institutional factors, linked as they are to society's governance and politics, as well as technical, economic and environmental determinants of sustainability.

For the Government of Mozambique and the UNICEF MOzambique Office the impact assessment is expected to provide information that will help to steer the programme and inform policy and policy implementation.  For the UNICEF central Evaluation Office the study is expected to provide an opportunity for global learning on impact evaluation of water supply and sanitation programmes.  For the Policy and Operations Evaluation Department (IOB) of the Netherlands Ministry of Foreign Affairs the impact assessment is one in a series of impact evaluations of Netherlands supported programmes for water supply and sanitation facilities.  The evaluations will provide information for a policy evaluation planned to be compleed in 2011.

Purpose/ Objective:

In 2008 the Government of Mozambique, UNICEF and the Government of the Netherlands agreed to an impact assessment at mid-term and end of the Government of Mozambique/ UNICEF Water supply, Sanitation and Hygiene programme -- the One Million Initiative -- Mozambique.  A substantial part of Netherlands development funding for water supply and sanitation goes to UNICEF programmes.  The ne Million Initiative is the largest Netherlands supported programme.  Programme implementation started in September 2006 and will end in December 2013.  The current report is on the mid-term impact assessment.
The impact evaluation is jointly executed by the Policy and Operations Evaluation Department (IOB) of the Netherlands Ministry of Foreign Affairs, Ms. R. Tesselaar and the central Evaluation Department of UNICEF, Mr. S. Bickel, with major support of the water supply and sanitation team of the UNICEF Mozambique Office and the Government of Mozambique water sector authorities.  Technical services for evaluation design, data collection and analysis have been provided by Professors J.W. Gunning and C. Elbers of the Amsterdam Institute for International Development, a research team of WE Consultant based in Maputo, and freelance consultants Dr. S. Turner and Mr. B. Matabire.

Within the framework of a partnership between the Government of Netherlands, UNICEF and the Government of Mozambique, which aims at accelerating the achievement of the Millennium Development Goals (MDG) with respect to the access to drinking water and adequate sanitation for families and school communities, the Water Supply, Sanitation and Hygiene Programme -- the One Million Initiative -- is being implemented in 18 districts of the provinces Manica (Gandola, Guro, Machaze, Manica, Mossurize and Sussundenga), Sofala *Chemba, Dondo, Gorongosa, Maringue, Muanza and Nhamatanda) and Tete (Angonia, Changara, Chifunde, Tsangano, Maravia and Zumbo).  As a result of this partnership it is expected that

One million people in the rural areas use safe drinking water, through the construction of new sources of water supply;
200,000 people use safe drinkign water, through the rehabilitation of their sources of water supply;
One million people use adequate sanitation facilities;
1.2 million people adopt appropriate hygienic practices;
400 primary schools (with a total of 140,000 pupils) use infrastructures of drinking water, sanitation and hygiene;

18 districts and 3 provinces have strengthened technical and management capacities for the planning, coordination and implementation of programmes for water supply, sanitation and hygiene education.
The total budget for the programme is EUR 32.64 mln, of which 65% is provided by the government of the Netherlands, 19% by UNICEF, 13% by the Government of Mozambique and 3% by beneficiary contributions.  The programme is by far the largest in the provinces and is spearheading rural water supply policy implementations.

The One Million Initiative is implemented as part of the Netherlands and UNICEF, partnership to enhance water supply, sanitation and hygiene in eastern and southern Africa.

Unusually, the programme proposal presents two logical frameworks with separate statements of overall objective and project purpse.  The first is for the 'technical component' of the proramme; the second, for the 'institutional component'.

This design approach is not carried through to the main body of the proposal, however.  There, five component are presentated: rural sanitation, rural water supply, hygiene education, school sanitation and hygiene education, and community-based water resource management.

Methodology:

The sample and data collection.
The methodology for the impact study entails a combination of quantitative and qualitative methods and techniques.  Impact is measured by comparing changes in impact variables over time and between locations with and without programme interventions.  The main data collection techniques comprise sample based surveys conducted in 2008 and 2010, water quality tests, field observations and itnerviews with key informants.

In order to obtain more direct measurements of the impact of the One Million Initiative, a sample was drawn of 80 communities from 9 of the 18 districts covered by the Initiative.
Details about the sampling procedure can be found in Chapter 4.

Half of the sample, or 40 communities, is representative of the general population of the 9 sample districts.  The other 40 communities were drawn from a poorer target section of the population.  When the sample was drawn this part of the population ws expected to get relatively more new safe water points and be more intensively affected by the One Million Initiative.  Thus, the complete sample of 80 communities has intentionally oversamled 'treatment communities' in order to get a better picture of activities under the One Million Initiative and achieve greater statistical precision when comparing (high intensity) treatment to no treatment (or low intensity treatment) communities.

For each of the sample communities a number of surveys have been carried out: a household sample was conducted among 20 households selected (by systematic sampling) from a randomly chosen contiguous group of approximately 100 households (or 500 persons).  This survey covers, besides general household characteristics, health and water and sanitaiton practices.  In addition, a focus group discussion and water point survey was conducted in the neighbourhood where the 20 sample households live.  Water samples were taken at a selection of households (10% sample) and at the water sources used by these households to test for microbiological contamination.  Also, information from health posts close to the sampled location has been collected that can be lnked to the various samples.  This information includes the number of recorded cases of particular diseases and disease-specific mortality data.

A separate survey has been conducted among 80 schools from the 2 programme districts participating in the Child-Friendly Schools for Africa Initiative (CFS), see page 54 in Chapter 3.
These districts are not covered by the other surveys.  All sample households, schools and communities have been visited twice, in 2008 and 2010, and will be visited again in 2013.

Findings and Conclusions:

Main findings and issues.
In 2008 the Government of Mozambique, UNICEF and the Government of the Netherlands agreed to an impact assessment at mid-term and at the end of the Netherlands supported UNICEF Water Supply, Sanitation and Hygiene programme -- the One Million Initiative -- in Mozambique.  This report is on the mid-term impact assessment.

The impact evaluation is a joint responsibility of the Policy and Operations Evaluation Department (IOB) of the Netherlands MInistry of Foreign Affairs and the central Evaluation Department of UNICEF.  The evaluation is impemented with major support from the UNICEF Mozambique Office and the Government of Mozambique water sector authorities.

Within the framework of a partnership between the Government of Netherlands, UNICEF and the Government of Mozambique, which aims at supporting the achievement of the Millennium Development Goals with respect to drinking water and adequate sanitation for families and schools, the Water Supply, Sanitation and Hygiene Programme -- known as the One Million Initiative -- is being implemented in 18 districts of Manica, Sofala and Tete provinces.  The programme is to run for seven years (September 2006 - December 2013).  It is expected that by the end of the programme

one million people in the rural areas use safe drinking water, through the construction of new sources of water supply;
200,000 people use safe drinking water, through the rehabilitation of their sources of water supply;
one million people use adequate sanitation facilities;
1.2 million people adopt appropriate hygiene practices;
400 primary schools (with a total of 140,000 pupils) use appropriate drinking water, sanitation and hygiene facilities;

18 districts and three provinces have strengthened technical and management capacities for the planning, coordination and implementation of programmes for water supply, sanitation and hygiene education.
The total budget of the programme is EUR 32.64 mln, of which 65% is provided by the Government of the Netherlands, 19% by UNICEF, 13% by the Government of Mozambique and 3% by beneficiaries.  The programme implementation strategies are aligned with the national water policy, which places priority on meeting the basic needs of the disadvantaged, on decentralised management and on the participation of users.

The programme is an important reference for the National Water Supply and Sanitation Programme (PRONASR), which is now in its initial phase of implementation.  Its approach is participatory and demand responsive, with user communities and schools expected to take leadership and responsibility for the maintenance and management of their imporved facilities and behavioural change, supported by Government, NGOs and the private sector.  The main water supply technology applied is a borehole fitted with a hand pump.  An important component is the engagement of local NGOs to carryout promotion activities in the targetted districts to build demand for improved services, as well as capacity to sustain services and strenghten the supply side for the construction of latrines and maintenance and repair of water points.  The One Million Initiative revised its participator sanitation and hygiene transormation (PHAST) strategy cy merging education components with a community-led total sanitation approach, amended to a community approach for total sanitation (CATS).  The implementation of water, sanitation and hygiene activities in the target provinces is complemented by the development and strengthening of government capacities at provincial and district level in order to ensure long-term sustainability and interventions.

UNICEF Mozambique Office has reported that by the end of 2009 the One Million Initiative had rehabilitated a total of 392 water points, completing this elements of its work plan (UNICEF, 2010a).  By the end of 2010 it had built 994 new water points.  The number of users enumerated at these water points totalled 967744, implying an average of 974 users per water point, being stil far above the national planning standard of 500 users per water point.  The number of users calculated on the basis of this standard would be 489500.  On both accounts the figures indicate that the programme has made good progress in achieving its target number of beneficiaries.

The 2010 programme progress report further indicates that following the replacement of the PHAST approach with a community approach to total sanitation (CATS), the programme has nearly reached its target of one million people using sanitation facilities.  Over and above these planned targets, 433 communities with a total population of 349,243 have so far reported to be declared open defacation free (ODF).  The performance of the schools component has been slower, although (again moving beyond the original strategy) 362 schools had been declared ODF to November 2010.  The sanitation facilities created are mostly simple pit latrines made of local materials.  The programme has reduced borehole construction costs from EUR 10,772 in 2008 to EUR 5,255 in 2010, although work in more difficult areas in 2011 may drive average costs up again.  The CATS approach to sanitation involving total communities has significantly reduced the average cost of latrine construction, which is now estimated at EUR 7.62.

The overall purpose of this impact assessment is to account for support provided and to be a reference for programme management and for policy development.  The focus of the assessment is on the impact of interventions on the welfare of the final beneficiaries.  The ultimate purpose of the support provided goes beyond access to facilities; the support is intended to improve health, reduce the time used for collecting water -- particularly for women and girls -- raise school enrolment, retention and performance of children and enable beneficiaries to improve their livelihoods.  There is consensus on the importance of such impacts but conventional evaluation studeis do not usually quantify them.  The study further includes an assessment of the sustainability of the programme benefits.  A mid-term assessment of sustainability may seem premature.  However, experience suggests that a number of factors commonly influence the sustainability of rural water supply and sanitation interventions and that it is useful to identify their presence or absence at an early stage of strategy implementation.

The methodology for the impact study entails a combination of quantitative and qualitative methods and techniques.  Impact is measured by comparing changes in impact variables over time and between locations with and without programme interventions.  The main data collection techniques compromise sample based surveys, conducted in 2008 and 2010, water quality tests, field observation and interviews with key informants.

 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search