We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Malawi: Evaluation of the Water and Sanitation (WASH) Programme in Malawi (2007-2013)



Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as 'Part 3'.

Background:

This report provides the findings of the evaluation of the Government of Malawi (GoM)-UNICEF Water and Sanitation Programme in Malawi. The overall purpose of the Malawi WASH programme was to improve child health, survival rates and well-being. This purpose was to be achieved by improving sanitation, providing safe drinking water and effective hygiene education.. The programme was implemented from 1 January 2007 to 31 December 2013.

The evaluation was commissioned by UNICEF Malawi and was carried out by one external (Dutch) and one internal (Malawian) consultant. The evaluation was carried out from September to December 2015 with most of the design and preliminary work being done in August 2015.

Purpose/Objective:

The overall aim of the evaluation was to assess the degree to which the programme achieved its objectives. The evaluation set out to:

  1. pinpoint strengths and weaknesses, and identify risks and opportunities for the WASH programme;
  2. assess the extent to which the programme focused on the deprived and vulnerable, and on systems and capacity development in the sector, including for financing, and
  3. Assess the programme contribution to risk management and internal UNICEF capacity, especially in relation to upstream engagement.

Methodology:

Quantitative and qualitative methods were used to answer the evaluation questions. The design of the evaluation did not use experiential (or quasi-experiential) approaches, because of the absence of a control/comparison group and baseline information, which would have allowed a robust measurement of impact. The analysis of change was therefore largely limited to the use of secondary data on programme recipients or target populations. However, an effort was made to collect primary data using a beneficiary questionnaire for households and school-going children in the target areas. In total, 667 households and 115 school going students were consulted.

Qualitative methods were also adopted, in order to gather an in-depth understanding of the perspectives of the various primary targets of the programme (direct beneficiaries, members of Water Point Committees and Village Health and Water Committees (VHWC), area mechanics, district programme staff, and staff from the implementing partner agencies). The evaluation was carried out in three of the target districts - Mangochi, Lilongwe and Mzimba - representing the regional, geographical, economic, cultural and social range of the programme. Besides, overall observation, the evaluation also conducted a physical inspection of the WASH infrastructure (water points with hand pumps and school sanitation facilities) and scored the hygiene status at both community and school levels.

Findings and Conclusions:

The GoM-UNICEF programme was highly relevant, as it was consistent both with UNICEF goals and the national policy. The relatively low coverage of water and sanitation facilities in the target areas added to the programme relevance. A weak point in terms of relevance is the rather poor and confusing log-frame, which features outcomes rather than easily measurable outputs. The performance related to relevance is satisfactory.

The efficiency of the programme leaves much to be desired. The programme reached and even surpassed its targets within the timeframe and budget, but short-cuts were made in supervision and contracting, which compromised the quality of construction. Other serious shortfalls include poor data management by UNICEF and, in particular, the lack of M&E in the districts. And so, one may conclude that UNICEF focussed too much on achieving the physical targets instead of concentrating on supervision and on strengthening the capacity of the implementing partners. This all leads to a less than satisfactory performance rate for efficiency.

The programme was quite effective and the majority of beneficiaries now have water of better quality and closer to their homes. Furthermore, the programme was flexible and made some proper decisions to deviate from the original proposal. On the other hand, despite efforts to train the district authorities the programme did not leave strong and effective districts behind. Nevertheless, the performance rate for effectiveness is satisfactory.

The programme had a positive impact on women and girls in that it decreased their workload and improved sanitation facilities at schools. However, the positive impact of the programme on school children is compromised by the lack of functioning hand washing facilities at the schools. Due to a lack of baseline data the programme impact could not be quantified, hence no performance rate for it could be established.

Recommendations:

  • The development of a proper theory of change should be given priority in new projects.
  • Clear roles and responsibilities should be well defined between UNICEF and the line ministries, and programme partners at district and community level.
  • Quality control, strengthening of the social infrastructure, and capacity building in the districts should be placed higher on the agenda.
  • Communications with the districts, especially on financial matters, should be improved.
  • UNICEF and the district partners should improve their data management, reporting and M&E. 
  • UNICEF and the GoM should focus more on the social aspects of WASH in rural water supply.
  • UNICEF and the development partners should be more innovative and allow experiments with advanced technical approaches. 
  • The development of an exit strategy is essential and UNICEF should get the commitment of government for sustained financial support for the districts, in order to avoid the loss of valuable trained staff.
  • Hand washing in schools and villages should be given the highest priority in rural WASH projects.
  • Rethinking the present sustainability strategy is essential. All the different aspects of sustainability should be reviewed in cooperation with development partners. If necessary, UNICEF could ask for assistance from international knowledge institutions.

Lessons Learned:

Organizational and institutional set-up-The roles of the different stakeholders need to be clearly specified.
Infrastructure and technical sustainability-The capacity of the DWOs to implement the programme and to carry out the necessary quality control needs to be developed.

Capacity building and awareness-The WASH programme interventions require transfer of skills to communities through training and awareness programmes. This is very critical to interventions that are promoting sanitation and hygiene practices.
Baseline surveys, data management and financial transactions-A proper data management portal could be developed where community data could feed into a district and national data bank. The financial transaction and release process used by UNICEF has to change to make sure that activities are not delayed.

National coordination and networking in WASH- There is need for the GoM to design a very clear framework to be followed by development partners in implementing WASH programmes. Some of the organizations championing WASH interventions should be inspected and be given the required support to address any problems.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
Malawi

Region:
ESAR

Theme:
Water, Sanitation and Hygiene (WASH)

Type:
Evaluation

Language:
English

Sequence #:
2016/001

New enhanced search