We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2018 Pakistan: Evaluation of Joint P&DD and UNICEF Country-Led Evaluation of the Clean Drinking Water for All (CDWA) Project Balochistan



Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

The Balochistan Clean Drinking Water for All evaluation was conducted as part of the National Evaluation Capacity Development initiative, which began in 2015 when Phase II of CDWA commenced. The initiative focuses on developing and implementing effective monitoring frameworks, establishing indicators focused on the Sustainable Development Goals and collecting and analysing accurate data for those indicators.

In Balochistan, between 2011 and 2015, the use of many water sources decreased, and the water supplied through drinking water sources is not always safe due to contamination from various sources. Many of the existing water sources in 2011–2012 were either unsafe at the consumer end or were non-functional.

In 2007, the Federal Government decided to install 567 water purification plants in Balochistan at the UC level to improve the availability of clean water in Balochistan. The project officially began in 2008 with a target of 575 (567 + 8 additional plants) water filtration plants, including both ultra-filtration and reverse osmosis plants. In 2015, the design phase for Phase II of CDWA began, with implementation beginning in 2017. This included 85 solarized plants, 24 RO plants and 61 UF plants.

In 2011, a third-party verification exercise was carried out by Balochistan Engineering Services Agency, which identified significant issues, including that 41 per cent of the installed plants were non-functional. In 2015, the Government of Balochistan asked UNICEF to support the province in developing an evaluation policy. Thus, a joint evaluation was suggested where the P&DD was supposed to select a project from the social sector annual development plan for such an evaluation. UNICEF agreed to provide technical support for the evaluation and agreed to support a joint country-led evaluation for an ADP project within the water, sanitation and hygiene sector.

Purpose/Objective:

The overall purpose of conducting an independent and objective evaluation was to gauge the effectiveness of the CDWA project implemented by PHED in 15 districts of Balochistan, over the period 2010–2011 up to and including May 2018. Additionally, it was to inform programming decisions for improving water supply to households while demonstrating accountability to the stakeholders, drawing lessons learnt and forming recommendations to inform continuity and scale-up.
The evaluation aims to: i) measure the programme in terms of how successful it was in addressing problems related to water; ii) whether there were more effective ways of addressing the problem for different cost and iii) to build the capacity of the GoB to conduct evaluations by helping to develop a policy for the evaluation of development projects in Balochistan. The evaluation is focused only on long-term outcomes and has therefore not employed any impact evaluation methodology using experimental methods.

Methodology:

For this evaluation, a summative-formative methodology  was deployed with qualitative and quantitative mixed methods.
The findings are based on 45 key informant interviews (KIIs), 30 focus group discussions (FGDs), a 300 respondent household survey (HHS) and an inspection of 47 WFPs conducted in 15 districts of Balochistan. An additional six WFPs were inspected during the pilot testing of the tools. In total 674 people (51 per cent men, 49 per cent women) were met over the course of the evaluation. The fieldwork was conducted from 28 May to 6 June 2018, where the qualitative and quantitative evaluation surveys were completed between 4–8 May 2018. See Section 2.7 of the report for details of the sampling for the HHS, KIIs, FGDs and observations.
The evaluators prepared data collection tools that meet applicable national and international best practices, including UNEG/UNICEF guidelines on participatory approaches, respondent-friendly methods of data collection, human rights, equity and gender in the design, data collection and data processing stages and in the formulation of recommendations. The latter ensures ownership of the actions to be taken in the post-evaluation period (see Appendices 4a through 4f for all the tools and guidelines used). Human rights, equity and gender aspects were included in the evaluation design.
The evaluators included quality assurance approaches and methods for both qualitative and quantitative data collection, including random re-testing, spot checks, verification visits, cross-checking, independent reporting by different team members and picture verification among others. The quality assurance function was embedded into all training, supervision and review processes. The qualitative and quantitative data was validated prior to analysis and was triangulated to achieve greater clarity. All final reports were professionally developed and edited.

Findings and conclusions:

Relevance
Overall, all respondents and participants in KIIs stated that the CDWA initiative remains highly relevant, given continued water scarcity, limited rainfall and the health impacts of drinking untreated water.
The original design of one plant per UC of 20,000 people was flawed to start with and between 2011–2017 the population density has increased significantly.
Effectiveness
Eighty per cent of the KII and FGD responses expressed dissatisfaction with the operational and technical issues resulting in insufficient clean water supply. Major hurdles to the maintenance, operation and improvement of WFPs include lack of funds; absence of skilled labour, insufficient power supply, poor maintenance of infrastructure by the PHED; poor monitoring; sabotage; corruption and political issues.
Efficiency
Management of human resources was not satisfactory – most operators have been on contract terms for more than eight years. No significant training was imparted to WFP staff.
A number of inconsistencies in the documents available.
Outcomes
Quantitatively, the long-terms outcomes of the CDWA cannot be identified at this stage as 60 per cent of the plants either do not operate as originally envisioned or are non-functional.
Of the 60 per cent of household respondents who were aware of the importance of always using clean vessels, for water collection, only 57 per cent confirmed daily cleaning of such vessels.
Sustainability
WFPs are necessary to improve the health of communities.
The sustainability of the outcomes of the CDWA intervention hinge on a number of factors: i) Ensuring a supply of water, ii) Ensuring proper maintenance and monitoring through proper coordination between key stakeholders (PHED, DHO and local governments), iii) Continued availability of water for WFPs and iv) Behaviour and attitudes of plant operators and beneficiary communities.

Recommendations:

Overall careful attention is required to look at the need to redesign the planning function with PHED and allied departments. PHED needs to be seen to value a consultative approach and the promotion of the message that “What the people need is, and will remain so, the key driver” of PHED work. PHED must also demonstrate cost efficiencies and cost-benefits, and should therefore invest in closely monitoring implementation across the board. If this is influenced by political will or influential persons, then report the same with pros and cons so that lapses are dully attributed.

The key recommendations are as follows:

1) Take stock of ALL 409 plants and therefore slow down further developments in Phase II. The census of WFPs must cover:

  1. All operational and non-functional issues;
  2. Assess and address access and safety concerns of the plant users/ beneficiaries;
  3. Retake water samples, retest and finetune operational plants.

2) Expand and improve the TOC to increase its future usefulness.
3) Redesign the planning function – START IMMEDIATELY.
4) Redesign annual budget to include all missing elements of management design and plan for ward-level WFPs. Solicit grants, rationalize the social workers and budget for the water testing laboratories.
5) Ensure water testing laboratories are present at the regional level and provincial centre;
6) Establish a properly staffed and trained PMERR Section and ensure that all PHED is fully made aware of the section’s mandate – START IMMEDIATELY.
7) Explore the value-add of community engagement, practices and procedures. Where the GoB may not opt for using NGOs it can certainly use registered community-based organizations. Community engagement can also be utilized in support of project activities to increase awareness of gender, equity and HRBA.
8) Test the capability and commitment of operators and train and retrain as much as required to ensure proper operations.

Lessons Learned:

Lesson 1: It is not enough to ONLY make accessible clean drinking water plants for a population. Related hygiene practices of the users must also be influenced through mass BCC techniques. Communicating for sustained change reigns supreme.
Lesson 2: It is not enough to hire and deploy WFP operators and then leave them to perform without continual supervision and refresher trainings. Operator performance and behaviours must be regularly monitored and repeatedly reinforced through guidance and refresher trainings.
Lesson 3: Never deliver an intervention and leave it to fate to operate flawlessly. It is a paramount need to closely monitor all aspects to achieve sustainable success.
Lesson 4: The message that must be clearly understood, owned and implemented across the board is: Strong and well-articulated planning, management, evaluation, reporting and research (PMERR) is based on a trained and properly staffed PMERR Section that:

  • Plans for IMPROVEMENT
  • Monitors for RESULTS
  • Evaluates for SUSTAINABILITY
  • Researches for DEVELOPMENT
  • Reports for TRANSPARENCY

 

Please find the attached labelled as follows:

  • Evaluation Report - Report
  • GEROS Evaluation Review - Part 2
  • GEROS Feedback Summary - Part 3
  • Evaluation Management Response (EMR) - Part 4


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year: 2018

Office/Country: Pakistan

Region: ROSA

Type: Evaluation

Theme: WASH; WASH - Household water security: systems, projects, strategies

Language: English

Sequence #: 2018/001 
New enhanced search