2013 Pakistan: UNICEF WASH Pakistan End of Programme Evaluation for Rural Sanitation in Flood-affected Districts (RuSFAD Phase III) 2011-12
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report."
Post Floods 2010, UNICEF with other development partners developed and implemented a large-scale sanitation recovery initiative. The initiative is known as Early Recovery Scaling-up of Rural Sanitation in Flood Affected Districts (RuSFAD). The project was developed jointly by UNICEF, UNHABITAT and Plan Pakistan (PP), as part of sanitation sector recovery initiative, following massive floods in Pakistan in 2010. The field implementation was undertaken by leading national non-profit organizations (referred to as implementing partners – IPs) across Pakistan. The project implementation was rolled out in three phases during 2010-12, with overlapping implementation of different phases. The third phase was planned during late 2011; however, its implementation started from early 2012. The first two phases covered 23 flood-affected districts (four during the pilot phase and 19 districts during the second phase), while third phase was implemented in 17 districts. WaterAid Pakistan (WAP) joined in later as ‘Independent Monitor’ and remained involved (partly) in Phase-II and then in Phase-III. The project aimed at providing, “Support for improved sanitation and hygiene practices by implementing the Pakistan Approach to Total Sanitation (PATS)”. The project's results framework is structured into five outputs all attributing to an outcome. The framework, however, does not explicitly define the desired impact. The outputs as listed in the framework are: a) facilitating provincial and district governments’ participation; b) application of participatory sanitation approaches in selected communities; c) communities achieving and sustaining Open Defecation Free (ODF) environment; d) strengthening markets for safer sanitation; and e) reduction in water-borne diseases. The evaluation has been undertaken for Phase III only including both flood-affected and polio-affected/high risk districts (17 in total) in four provinces, i.e. Punjab, Sindh, Baluchistan and Khyber Pakhtunkhwa.
Post Floods 2010, UNICEF with other development partners developed and implemented a large-scale sanitation recovery initiative. The initiative is known as Early Recovery Scaling-up of Rural Sanitation in Flood Affected Districts (RuSFAD). It was jointly developed by UNICEF, UN HABITAT and Plan Pakistan (PP). The field implementation was undertaken by leading national non-profit organizations across Pakistan. With some overlapping, the project was implemented in three phases during 2010-12. The third phase was planned during late 2011; however, its implementation started in early 2012. The first two phases covered 23 flood-affected districts (four during the pilot phase and 19 during the second phase); whereas, the third phase was implemented in 17 districts. WaterAid Pakistan (WAP) joined later as ‘Independent Monitor’ and remained partly involved in Phase-II and fully during Phase III. This evaluation is for RuSFAD phase III only.
The remains a multi-purpose evaluation with articulated focus on accountability and learning. The evaluation commissioned to meet the contractual obligation towards donors and internally to address UNICEF’s programme management requirements. It applies to a varying degree to the other stakeholders involved in programme design and delivery. The evaluation findings and recommendations are to inform the future programme design in Pakistan. Moreover, as UNICEF globally is rolling-out Community Approaches to Total Sanitation (CATS), hence the results would enrich the organization wide discourse on CATS design & implementation. Last yet not least, the evaluation is to inform th public stakeholders of PATS related design and implementation challenges and opportunities to facilitate public sector ownership, replication and scale-up of PATS across Pakistan.
The evaluators used mixed method approach for the evaluation, in view of the aritcluated expectations in the evaluation TORs. The evaluation team employed both primary and secondary methods to draw qualitative and quantitative information. The methods selection and tools design was driven by the evaluation expectations. The evaluators used the following methods for the evaluation;
‒ Review of secondary sources;
‒ Post-KAP survey (quantitative);
‒ In-depth semi-structured interviews (qualitative tool);
‒ Focus group discussions (qualitative tool);
‒ Field observations; and
‒ Field photographs.
The evaluators used the quasi-experiemental research design for the evaluation, where pre and post-test model was applied to draw comparison between baseline and endline numbers, to assess the extent of change in knowledge, attitude and practices of the beneficiairies. The evaluation excludes the creation of counterfactuals by creating evidence for control and experiment groups. This was excluded for the fact that this remains an end of project evaluation rather impact. Also, as the project was implemented in post disaster relief/recovery context where more or less all communities affected by floods have had either received or continue to receive humanitarian assistance, which constrained to have a authentic control group.
For quanitative information, different tools and packages were used such as CsPro, SPSS and Excel. Qualitative data analysis used data reduction, data display and conclusion/verification techniques. Both qualitative and quanitative findings were corroborated for analysis.
Findings and Conclusions:
Overall the project has performed well in terms of achieving the output targets, especially as the project was implemented in post flood recovery context. Also, that in some areas the implementation was affected by seasonal floods. Project internal reports indicate that project has either achieved or exceeded targets for 31 out of 50 output indicators (refer Table 1 for details). The output achievements at 62% are healthy.
The key evaluation findings and conclusions are described below:
1. Application of PATS in post flood recovery apparently worked well; however, the implementation did not follow the complete package of PATS conceptual framework or pillers;
2. The project successfully demonstrated engagement with district level public stakeholders, however with limited success to leverage the same at provincial level to advocate policy change;
3. The performance may have been good, nevertheless, the targets were ambitious and it stretched partners at the cost of reflections and quality checks on delivery;
4. The project’s contributions to partners’ capacity building are duly acknowledged, however may have been more beneficial had it followed structured approach. It may have set the foundation for better replication and scaling-up.
5. Behavioural changes are evident pointing to success with BCC component. The contextualization of messages and delivery approaches resulted in enhanced efficacy;
6. The supply facilitation through Sanitation Marketing (SanMark), masons training facilitated both the accessibility to materials and services and slight reduction in costs (indirect mostly). However the low cost latrines concept is still illusive and designs offered lacked inclusion;
7. Third party monitoring proved useful for management decision-making, with limited success with knowledge documentation.
The key evaluation recommendations are outlined across policy, design and operations and are framed in order of priority. The most significant recommendations are:
i) demonstrate greater coherence and consistency in the programme design with result based management principles including budget;
ii) rationalize targets and provide adequate time for project softer interventions (mobilization, for instance) and enable reflections and knowledge integration in subsequent phases;
iii) include parallel implementation of community and school led total sanitation approaches;
iv) lay adequate focus on advocacy and engagement with provincial stakeholders for policy reforms;
v) incorporate a more feasible exit plan in future design;
vi) consolidate the findings from RuSFAD implementation as part of knowledge documentation;
vii) encourage implementing partners to explore triggering approaches that ensure respect and dignity of beneficiaries; and
viii) define roles of partners in project implementation more clearly and build their capacities in strategic domains.
Full report in PDF
PDF files require Acrobat Reader.