We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Pakistan: Evaluation of the UNICEF Sanitation Programme at Scale in Pakistan (SPSP) – Phase 1 (2013-14)



Author: AAN Associates

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

Sanitation Programme at Scale in Pakistan - Rural’ (SPSP Rural)’ is a multiyear (2013-17) programme with first phase implemented through 2013-14. The programme targets rural parts and particularly the flood and security affected (insecure) areas. It builds on the experiences and learning drawn from UNICEF’s Post Floods Recovery (2010) Programme i.e. ‘Rural Sanitation in Flood Affected Districts’ (RuSFAD) implemented through 2010-12.

The programme aims that; ‘By 2017, most vulnerable and marginalized children and women will be using safe drinking water, improved sanitation and hygiene practices’. The SPSP Phase I (the Year I) intended to reach out to 1.3 million people by implementing the Pakistan Approach to Total Sanitation (PATS)1. The PATS combines the School and Community Led Total Sanitation (SLTS & CLTS) approaches (referred to as Community Approaches to Total Sanitation (CATS) by UNICEF). The programme consists of six interlinked components, which include: i) Demand for sanitation; ii) Supply facilitation; iii) Health and hygiene promotion; iv) Linkages with duty bearers; v) Attaining total sanitation and; vi) Knowledge management.

The programme design and delivery involved multiple stakeholders. UNICEF is the lead agency with contributions to programme design, resource mobilization, contracting the implementing partners (national and international non-profit organizations) and technical oversight. The field implementation has been carried out by non-profit partner agencies, and been referred to as ‘Implementing Partners (IPs)’. The programme has leveraged partnership with government agencies (mainly Public Health Engineering and Local Government Depts.) for some component. The field monitoring functions were performed by a ‘Third Party Monitors (TPM)’ in assistance to UNICEF.

(Read more in the main report enclosed)

Purpose/Objective:

The evaluation was commissioned by UNICEF Pakistan with the purpose to assess the programme achievements and to use the findings to inform the next phase. Objectives as stated in the TORs include, determining the relevance, effectiveness, efficiency and sustainability (as per UNEG standards), whilst also assessing the extent to which UNICEF core programme strategies and normative principles (Human Rights Based Approach, Impact of Unforeseen Events and Financial Risk Management) were met.

The evaluation has been carried for the activities undertaken in 13 districts and (FATA) agencies during Phase I (2013-14). The primary audiences or users of the evaluation include DFID, UNICEF, Implementing Partners (IPs) and government stakeholders. The secondary audiences include WASH sector partners and beneficiary communities.

(Read more in the main report enclosed)

Methodology:

The evaluation methodology was evolved with a considered view of this being a formative evaluation and a clear focus on documenting key learning coupled with identifying areas for improvement. The intent was to have the next phase informed of the opportunities to improve both the design and delivery.

The evaluators used mixed method approach for the evaluation. To map the extent of change for key quantitative output/result indicators, quasi-experimental research design was used to collect the information and hence draw comparison with baseline results. For this, a representative Post-KAP survey was administered in programme areas. The evaluation design however, excludes the creation of counterfactual; primarily for being formative evaluation rather an impact. The other methods used include secondary sources review, in-depth interviews, focus group discussions, bottleneck analysis (workshops) and observation checklists. For the evaluation design and implementation, the consultants followed UNICEF adapted United Nations Evaluation Guidelines (UNEG), accredited researching norms and ethics. An Evaluation Reference Group (ERG) with representatives from UNICEF, DFID, and government departments, which provided evaluation oversight and shared their own experiences with the evaluators.

(Read more in the main report enclosed)

Findings and Conclusions:

Relevance: The section carries findings and commentary on the level of coherence (consistency) of programme objectives and approaches to national and provincial policies and priorities, needs of men and women beneficiaries (including vulnerable) and total sanitation approaches. Also level of integration for UNICEF core strategies or crosscutting priorities i.e. gender equality, equity, and disability in programme design and implementation. By and large, the SPSP Phase 1 design was found coherent and consistent with the national policies and development commitments. The geographic prioritization seems appropriate, for the fact that most of regions picked up for programme implementation showed having lower sanitation coverage than the national average.

Effectiveness: The programme has performed well as the results indicate implementers being successful in achieving or in most exceeding the set (output) targets. The indirect implementation approach, where IPs were contracted, appears to have worked well to achieve targets in short time span. The limited engagement of public sector in programme delivery appears as a weak link.

Efficiency: The programme implementation was found efficient especially where all (significant) performance targets have been met, despite delayed start. However, the fact that implementers were able to exceed most of the output indicators raises questions on whether realistic targets were set and corresponding resource calculations were appropriate.

Sustainability: The SLTS component without continued support and engagement of district education offices is less likely to sustain either. The rehabilitated WASH facilities at schools and health outlets may not sustain without proper O&M mechanisms. The changes brought in the knowledge, attitudes and behaviours are the most likely and significant results that may sustain.

(Read more in the main report enclosed)

Recommendations:

Future programming should provide adequate time for all stages of the programme i.e. inception phase, PATS implementation phase and exit phase. This should include defining the duration it takes for a standard PATS implementation cycle.

Following are key recommendations across design, implementation and policy areas;

1) Future programming should provide adequate time for all stages of the programme i.e. inception phase, PATS implementation phase and exit phase. This should include defining the duration it takes for a standard PATS implementation cycle.
2) The programme Theory of Change needs to be articulated with clear linkages, pre-requisites, risks and assumptions. Similarly a sustainability strategy, focusing on sustaining ODF status should be developed. PCAs should include specific IP exit plans, and stakeholders who will be ultimately responsible for sustainability should be clearly informed of their role along with their consent to delivery it (community as well as role expected from Government Departments).
3) Following aspects are recommended for the supply facilitation component,
a) Sanitation Mart Owner/Entrepreneur
Distinction between Sanitation Mart Owners and Entrepreneurs should be reassessed and if required dropped. The DSI action research points to the fact that market forces or business opportunity works well with sustained supply of materials, given, those in the business are made to realize that how CLTS and SLTS approaches can contribute to increasing demand for sanitation. This may need to be focused in the next cycles.
b) Household Subsidy
Modality and intended benefits of providing a household level subsidy should be reassessed, including possible detrimental effects on the community. Other mechanisms should be tested (such as providing collective incentives at the community level). For equity purposes and the needs of the vulnerable revolving credit/contributory fund can be considered as an option to be managed by the VSC.
4) The future programme design must clearly articulate and provide adequate implementation guidance for integration of gender equality, equity and vulnerability including addressing the needs of the disabled.

(Read more in the main report enclosed)

Lessons Learned:

a) Access to Sanitation Facilities and Reduction in Violence Against Women
The evaluation has contributed to unravelling a critical link between open defecation and extreme cases of gender violence in rural Pakistan. A significant number of respondents shared that the availability of latrines inside the house, has significantly reduced the incidences of extreme gender violence such as Karo Kari (honour killing). To them, the construction of latrines have reduced women’s need to step out (as for defecation they were coming out at odd hours), thus the protection risk (or probability) of being accused of illicit relations (with men), and consequently being subjected to extreme violence. The lesson that is derived from this instance is that though an indirect result of WASH intervention, the benefit of contributing to a reduction in violence is a substantially important aspect and should inform the future programme design and delivery.
b) Shame, Shock and Disgust versus Pride, Self-Respect and Dignity
The two approaches generally are not used together for triggering sanitation, however, SPSP Rural I demonstrated that by using them intelligently (where shame, shock and disgust follows the other), better results could be achieved.
c) Business Incentive and Opportunity Is a Larger Motive for Response
DSI’s action research suggests that manufacturers, entrepreneurs and mart owners (with business risk bearing attributes) see supply to the underserved areas as an opportunity to expand their business further. Given purely business incentives, they are more likely to sustain sanitation supply in order to meet the heightened demand and maintenance requirements in communities.
Hence, it could be argued that the programme may benefit more, if it may focus on identifying and working with manufactures/entrepreneurs, with a clear intent to help them realise the real market potential and opportunity offered.
d) Using Subsidy to Address Equity Can Lead to Dependency
To a degree the subsidies have served the equity purpose. However, the subsidy planning and delivery mechanisms faced challenges, which need deeper thinking and innovative solutions. The solutions may need to take note of the evaluation finding that such incentives contribute to reducing willingness of families to make financial contributions to construct latrines.

(Read more in the main report annexed)



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Pakistan

Region:
ROSA

Theme:
Evaluation; Water and Environmental Sanitation

Type:
Evaluation

Language:
English

New enhanced search