Evaluation database

Evaluation report

2014 Global: Evaluation of the WASH Sector Strategy “Community Approaches to Total Sanitation” (CATS)



Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

In the context of the recent evolution of the sanitation sector, CATS can be seen in a two-fold way: as a move from technically-based supply-driven approaches towards behaviour-change, demand-driven approaches; and also as a recognition of the centrality of the adoption of a new social norm around ending open defecation as a key issue to be addressed, with impact on and linkages with other sectors (health, education, etc.).

CATS successfully contributed to shift the sanitation sector towards demand-driven and not directly subsidized approaches.

Our evaluation shows that CATS has given a new momentum to rural sanitation in the 50+ countries supported by UNICEF. This new momentum has translated into a change in how rural communities regard sanitation, invest into it, commit to new behaviors around ending OD – and eventually improve their living conditions.

The CATS principles are now shared by most of the countries where CATS has been deployed, with a relatively high degree of ownership, at all levels, from central to local governments. UNICEF and its partners have successfully advocated CATS principles and managed to influence other key development partners.

Compared with the situation before CATS, that was characterized by the predominance of supply-driven, heavily subsidized and low-efficiency programs, this is a dramatic and positive change. UNICEF is now recognized by Governments and development partners as a major actor of change in the sanitation sector.

Purpose/Objective:

The evaluation was designed to answer the following questions:

Overarching questions and objectives

Outcome objective: What are the results achieved by CATS (output and outcome levels) and what is the quality of evidence validating these results?

Effectiveness objective: What are the key social and technical factors that can explain the success or failure of CATS in a given country/community context?

Efficiency objective: What are the key financial and managerial factors that maximize the efficiency/value-for-money of CATS? How can they be optimized?

Sustainability objective: What are the key factors required at country/community levels to improve the adherence to new ODF behaviors created by CATS?

Methodology:

The methodology consisted of the following:
a. Hiring an evaluation team through a competitive bidding process;
b. A preliminary period of investigation to determine what data could be available; finalization of an evaluation design including matching expected data against information needs. This inception period included a literature review of CATS and CLTS documentation across organizations and at different scales of programming, in different regions. It also included the development of a theory of change guiding CATS programming, as a basis for finalizing the evaluation questions and methods;
c. Inputs from over 200 persons directly involved in CATS implementation (60% UNICEF staff, 40% counterparts and partners at country and global level) via an online survey available in English, French, and Spanish;
d. Key informant interviews with 11 external experts knowledgeable about CATS approaches and UNICEF’s role in the deployment of CATS;
e. 10-14 day case study visits to 5 nations (India, Nepal, Mozambique, Sierra Leone, and Mauritania) by 2 persons from the evaluation team, supplemented by national consultants. These visits included meeting key informants in government etc., field visits, and 1-2 half day seminars with community, NGO/CSO, and government representatives. The 5 nations were selected for the breadth of ecologies, partnership arrangements, overall CATS approaches (e.g. SLTS in Nepal), policy contexts, and cultural contexts;
f. Webinars with UNICEF staff on CATS sustainability issues in French and English, and on Social norms and CATS (English); the webinars were facilitated by the evaluation team;
g. Intensive engagement with UNICEF HQ [Evaluation Office and WASH section] to provide quality assurance and oversight over the whole evaluation process;
h. Quality review of the inception report and the final report drafts by a panel of 5 external experts drawn from major WASH organizations, all of which are familiar with CATS and with evaluation methods.

The methodological limitations include lack of data within the literature review in some areas, unbalanced participation in the survey and potential representativeness issues with the 5 case study nations and, within them, the field sites visited. Mitigation measures were taken but some findings are more certain than others.

Recommendation:

Continuing things that are working well
CATS has proven to be a successful approach in addressing some of the sanitation sector’s bottlenecks and blockages. First recommendation from this evaluation is therefore to continue doing things that are working well. This is a sampling of those noted in the report:
• The implementation of CATS through government channel whenever it is feasible;
• The strong engagement at policy level to build a sound enabling environment;
• The integration of the social norms concepts in CATS design and implementation;
• The use of non-monetary, pride-based community awards as positive motivation;
• The integration of CATS M&E system into the sector overall M&E framework.

See detailed recommendations in the attached report, section I, page 85.

File Attachment(s)

Case Studies:

You will find the full CATS report and related documents below as follows:

UNICEF final evaluation report - "Report"
Consultants' final evaluation report - "Part 2"
GEROS quality review - "Part 3"

 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Region:
Global

Theme:
Water and Environmental Sanitation

Partners:
HYDROCONSEIL-WEDC-ECOPSIS

Type:
Evaluation

Language:
English

 

New enhanced search