We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Kenya: End of Programme Evaluation - UNICEF Kenya WASH programme 2008- 2014



Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

The UNICEF Kenya WASH programme was an integrated Water, Sanitation and Hygiene (WASH) project initiated in 2007, and begun with a planned timeframe of six years from 1 January 2008 to 31 December 2013. Subsequently, a 12-month programme extension was agreed and the programme officially ended 31 December 2014. At inception the programme comprised of seven components namely: new water development; rehabilitation of existing schemes; school wash; health facilities wash; household sanitation; hygiene education promotion; and institutional capacity building. The programme was originally designed to bring safe and sustainable sources of drinking water and hygienic sanitation facilities to 1.6million new users in then 20 arid, semi-arid and flood prone districts and two post-election violence affected districts in Kenya. The components of the UNICEF Kenya WASH programme represented an integrated Water, Sanitation and Hygiene (WASH) project. The components of the Programme were:

  1. Water supply
  2. Household sanitation and hygiene
  3. Wash in schools
  4. Health facility WASH
  5. Institutional Capacity Building and strengthening

The total planned budget was US$ 70.5 million. A grant of US$ 41 million was approved by the Government of the Netherlands (GON). The balance of US$ 29.5 million was to be contributed by the Government of Kenya (US$ 11 million) through the ministries of water, education and health, the programme recipient communities (US$ 10.9 million), and UNICEF (US$ 7.6 million).

The UNICEF Kenya WASH Netherlands funded project was planned to have an end of programme evaluation. This end of programme evaluation was in part in fulfilment of the funding requirements of the Government of Netherlands.

Purpose/Objective:

The objectives of the evaluation were to:

  1. Review the overall results achieved; assess the relevance and performance (effectiveness and efficiency) of the programme as well as the impact of the interventions of the participating communities and institutions and the sustainability of the results.
  2. Analyse the factors contributing to the successes and weaknesses of the programme.
  3. Document lessons learnt and best practices, and recommend actions for improved design, implementation, management and monitoring and evaluation of future WASH programmes in Kenya.

Methodology:

The evaluation exercise was carried out in 16 out of the 21 districts in which the programme was implemented. The evaluation covered 5 components of the programme and focus on areas where most investment had taken place post the medium term review (MTR) including Lake Victoria South, Lake Victoria North, Tanathi and Coast Water Services Board regions. Ten districts were visited namely, Kisumu, Siaya, Rachuonyo, Bondo, Busia, Kitui, Kajiado, Kwale, Mwingi and Garissa. To enrich the evaluation, another 6 districts were added, 3 of them being the traditionally marginalized districts of Marsabit, Tana River and Mandera and the other 3 being Nyando, Kieni and Isiolo Districts.

A mixed methods approach, was used to triangulate data, address data gaps in order to maximize the credibility of the evaluation findings.  The evaluation entailed a comprehensive literature review on programme activities, structured interviews at different levels of programme intervention and a representative programme activity output and user perception survey including technical audits of those outputs. The results of this survey combined with the structured interviews and independent observations of the consultant team allow to draw conclusions also on the outcome level.  Due to budget constraints the impact evaluation was constrained to establish anecdotal and observed evidence (as part of the activity and user perception survey) of change positively impacting the health and livelihoods of the beneficiaries in the program area

Sample size
Component No. of Projects Estimated Sample size Actual Sample Size
1 Water Supply 1100 297 323
2 Household Sanitation 19079 400 619
3 School WASH 186 122 127
4 Health WASH 36 32 30

Data collection took place between 17th August and 23rd September 2016. Pre-testing was conducted in Kajiado District.

Findings and Conclusions:

The findings indicate that 99% of households covered under the CLTS model of Household sanitation WASH constructed toilet facilities and there was recognizable behaviour change and adaptation of hygiene practices.

30 % of households visited the tippy taps were not present and the structures did not conform sufficiently to the element of privacy under the CLTS protocol 3.

The reviewed documents and the feedback obtained from the government actors indicated that the project components were well aligned with government and sector priorities.

The lack of institutional capacity in project partners continued to result in project delays. Quality concerns observed in some projects arose from these gaps which contributed to supervision gaps and inadequate monitoring skills.

The project freed up time for individuals and allowed them to engage in other activities, most of which advance their social-economic well-being
Coordination with Government Agencies was one of the most successful aspects of the project. However to achieve successful coordination various modifications were made during the course of the project to align project activities strongly with activities of Government agencies that were supportive of the project.

The weaknesses identified in this project are summarized as follows:

  • Failures- Lack of Institutionalization
  • Weak Coordination
  • Weak Funding Structures
  • Existing Norms and practices
  • Poverty, inadequate incomes and lack of livelihoods
  • Lack of Capacity in Partner Organizations

Recommendations:

Design

Future project implementation design should include a dedicated PMU with a structured leadership such as CHIEF executive PMU. Ideally it should be based at UN- Kenya office for continuity. In the event there are no specific sector structures or where staffs are lacking, the design should consider contracting leadership staff to work at the PMU. The education and health sector did not have sectorial structures. A project of this scope requires sector structures for effective coordination. The technical activities in WS appeared well documented because they are supported by existing industries. Thus there are well established standards. The same cannot of the other components especially at the beginning and the related training.   

Implementation

There should be a core competent team to support implementation across the entire geographical spread. This allows all areas to enjoy identical benefits from the intervention. Where delays exist in developing those teams, perhaps different project schedules should be adopted.
Implementation should be supported by a detailed program of activities i.e. work plans containing documentation guides on implementation, supervision protocol and evaluation procedures.
Team formation should allow some sufficient period for team orientation and collection. The team should exist for a period prior the implementation. This approach reduces learning on the go situations.

Management

 An operational PMU with a precise project leadership provides a concise institutional arrangement with converging authority. This provides an apex for the ultimate responsibility for supervision and quality assurance.

Lessons Learned:

  1. Capacity Building and Institutional Strengthening
    The programme had a strong component of training and collaborative partnerships with sector institutions and grass root based organizations. The capacity building provided for traction to ensure continuity on account of capacity in the post programme period. This is already being realized especially for WSTF and WSBs in Water supply. The use of local organizations also enabled the programme crystallize community buy in, which then provided a useful link to the community stakeholders and further the sense of ownership.
  2. Rigorous and robust Financial Procedures
    The use of rigorous and robust financial procedures achieved three important goals. It provided a positive signal to partners especially contractors that there was assurance of payments thus raising confidence in the programme. This kept at bay brief case operators and unscrupulous gate keepers. Second, the procedures provided a strong mitigation against the risk of fraud and embezzlement.
  3. Mainstreaming CLTS into the national policy
    Following the success of the CLTS approach in attaining ODF in targeted communities, the programme enhanced this approach to a national policy initiative. In 2014 the protocol for CLTS implementation was adopted by the National Government as the framework for promoting household sanitation and hygiene. Some of the County Governments have included in their financial budgets funding for CLTS implementation.
  4. Communication of Programme Results to foster innovation and technology adaptation
    The importance of communication of programme results should not be overlooked. It can enhance wider base of supporters for an initiative and more adopters of a concept, technology or idea. This is evidenced by CLTS which offered early proof of concept. The same was achieved in water supply with the adoption of shallow wells by the National Government and much more rapidly by County Governments.


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
Kenya

Region:
ESAR

Type:
Evaluation

Theme:
WASH - Water, Sanitation and Hygiene

Language:
English

Sequence #:
2016/014

New enhanced search