Evaluation database

Evaluation report

2000 IND: Learning from Experience: Evaluation of the UNICEF Water and Evnironmental Programme in India 1996 - 1998



Author: Kolsky, P.; Baumann, E.; Bhatia, R.; Chilton, J.; van Wijk, C. ; UNICEF NYHQ

Executive summary

Background

The water and environmental sanitation (WES) programme in India is the longest running and one of the most prominent WES programmes that UNICEF supports in countries around the world. Although the amount of UNICEF financial support in relation to total government expenditures is small, UNICEF has played an important and catalytic role in developing, testing and advocating key technological and institutional changes that influenced government policy and investment priorities to expand WES services to the Indian population. These included large-scale government investments in rural water supply and sanitation and the adoption of new drilling techniques, contributions to the implementation of a successful handpump-based rural water supply programme and exponential increases in water supply coverage. The evaluation was carried out in late 1998 and 1999 and was funded by the Department for International Development (DFID, UK); Netherlands Ministry of Foreign Affairs; Swedish International Development Agency (SIDA); UNICEF India; and the UNICEF Division of Evaluation, Policy and Planning (EPP). EPP has managed this evaluation as part of a multi-country thematic evaluation to draw lessons for global application.

Purpose / Objective

To learn lessons from this rich experience and to guide future directions, UNICEF commissioned an external evaluation of its WES programme in India. The EValuation had the following specific objectives:
- examine UNICEF's role over time, identifying lessons learned from each phase of the national WES programme
- assess the current status of the programme: its strengths, weaknesses, achievements, failures and constraints
- identify potential areas for future UNICEF contributions in India
- share lessons from India with other countries

Methodology

The five-member team, composed of international and Indian experts, conducted extensive interviews at all levels, visited field sites, developed a mail survey for UNICEF partners and staff, and reviewed an extensive number of documents. In consultation with UNICEF India, EPP formed a national advisory group to support the work of the team.

Key Findings and Conclusions

Water Supply:
Given the longer history and the GOI' emphasis on water supply coverage (relative to sanitation and hygiene promotion), the evaluation team found that many of UNICEF' long-term, tangible WES successes particularly lie in rural water supply. However, tasks remain in ensuring continued quality of borehole construction and rehabilitation and handpump manufacture. New challenges lie in the management of a mix of supplies through a range of institutional mechanisms at the community level, and in better management of water resources and water quality. The balance between achievements in coverage and quality is delicate. Considering the magnitude and spread of the water supply programme and its supply-driven implementation, some trade-off is inevitable between the quantity and the quality requirements of sound construction practise.

UNICEF successes include contributions in drilling and handpump manufacture, which resulted from a strong commitment to quality over the long haul. The evaluation team considered the choice of technology and approaches to be appropriate and cost-effective, although a lack of documentation meant that the team could not rigorously verify the latter point. Management information systems (MIS) could provide useful data on the technical and cost feasibility of various water supply options, but UNICEF has not used MIS for this purpose.

Environmental Sanitation:
With a history only going back to the 1980s, environmental sanitation has not shown the same results as water supply coverage. Only about 26 per cent of the entire population (urban and rural) have what the GOI considers adequate coverage. UNICEF has taken a leading role in shifting the focus in sanitation from a hardware fix to a package of services that combines latrine construction with hygiene and health activities, and from full construction subsidies to partial subsidies based on demand.

Over the past 15 or 20 years, sanitation has seen a number of promising efforts that have not proven sustainable over the long term, such as the too costly twin-pit pour flush (TPPF) latrine advocated by UNICEF and others in the late 1980s. UNICEF is to be commended for its courage to experiment, and the wisdom to see the need for experimentation. The success of some of approaches was seized too quickly both within UNICEF and the GOI in hopes of devising a strategy that could be scaled up in the same way as rural water supply had been.

Social Aspects of WES Services:
UNICEF recognised at an early stage that technological improvements have to be fully combined with user participation if systems are to be fully utilised and sustained. However, the switch to a systems approach that enables communities to manage and sustain their own water services, sanitation, and hygiene programmes with no or a minimum of continued backstopping from the government has not yet been made

Existing efforts, including information, education and communication (IEC) materials and person-to-person contacts, are generally satisfactory. Although UNICEF recognises gender as an issue in water and sanitation, designing and implementing truly gender-sensitive strategies rarely occurs. Other approaches such as public health communications, community management and Convergent Community Action (CCA) may be more cost-effective ways to bring about change, but these newer approaches have not been sufficiently tested by UNICEF in WES activities in India.

Cost-Effectiveness:
Although UNICEF clearly bore costs in mind over the years, cost-effectiveness has not been a systematic component of decision-making. In addition, a lack of documentation makes rigorous cost-effectiveness appraisal impossible. More qualitative assessments generally support the contention that fundamental decisions about water supply technology selection appear to have been sound.

Impact on Rural Women, Men and Children:
To try to separate out the direct impact of UNICEF WES efforts, as opposed to its substantial indirect impact through its work with the GOI, the evaluation team sought the views of rural users. They found some improvements in UNICEF-supported districts. Because UNICEF-pioneered action has invariably been taken up by the GOI, differences in the two types of districts may have been greater initially than over time.

Villages involved in UNICEF activities reported more water use and improved hygiene than villages that had access only to GOI services. Women in UNICEF-supported districts exhibited greater knowledge and use of oral rehydration therapy (ORT) to combat diarrhoeal diseases. Some improvements in general health and social conditions were noted by villages involved in UNICEF activities. The improvements include better education, including more girls attending school; better health care facilities; and a less restricted life for women in less conservative communities.

UNICEF Organisation and Management:
The practise in several field offices to share software professionals between sectors (e.g., WES and health) may have resulted in insufficient attention to WES social complexities. Although actual expenditures are difficult to track, the WES budget allocated to software (approximately 10 per cent of the total) seems low for an organisation that states that it is shifting more towards social aspects in its policy and strategies.

Field studies have not been set up to compare approaches or systematically assess impact. Examples of ways to increase the use of these studies include developing a common set of criteria against which to measure results, focusing on more significant comparisons, and focusing more on data analysis and less on data presentation. Greater use of participatory research methods would also yield a fuller picture of local WES behaviours and the effects of UNICEF and GOI programmes on those behaviours.

Partnerships:
UNICEF staff, partners and the evaluation team concur that UNICEF' long-standing partnership with the GOI over the past 30 years has greatly enhanced its access and credibility. The duration of the commitment, as well as the style of close collaboration, contribute to the effectiveness of the relationship. Partners cite institutional support, particularly training and information sharing, as UNICEF' greatest contribution to their capacity.

Lessons Learned

Long-term commitment and partnership produce results. UNICEF staff, partners and the evaluation team concur that UNICEF has earned the trust and confidence of senior-level officials and credibility at the national, state and district levels through its 30-year-plus relationship with the GOI. This gives UNICEF a tremendous comparative advantage.

An external agency such as UNICEF has greater freedom to test new approaches than a government has. This relative freedom suggests an important role for UNICEF in many sectors as an organisation that can develop and test new approaches.

Going to scale too quickly has adverse repercussions. It is tempting to expand on pilot projects that seem successful. However, in the long run, it is better to move slowly through a systematic approach that is gradually scaled up, to learn whether a promising strategy is indeed replicable on a larger scale.

Institutional arrangements at the district and community levels can help or hinder the implementation of centrally made decisions. A supporting national policy framework is important to move ahead in WES and many other sectors. However, what is taking place on the ground will determine the likelihood that a policy is successfully implemented.

A gender and poverty perspective must be consciously planned for and its systematic implementation monitored. Even with the best of intentions, incorporating such a perspective in participation, education and training will not happen without ongoing and deliberate attention.

Programme staff needs to be realistic about how much work they can take on and still be effective. With many pressing tasks, committed staff often shoulder large workloads and find it difficult to scale back or end an activity. However, staff must recognise that with too much to do, they cannot maintain high-quality work and the overall programme suffers.

Cost data are needed for more effective analyses. It is difficult to collect and keep track of this cost information. However, the lack of such data impedes cost-effectiveness analysis which, especially in an era of limited resources and greater accountability, is necessary for decision-making.

Recommendations

Focus efforts on areas where it has a comparative advantage, such as sanitation and hygiene promotion. This means that UNICEF should leave the leadership of some important areas, such as water quality monitoring and urban WES services to other organisations, and instead contribute through collaboration and partnerships. UNICEF should make sanitation and hygiene a higher priority.

Change staffing and staff development system to ensure that social and institutional aspects of WES are adequately addressed. UNICEF needs to look at whether staff has the appropriate expertise to take on these new challenges and that they are not carrying excessive workloads that can compromise quality.

Address the challenges of decentralisation, which offers new opportunities for community-managed services. The GOI and external support agencies (ESAs) need to develop general WES guidelines and frameworks, and agree on common objectives and indicators in approaches that would otherwise be locally specific, and then meet regularly to compare progress and results. UNICEF may be in a prime position to develop and facilitate such a partnership.

Improve monitoring and evaluation of projects and alternative strategies. Recommended improvements include tighter specification of evaluation studies; closer partnership with those involved in the studies; and definition of clearer outcomes, objectives and criteria for success when the projects are planned. Areas where cost data are needed but lacking include borehole drilling and rejuvenation technology, sanitation technology and promotion, and approaches for behavioural change. Subjects for further exploration identified in this evaluation include cost-effectiveness analyses of borehole rejuvenation techniques; evaluation (jointly with the Bureau of Indian Standards) of handpump manufacturers? qualifications and standards; pre-testing of IEC messages; and cost-effectiveness analyses of CCA, social marketing and community-managed services.

Strengthen its partnerships through working with fewer, closer partners. If UNICEF works on fewer issues, it will work with a smaller set of partners. This can potentially build stronger relationships.

Protect and preserve long-standing, well-deserved legacy and reputation by ensuring that the quality of its programme remains high. UNICEF has built its reputation through hard work over more than 30 years. As it withdraws from some of its activities, it must do so in ways that encourage others to continue to provide quality WES services to the people. By working closely with partners and developing and testing new approaches, UNICEF can sustain its achievements and maintain its hard-earned legacy and reputation in contributing to WES services.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2000

Region:
ROSA

Country:
India

Type:
Evaluation

Theme:
Water and Environmental Sanitation

Partners:
DfID, SIDA

PIDB:

Follow Up:

Language:
English

Sequence Number:
2000/800

New enhanced search