We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2000 CEE/CIS: Aral Sea Project for Environmental and Regional Assistance (ASPERA) Evaluation



Author: Lechtig, A.

Executive summary

Background

This paper presents an evaluation of the Aral Sea Project for Environmental and Regional Assistance (ASPERA), which was set up to focus on the problems of women and children associated with the Aral Sea phenomenon. It covers one geographic area in each of the three countries affected: Kyzylorda Oblast in Kazakhstan, Dashowuz Velayat in Turkmenistan and Karakalpakstan Autonomous Republic in Uzbekistan. The program, launched in 1995, has five components: maternal health, child health, nutrition, basic education, and water and sanitation.

Purpose / Objective

This evaluation context is defined by the need to assess the program situation five years after its inception. The objectives of this evaluation are:
- To assess the effectiveness of ASPERA in meeting the 1995 objectives in immunization, CDD/ARI, maternal health, nutrition, primary education, and water and environmental sanitation
- To assess the usefulness of the cross-sector, cross-country approach to project management
- To assess the project's capacity to reach the target group
- To examine the management structure, relationship and coordination among the various UNICEF, government, NGOs, and other international partners, and to assess the success of UNICEF in building capacity at the local level
- To assess the project's record in raising donor contributions
- To make recommendations for future work

Methodology

Each of the three geographic areas was visited and, in each of them, site visits were made to the health and education facilities at the Oblast and Rayon levels. The evaluation was based on:

- Review of available documents, materials and background information in the UNICEF CARK Area Office, Kazakhstan, Turkmenistan and Uzbekistan offices
- Focussed discussions with UNICEF Country Assistant Representatives and Project Officers involved with the project
- Focussed discussions with local government officials, health staff, school directors, NGOs and other partners in the three provinces
- Review of available reports, health and educational materials, and statistics in the relevant areas
- Selected interviews and group discussions. For this purpose, an interview format was developed, covering key questions, with inputs from the local UNICEF Offices

Key Findings and Conclusions

Of the 1995 original objectives, those in child health related to immunizations, control of diarrheal diseases and those in nutrition related to breastfeeding have been achieved with a clear contribution from UNICEF. Most of the other 1995 objectives in child health (acute respiratory infections), in maternal health, in malnutrition (including iron deficiency anemia and iodine deficiency), in basic education, and in hygiene, water and sanitation have not been achieved.

A first lesson learned here is: precise objectives are first step for success. An examination of the ASPERA objectives, which were achieved by the program, shows that many of them were very concrete, precise and easy to quantify. They were, for example: "Universal availability of vaccines in project areas"; "100% functioning of cold chain facilities in project areas"; "Maintain high vaccination coverage at no less than 80%". On the opposite, most of those objectives, which were not achieved, were too general and too vague.

The cross-sector, cross-country approach to project management has been, in practice, reduced to a multi-sector, multi-country approach. Renewed effort is required to build the original idea of intersectoral, inter country approach at sub-national level.

The management structure is not adequate to the tasks and it requires specific corrective action. Equally, the current coordination among UNICEF, government, NGOs and other international partners is less than adequate and needs to be improved. The performance in raising donor contributions has been well below the planned levels and it needs an urgent review.

Ownership requires participation. In some of the field visits, it was perceived that there was a lack of sense of ownership of the program by the population, which it is supposed to serve. If this feeling gets wider, it will affect negatively the program self-sustainability. ASPERA included in several program papers, the issue of participation as an important one to put into practice. However, it was evident that, frequently, the beneficiaries and the services personnel did not participate in program planning, implementation and monitoring. Because of this fact, they were not aware of program objectives and strategies. This was the case in Kyzylorda and in Karakalpakstan in regards to sanitation. In fact, in most sectors, there were neither consultative nor participatory processes with the potential beneficiaries, only with government counterparts.

The current coordination performance between UNICEF and government, NGOs and other international partners shows great variation across the three geographic areas. In Kyzylorda, the focal point was either not detectable or had visited the place infrequently. Although the participants identified up to three focal points in UNICEF for the coordination, it was clear that there was no systematic counterparts coordination from UNICEF. In Karakalpakstan Autonomous Republic and in Dashowuz, the focal points visited the places about once every month and it was obvious that they knew well what was going on. Far from the action is far from the program. The different performance detectable in the last two geographic areas may be explained by this factor.

When information was used, decisions were better. It was impressive to observe how, based on analysis of existing fresh information, decisions were made that allowed the achievement of objectives in immunizations and in reduction in mortality by diarrhea in the three geographic areas. On the contrary, during the site visits, several cases were perceived where an opportune decision was not made based on the information already on hand. This was the case of dry washing basins and dry drinking water facilities because no water was available, and no alternative intervention has been implemented to solve the problem.

A glass half full may summarize the current status of ASPERA. On one side, this may be understood as very little achieved as compared with the original objectives. On the other side, half a glass may be an achievement when several factors are taken into consideration: the stringent financial situation of the program; the extremely difficult physical environment that the program was covering; as well as the fact that the countries involved in the program were at the same time in the middle of a very delicate transition situation in political, economic and social terms.

Recommendations

Stimulate renewed effort in terms of a careful conceptualization of the international and intersectoral processes involved - As a beginning, separate both processes. The intersectoral approach should be stimulated in the form of integrated approaches to solving real problems: for example, the need for an integrated hygiene, water and sanitation approach will also help ensure better health and better nutrition. In a second step, it could be extended to become an international approach in Central Asia. This is required in order to gradually build the original idea of intersectoral, inter-country approach as part of the day-to-day work.

Use the information for making better decisions daily - ASPERA program information is produced and collected at a high cost. The only way of justifying the investment is by using its results. If the information is not used in daily practice, then it is not worth to invest resources in collecting it. Analyze regularly all the information from monitoring and evaluation, and use it to improve management decisions of every day.

Improve the coordination with local counterparts - Ensure that the focal point for the program works on an almost permanent basis on the program field site. The remoteness of the three geographic areas from their respective capitals could make this recommendation more expensive than originally planned. However, the additional investment, if any, will have a clear return in terms of program effectiveness and program efficiency as well.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2000

Region:
CEE/CIS

Country:
Intra-regional

Type:
Evaluation

Theme:
Area-based Programmes

Partners:

PIDB:

Follow Up:

Language:
English

Sequence Number:
2000/800

New enhanced search