Evaluation database

Evaluation report

2003 BOL: Evaluation of Sida-Funded Projects Through UNICEF - Bolivia 1989 - 2002

Author: Dahl-Østergaard, T.; Rojas, R.; Moore, D.; Rozo, P.

Executive summary

Background:

Sweden has been one of the most important donors to the implementation of UNICEF programmes in Bolivia, providing a total of SEK 165 million since 1989. The support has been focused on three projects: the Bilingual Education Project, the Health Education Project, and PRO ANDES. These comprise 11 separate components, all of which are referred to by Sida as the Social Development Programme.

Purpose/Objective:

The regular reporting on the Programme has not monitored the impact of the activities. Hence, in view of the overall poverty reduction objective of Swedish development assistance, the main purpose of the present evaluation was to assess the poverty reduction impact and likely sustainability of the Programme during the 13-year period from 1989 to 2002.

Methodology:

This evaluation is based on primary and secondary data collected between 7th and 22nd October 2002, when the team worked together in Bolivia. Fieldwork was carried out in the departments of Potosi, Cochabamba and Chuquisaca.

As it was impossible to undertake a quantitative assessment of the impact, it was decided to focus on the perceptions of the beneficiaries. The sustainability of project interventions focused on the coherence between project initiatives and the priorities and poverty escape strategies of the poor themselves, and the coherence between project interventions and the policies at national, departmental and municipal levels.

Findings and Conclusions:

Impact:
There is no doubt that the beneficiaries perceive a lot of impact from the Sida-supported activities that have been implemented through UNICEF. It is equally clear that the poverty reduction impact - in the eyes of the beneficiaries - is more than a matter of improving income and consumption. The beneficiaries registered poverty reduction in all of the five dimensions of poverty mentioned above.

The positive impact of improving the self-esteem of children and women is marked in connection with the intercultural and bilingual education activities and the child development centres (Wawa Wasi). The importance of being able to begin the schooling career in a child's own language and respecting its own culture should not be underestimated. The improved relations between school and parents plus grandparents also mean that children look upon their parents and grandparents with more respect, and the separation between life at home and life at school becomes less pronounced. A healthy degree of self-esteem in a child is something that may be regarded as a lasting impact, as well as a foundation on which to build a productive future.

Improved self-esteem is also an important feature in connection with the improved water supply and the solar heated showers. The possibility to be clean and hygienic, to have white teeth and so on, is crucial for anyone's desire to interact with other people.
Another related impact is that there is more dignity in the way teachers, doctors and nurses interact with children, students, parents, pregnant women and patients in general. A new norm of mutual respect seems to have taken root.

Regarding the livelihood dimension of poverty, the beneficiaries value time saving as an important impact resulting from the water supply and the Wawa Wasi projects. The water systems save time, compared to the situation before when children, especially, had to spend much time collecting water. Similarly, women attach significant importance to the extra time they gain by having their children in the Wawa Wasi.
Somewhat surprising, the parents place great emphasis on the meals given, and not on the pedagogical activities and the pre-schooling of their children, as an impact from the Wawa Wasi. The satisfaction of this basic needs dimension of poverty through the child development centres reflects how poor the beneficiaries are, and how malnourished their children still are.

One problematic theme emerged from the beneficiary assessments. This relates to the social deprivation dimension of poverty: exclusion. This issue appeared in connection with access to health clinics, water supply, latrines and training through the Kallpa Wawa. Those who live far away from the centre of activities tend not to receive the same attention. Unfortunately, the difficult-to-reach are usually also the most poor, and those in greatest need of support. Given the difficulties involved in reaching the most remote settlements and the achievements that have been made, after all, our remarks about exclusion should not be regarded as a major criticism, but rather a word of warning and a call for special attention.

An observation should be made about monitoring. A reflection on the beneficiaries' statements gives rise to questioning the accuracy of some of the output reporting. The monitoring reports regarding the water systems provide information about the number of systems set up and the number of people that have ostensibly been covered. This may be accurate at the time when such systems are on the drawing board, but what is the situation one year later, or five years on? And what about the quality of the water? Similarly, the reporting on the Yuyay Jap'ina report on the enrolment in the two-year literacy causes, but they say nothing about the level of literacy attained, or about drop-out rates and other problems of quality and retention of learning.

Finally, a word of caution is in order about causality. The objective of this evaluation is to assess the impact of specific, Sida-funded activities. In all of the above positive tendencies, many different actors are involved, and Sida/UNICEF plays just one part which, in many cases, cannot be distinguished from the rest.

Sustainability:
Given the deficient monitoring system, the diverse range of projects, and the length of the period under study, this evaluation is no exception to the general rule that the sustainability criterion is analysed in somewhat hypothetical terms.

The overall conclusion of the evaluation team is that the greatest likelihood of sustainability is found in the projects that have become integrated with national policies and programmes (EIB and health). On the other hand, the activities that have been implemented by UNICEF more or less in isolation (Yuyay Jap'ina, Wawa Wasi, Kallpa Wawa) have substantial limitations insofar as sustainability.
However, while it is the preferable route to follow, the anchoring of UNICEF activities within the framework of public policies is no guarantee of sustainability. The financial and human resources available to the involved government entities impose the limit on their ability to ensure sustainability.

The economic crisis, and its manifestations at both government and community levels, underscores the pervasive and substantial dependency on external donor support in Bolivia. This has developed almost to the point of becoming a permanent state of affairs. As one mayor said when he was confronted with the fact that UNICEF would withdraw from his municipality at the end of the year and asked how he would ensure the continuation of activities: "They will continue, but I don't know how. It is always possible to get funds from international donors."

The Bolivian Government, however, is not only suffering from economic problems. The political culture and traditions that result in frequent changes of staff at all levels is a serious threat to the likelihood of achieving sustainability in any programme implemented through the government structures. If a UNICEF programme is not integrated with national policies and programmes, then it becomes all the more important to ensure an optimal integration with livelihood strategies of the target groups. This would call for tailor-made solutions at the project level, rather than the application of blueprint options regarding the design of a water system, for example. Ongoing impact monitoring could help in keeping project activities in line with the livelihood strategies, which is likely to change over time, of the target groups. Finally, the attention given to the termination of projects is inadequate. It appears that projects are finished as a function of the funding available or the duration of the project period that is defined for an intervention. One of the mayors interviewed stated the problem very clearly: there is a lack of an exit strategy.

Other Conclusions:
The Social Development Programme has contained a mix of service delivery and institutional development/capacity building. The balance tilts towards an over-weight of service delivery, but it is difficult to aggregate the numerous project activities over the 13-year period evaluated.

With some exceptions - EIB being the most notable one - UNICEF has not been sufficiently focused on the launching of pilot activities that could be tested with the explicit purpose of possible scaling up into government policies and programmes.

Recommendations:

UNICEF should, in the future, apply their considerable expertise to launch and test - in close collaboration with the Government of Bolivia - innovative pilot activities in areas of Government priority and community demands. The conditions under which possible scaling up into government policies and programmes should be done, should be made explicit from the outset. In other words, a clear Government commitment should be established before a new programme is launched.

Ideally, the exit strategy of any programme or project activity should be formulated already in the design stage. As a general rule, project activities should not be terminated before the objectives have been reached; if this is done, it is basically abandonment, not the achievement of development objectives. This kind of phase-out strategy necessitates well-defined project impact targets and regular monitoring of their achievement.

One project component would appear to merit special attention: the Wawa Wasi. This component is highly regarded among the beneficiaries and, in the view of the evaluation team, this is perhaps the component that has achieved the most significant level of impact. Yet, it is also one of the components with the least likelihood of attaining sustainability in the services delivered. Somehow, this paradox calls for special consideration.

Finally, consideration should be given to the possibility of negotiating conditions and contract-like agreements with the municipalities and the Government to ensure the continuation and sustainability of some of the project components that are about to terminate in the near future.