Evaluation database

Evaluation report

2003 CEE/CIS: Study of the Northern Caucaus Emergency Programme, Nov 1999 to Dec 2002

Author: Broughton, B.

Executive summary

Background:

The UN's stated objectives in the Northern Caucasus have, from the outset, included protection of the right not to be forcibly returned from Ingushetia to Chechnya and, more broadly, the right to safety, in both cases in the context of international instruments and Russian legislation. The UN agencies, including UNICEF, have succeeded in providing a safe haven in Ingushetia by meeting basic needs and providing essential services, and the UN's engagement of the authorities through the provision of this assistance has helped to restrain the authorities from forcibly returning IDPs (although not always prevented it as the closure of Aki Yurt attests).

Purpose/Objective:

The objective of the study is to 'produce a document encompassing the description of the beginning, development, operational increase and implementation of the UNICEF Emergency Programme (EP) for the Northern Caucasus, including the full set of experience and lessons learned acquired between the end of 1999 and the end of 2002, and general comments on funding and resource mobilisation.' The terms of reference also provide that the study should 'include indicative advice/guidelines for future developments of the Programme' and that the study 'should be used as a baseline document for a comprehensive evaluation of the same programme in two to three years from now.'

Methodology:

A questionnaire was completed by UNICEF managers and staff in advance of the study. The Emergency Programme Coordinator subsequently met with field staff and developed a preliminary list of lessons learned. The consultant reviewed key documents, including CAP documents and UNICEF Donor Updates, SitReps and Annual Reports. During a two-week visit, interviews were conducted in Moscow and in the Northern Caucasus (Vladikavkaz and Nazran) with UNICEF managers and staff, UN agencies, several partner NGOs and two donors. Visits were made to the Vladikavkaz Prosthetic Workshop in North-Ossetia and the Sputnik IDP camp in Ingushetia.

The study relies on the documents studied and the people interviewed. It was not possible, in the time available, to systematically cross check statements and reports on progress and reports. There was a high level of involvement of UNICEF managers and staff, arguably detracting from the independence of the study, but a necessity. Their contributions were invaluable. There was very limited consultation of beneficiaries and community leaders and no attempt was made to involve children and women in the conduct of the study.

Findings and Conclusions:

UNICEF is quite properly credited with having undertaken excellent work in the education sector in the Northern Caucasus, in which it is the lead agency. The work has been relevant and necessary, clearly within UNICEF's mandate to lead efforts to ensure children's access to basic education and, in the circumstances of displacement, preserve educational attainment. Moreover, basic education, recreation and sport do serve a protection function in so far as schools are normally safe environments, while recreation and sport are therapeutic and constitute an alternative to illegal, dangerous activities. This has worked well in Ingushetia, where school enrolment is now 85% for primary-age children and 60% for secondary-age children (even if it took longer to achieve than had been anticipated). In Chechnya, as a result of funding and security constraints, UNICEF and its partners have only been able to provide limited assistance to education, principally the rehabilitation of a small number of damaged schools.

UNICEF's action in raising awareness about the dangers of mines/UXO and directly assisting victims has been important, and the establishment of the IMSMA database has been an excellent initiative. Without detracting from the effort involved in mine awareness, it would be reasonable to question its effectiveness in changing behaviours in the absence of data to demonstrate results. Assistance to mine/UXO victims has been comprehensive and considerable, with 260 victims fitted with prosthetics and orthotics, and some degree of physical and psychosocial rehabilitation. UNICEF, however, should recognise that it has come to a crossroads. It can't support the Vladikavkaz prosthetic workshop in North Ossetia indefinitely. UNICEF needs to urgently develop a strategy for phasing out assistance to this workshop and help develop prosthetic, orthotic, rehabilitative and/or psychosocial capacity within Chechnya.

A large number of people in Ingushetia and Chechnya were assisted by UNICEF's provision of medical supplies and equipment, if only in a small way for some bulk items. It is impossible to say how many, because UNICEF has not had control over their final distribution. However, even if targets were not met and impact data is not available, preventative and curative aid provided through the MoH network and partners must have played a role in preventing the outbreaks of epidemic diseases that were initially feared and in decreasing morbidity related to communicable diseases, and mother and child health. The Representative's overall assessment is that although UNICEF's assistance in this sector has been useful, UNICEF has found itself playing a rather limited traditional supply-oriented role. This is not to say that UNICEF has not been focused. The assistance in rehabilitating the cold-chain and EPI is a case in point – very valuable, even if mainly involving supply.

UNICEF has managed to assume a key role in the provision of water and sanitation in Grozny, thanks to the partnership with the Polish Humanitarian Organisation. Very tangible results have been achieved: the considerable quantities of water provided remain indispensable and the attention to garbage collection and incineration of medical waste is smart. The main challenge now is sanitation, particularly the disposal of faecal waste. In Ingushetia, in contrast, UNICEF settled down into providing what could be described as well-focused but limited assistance. UNICEF deserves credit for identifying a niche supply role (bladders, chloramide, etc.), although it is evident that UNICEF stepped back from a potentially bigger role (although probably wisely under the circumstances).

The Consolidated Appeals for the Northern Caucasus have been relatively successful and UNICEF, in particular, has been very effective in raising funds ($14.08 million from November 1999 to end-2002, representing 96% of the total requested). There have been, however, difficulties in the flow of contributions, most notably in 2002, where 60% of requested funds were not received until after September. The donors interviewed for the study were very positive about UNICEF – good profile as an agency, clear priorities and a good program, with education a very strong component. This success does not come easily and the EP Coordinator spends the majority of his time liaising with donors and managing allocations.

Although the EP is well-managed, there is no document that translates the CAP into a proper plan of work with targets, dates, indicators, etc. More rigour is required in the implementation of the EP and this calls for the development of rolling plans. In addition, transition plans are needed for each sector to support the shift in focus to Chechnya.

A weakness in the EP is the lack of application of UNICEF standards and experience. It was initially believed that IDPs would not remain long in Ingushetia, resulting in limited investments in staffing and training. Nevertheless, UNICEF has reached the point three years later where the EP, in terms of programming and staff, is insufficiently infused with UNICEF standards and experience. Putting it a different way, it is evident from talking to implementing partners about UNICEF's activities and from speaking to UNICEF staff, that UNICEF is not adding value as UNICEF (as distinct from any other funding agency). Changing this will require familiarising staff and implementing partners with UNICEF standards and experience, training staff in their application, and reflecting them in performance indicators as well as in agreements with implementing partners.

Monitoring and reporting also needs to be strengthened substantially. There are two problems. Firstly, just as proper workplans have not been developed, performance in meeting targets and objectives is not tracked or reported systematically. Secondly, everything that is reported is at least partially intended for donor consumption (for example, through Annual Reports and SitReps). As a result, progress and performance are not assessed with sufficient frankness or rigour. UNICEF needs to make a commitment to intensify regular monitoring and reporting, and apply more rigorous and systematic standards to test the effectiveness and efficiency of the EP. An internal reporting system, parallel with the existing external reporting, would enable managers to make decisions that can continually test progress and enhance performance.

Recommendations:

Notwithstanding the difficulties, it is recommended in this report that UNICEF 'stands on its mandate' in relation to the protection of children and women in Ingushetia and Chechnya, putting its positive image and credibility to maximum use, even if this involves some risk. The Country Office should start by defining the rights of children and women that need to be protected in the context of the Northern Caucasus and then determine (a) how sector-based interventions in Chechnya (i.e. education, health, etc.) could incorporate protection for children (particularly male adolescents) and women caught in the conflict, and (b) which protection needs can't reasonably be expected to be met through sectoral programming and how they could otherwise be addressed. The latter is likely to require a stand-alone protection component, ideally involving several agencies working together.