Evaluation database

Evaluation report

2009 Romania: Report From the Evaluation of Education Projects in Osterode



Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:

Presently, Osterode at the moment hosts 92 families, with a total of 380 people, out of which 235 are children. RAE communities began to move from Zitkovac and Kablare camps in North Mitrovica in the former French military base “Osterode” in February 2006. During 2007, a small part of RAE families, around 15, were returned to locations where they previously resided, namely in Roma Mahala in south part of Mitrovica. Osterode has been in the media headlines and subject to numerous reports made by international organizations due to the reported high level of lead exposure, leading to possible health risks among the population.

The educational projects in Osterode Camp supported by UNICEF have been quite successful in achieving their anticipated results. While depending on the nature of activities and the intervention type, the level of success in achieving the desired impact varies, the overall implementation has been perceived by most of the respondents as successful. The success of project activities during the intervention period has been hindered mostly by the implementing environment, which most of the time has been unstable, over politicized, and quite challenging to operate in. Viewed from this perspective, the projects have been adequately designed as they were implemented without any major problems. Freedom of movement of UNICEF staff from Prishtina Office (local programme staff of Albanian ethnicity) to the north after the declaration of independence has had a visible impact on the overall project continuation.

The problems and issues addressed through the UNICEF rights‐based programming have generally been focused in the biggest priorities of the communities, education and health. The only two issues categorized as being of highest priority for the beneficiaries that have not been addressed by the UNICEF projects is the issue of final re‐settlement of the camp population and poverty. The latter two issues are extremely complex and very costly, and not necessarily are part of UNICEF mission.

Concerning the activities that have been implemented the rights‐based approach inherent in all UNICEF programming has had some difficulties to be implemented as it has created a dependency of the population on the projects and continuous support. This has been caused mainly by the inability and/or unwillingness of the primary duty bearers—the local authorities, to take over ownership and assist UNICEF in phasing out its support.

While the accurate measurement of net impact that the projects have had is very difficult to carry out due to the absence of a baseline study, both concrete and soft results can be identified. Attendance rate of 95% in elementary education level and the overall number of 42 children attending pre‐school education is a quantifiable result achieved primarily because of the projects implemented. Increased willingness of children to attend school, qualitative improvement in their performance due to extra assistance with homework and learning ability are also visible results but cannot be quantified.

Additional impact noticed during the camp visits are the increased level of knowledge on good parenting concepts and practices and relatively good knowledge on health related issues and more precisely, the lead poisoning and treatment which presents an immediate threat to children and adults living in the camp.

Communication and coordination between all stakeholders has not been at its best during the project implementation, hence causing at times unsuccessful information sharing between the interested parties. This improvement would increase the impact of the activities in the future, contribute towards developing local ownership of activities and consequently, towards the sustainability of interventions.

On the content aspect, one remark that should be made is to increase the focus of projects on language skills of children. The reason behind such suggestion is that language skills have been identified to be the biggest obstacle for children to continue education beyond elementary school as their current conduct of Serbian (the official language of instruction in the north where they will presumably continue education) is not satisfactory. An additional component hindering maximum participation and achievement of children in school is the lack of didactic materials due to the poor financial situation of their families. As a measure having an immediate impact and as a pre‐condition for the future education of the RAE children in Osterode, UNICEF should seriously consider supplying these children with the necessary things that currently are unavailable to them.

Concerning the health situation, the projects have had an impact only in raising the awareness of the population about the risk of lead poisoning and its effect on their health, however, it has not helped them significantly to lower the lead level in the blood. To this end, only permanent settlement in an area free of exposure could yield results and that these results can only be achieved in longer periods of time.

UNICEF has been very active in Osterode camp since 2006 through continuation of projects implemented by the Association for Peace Kosovo (AFPK) and Caritas Kosovo with their offices in the Osterode Camp.

Purpose/Objectives: 

The main objectives of the implemented projects have been education and recreational activities aiming at improving the psycho‐social and physical abilities of RAE children living in Osterode camp and their successful integration in the educational and social structures in the north of Kosovo.

Projects have been granted to Caritas Kosovo and AFPK due to their continuous relationship with UNICEF and their experience in working with the RAE population in Osterode camp. Projects before 2007 have been implemented jointly with Caritas Kosovo, however, due to internal issues within Caritas Kosovo, AFPK continued implementing successive project activities in 2009 on their own. In addition to educational and recreational activities, it was agreed with AFPK in addition to health education for lead pollution to expand and implement activities on better parenting with aim to improve child care, development and safe motherhood of RAE parents living in the camp.

Methodology:

The methodology of the projects is based on educational support, health prevention and supporting interaction and communication between members of population, such as children, parents, teachers and others.

The project methodology, according to the documentation provided by UNICEF for the purposes of project evaluation, that includes various documentation given by AFPK is based on activities , aimed to directly intervene at improving school and other skills of RAE children in the Osterode Camp. Activities consist of those with children, such as educational and recreational, and those with parents, that are further divided into three subgroups, including home visits from RAE facilitators, round table discussions with parents, and round tables with teachers. Educational activities with children include preschool and homework support activities, catch‐up and Roma language classes, as well as creation of the children profiles by pedagogues. Recreational activities include course in Roma folklore, sports, film projections with educational purposes, summer camp and other special events.

Key Findings:

UNICEF is the only international organization still supporting projects in Osterode and the impact of its projects has been significant. Generally, the projects have addressed the biggest concerns of the population living in Osterode, however, the evaluation indicates that there is room for additional interventions and application of new methods. Below the findings of the evaluation are disseminated according to the source of feedback received, thus providing a clear picture of attitudes and opinions of various stakeholders involved.

The overall project efficiency (project cost per beneficiary) requires the application of a very complex methodology and will be impossible to do it at this stage, because children (the main project target) may have benefited from more than one activity, hence, making the ‘per activity’ cost impossible to calculate. Because of this situation, the evaluation team has chosen to conduct simple calculations of overall cost of UNICEF support in Osterode divided by the number of beneficiaries (direct and indirect) and generate a rough estimation cost efficiency, which is considered to be at a very satisfactory level.

The basis for this calculation has been the number of camp inhabitants at the time of intervention as directly or indirectly all people in the camp have benefited from UNICEF supported projects.

In 2006 and 2007, UNICEF has spent around 42,000 EUR in direct activity funding for its implementing partners. When considering that the camp population at that time was a little over 400 persons, which directly or indirectly have all benefited from these activities, per person cost of investment for these two years is calculated to be at 105 EUR, or 52,5 EUR per person annually.

During the 2007, around 15 families have resettled in the Roma Mahalla in Mitrovica South leaving the camp population at an estimated 380 persons, out of which 235 children. The investments made by UNICEF in direct project activities in 2008 and 2009 have been 48,576 EUR. When this investment is divided by the number of persons residing in the camp, the per‐person investment is around 128 EUR per person, or around 85 EUR annually. Both periods (2006‐2007 and 2008‐209) indicate a high level of cost efficiency. It should be noted that these costs do not include the operational costs incurred by UNICEF.

Conclusion:

The only remark concerning community involvement and inclusion of local authorities in achieving sustainable impact on the target population relates to the phasing‐off of the UNICEF support. To this end, it should be noted that UNICEF could have generated greater support from the municipal authorities in the north, hence, reducing the dependency of the communities from the UNICEF support, and creating grounds for gradual withdrawal from the camp.

The Osterode camp has been provided only as a temporary solution and this has influenced the UNICEF activities in terms of challenges to long terms planning and sustainability of projects. The latest initiatives from the local Kosovo government and the international community to find a permanent solution to the RAE housing problem will inadvertently change some of UNICEF projects components in reflection to the new situation in the ground. The housing solution could both prove advantageous, since it could provide an opportunity for long term solution to health and education issues, but could also become disrupting, considering that there is a level of reluctance from some of the residents to move to Roma Mahala.

In order to clearly and accurately illustrate the communication and coordination between various actors during the implementation of the UNICEF supported activities, below the positions of the actors and their role in improving the situation on the ground is presented in a graph.

As it can be seen in the visual presentation on page 24 of the communication flow, the information dissemination and involvement of various actors has been at times unsatisfactory. The strongest communication and coordination can be noticed between the camp population, their leader (internal constituencies) and AFPK the implementing partner, which in the graph appears as semi‐internal actor1.

Second to this, is the communication between UNICEF Prishtina Office and the UNICEF Zone Office in Zvecan, which most of the time has been very effective, with the link being broken down occasionally, and the communication between UNICEF Prishtina and its central level counterparts such as WHO, or other UN agencies partaking in the UNKT meetings. The reason why the communication flow between these actors is considered to need improvement is because of the multi‐level communication which is the cause for occasional incomplete dissemination of information within this group, partially caused by the complexity of the problems and the situation in the ground.

The worst information flow concerns the coordination of all stakeholders with the municipal authorities in the north. While this may have been a product of various factors, the most important being the overall political situation after February 2008 Declaration of Independence of Kosovo, other factors may have also been influential in causing this shortcoming.

In addition to the communication, the complexity of the situation has also hindered the project success.  Lack of a final settlement solution for the camp population has partially been caused by the circumstances, but also various forces and factors, including here, the alarming economic situation, RAE culture, as well as the environmental hot spot in which the camp is situated, have also been quite influential. The chart below illustrates the implementing environment visually, and acknowledges the influences of these forces in project implementation and hindering UNICEF and its implementing partners in achieving the desired results.

AFPK has been situated as both an external and internal actor because of its specific role of mediating between the external interventions, and local community. The presence in the camp and at the same time, activities outside the camp is the reason for positioning AFPK in the borderline between internal and external forces.

Recommendations:

Based on these findings there are possibilities to improve aspects of programming and implementation.  The recommendations bellow have been divided into recommendations aimed at UNICEF projects components and recommendations towards external stakeholders, which include recommendations for systematic changes that are beyond intervention possibilities of a single agency.

Recommendations for UNICEF:
Organizational Issues
􀂙 Reporting should be reviewed on several levels, including reporting between local offices of UNICEF and the central office, as well between the implementing partner for UNICEF projects in Osterode and facilitators. During the project evaluation it was evident that a certain number of information was not available to central offices of UNICEF, and communication was held between offices due to internal communication issues.
􀂙 Project evaluation and monitoring has become heavily dependent on reporting by the local implementing partner and Roma facilitators. This presents a specific challenge to UNICEF projects, since it is a natural tendency of these groups to present overly positive evaluation of projects in order for them to continue working with UNICEF. A regular visit by local UNICEF officers and their reporting to central offices in regards to project development would mitigate such problems in the future as it would improve the direct project monitoring.
􀂙 Facilitator performance needs to be linked with their payment. During project evaluation, it was apparent that facilitators considered that they were eligible for payment no matter what the outcome of the projects was. Facilitators should be clear what they are being paid for and also when the project is expected to end. Activities aimed at raising the awareness of facilitators on education and health issues were aimed at creating future members of community that would be able to spread this information to their fellow residents. Even though UNICEF has told the facilitators that their engagement is temporary, they continue to perceive themselves as being irreplaceable.
􀂙 When asked about UNICEF and its projects, parents and teachers had difficulties in identifying which projects were implemented by which organization. This shows that promotion campaigns aiming at promoting both project outcomes and UNICEF have been insufficient. Projects need to contain budget lines and components specifically aimed at promotion.
􀂙 Teachers have complained that RAE parents do not come to meetings consistently. UNICEF has to find a way to make this presence mandatory if necessary, to the need of parents to understand the importance of education of their children in these meetings, as well as monitor their children’s progress.
􀂙 Timing of projects needs to be synchronized with school year activities, so there is minimal discrepancy in activities.
Education Issues
􀂙 Child centered programming should be implemented as part of projects, since during the evaluation children showed passivity in answering. Parents, and local NGO staff either answered for them, or required specific answering from children. Even though there was no direct classroom observing, focus group discussions show that Osterode children remain passive observers instead of being active participants.
􀂙 Recreational activities need to be organized in more efficient manner, since it has been noticed that many problems, such as reluctance of locals to accept RAE population in their land in the lake in north of Kosovo, were not foreseen. Recreational activities need to be intertwined with project components that have the objective to teach children through play. It seems at this moment that there is a clear division between recreational activities and educational activities, whereas there could be components that mix the both.
􀂙 Activities like homework support and language support need to be intensified for children from 5th to 8th grade due to an identified deterioration in their performance and increased dropout rate with children in this range.
􀂙 One of the biggest obstacles for children to be integrated within the social structures in Mitrovica North has been insufficient knowledge of Serbian language. Projects need to expand their activities aiming at teaching children the Serbian language. A possible recommendation would be to hire a local language teacher that would work exclusively with Roma children to teach them Serbian language in Osterode camp grounds. Exposure of children to two languages, Albanian language at family surroundings and Serbian language in schools also could prove an asset in the future, since Roma children could become basically proficient in both languages thus becoming adaptable in all the future scenarios.
􀂙 Roma language instructions have been put into question by respondents, due to the lack of usage of this language within the camp grounds. Considering the burden on children to learn Serbian language, their use of Albanian language in family surroundings, activities need to reconsider the importance of Roma language, and arrange activities that best reflect the practical need of each of above mentioned languages in school and other surroundings. This is not to say that Roma language has no significance whatsoever. Roma cultural activities, as part of the recreational project components are seen to be extremely beneficial to the general well being of children and they should be continued. However, as Serbian language instructions should be increased at every stage of child education, the Roma languages should be reevaluated
as not to overburden the education curricula.
􀂙 When asked in focus group sessions on what they would like to become when they grow up, children almost unanimously expressed that they would wish to become teachers or pedagogues. This is normal considering the level of presence of teachers and pedagogues in their lives, and also is a positive role model. But this also shows that parents are not serving as proper role models in children lives. Future activities should include presentations by persons that could serve as role models, preferably members of the RAE community inside or outside of the Osterode camp, and also general introductions to available professions in Kosovo.
Health Issues
􀂙 Insufficient child nutrition, especially during breakfast time, has been identified as a major factor for reluctance of Roma parents to send their children to school. As basic as this rationale is, UNICEF staff needs to take into consideration that projects need to include a certain amount of budget for food, if school attendance is to be successful, at least at this stage of project.  Alternatively, UNICEF should advocate with school authorities and other institutions at the local level to provide food and other necessary items to Roma children so they could continue education.
􀂙 There is a general satisfaction with the BPI initiatives and results are very satisfying, according to most of focus groups feedback. It is recommended that there is a continuation of these activities in the future. Inclusion of larger number of women facilitators may be one of the necessary changes to maximize the impact of BPI activities as it would facilitate better interaction between project staff (facilitators) and mothers.
􀂙 Health education campaigns on led poisoning and prevention have been very successful ad should be continued. This is proven by the vast knowledge of parents about led‐related issues, however, the follow up activities (i.e. treatment) have lagged behind due to the socio‐economic situation of RAE communities in Osterode and the overall situation of these communities.
􀂙 Women participation in activities has been seen as especially beneficial during BPI implementation. Considering that this evaluation came from RAE community male representatives during focus group discussions that are usually conservative in this respect, this presents a very good opportunity to use such window of opportunity in the future, not only in health related activities, but also in educational activities. Project activities should include female facilitators, and other possibilities for direct involvement of women in most of activities envisaged by UNICEF projects.
Recommendations for External Stakeholders (including recommendations for systematic macro changes):
􀂙 Roma population in the Osterode camp is aware that at one point in time they will move from the camp into a different location. Even though that would mean a possible end of lead poisoning, the majority of Roma population fear from moving to Mitrovica South due to possible retaliation from the local Albanian population. In addition, most of the population is exhausted from moving from location to another, and they have given up hope almost entirely. However, UNICEF needs to be aware of these developments and act accordingly by making the necessary changes to the project.
􀂙 Moving of RAE residents in Roma Mahala presents a direct challenge for their future children education. Some of RAE parents have said that even if they move to the Roma Mahala in the future, they would want their children to continue the education in north of Mitrovica. This request has to be taken into account, as possibilities of organized transport by local authorities should be seriously considered. However, keeping RAE children from schools in south of Mitrovica could cause a form of segregation of children from the local Albanian community, so this issue should also be considered. To this end, UNICEF should actively engage in advocating on behalf of these communities with the local authorities.
􀂙 Some members of the RAE community in Osterode, and especially facilitators, have been unhappy about the health interventions on their children, especially the medicine given to them. It is advisable that activities and interventions are devised in the future that will regain the trust of the community.
􀂙 Living conditions of the RAE population in Osterode remain very unfavorable. Unemployment is set at a very high rate, and social assistance is not provided on regular basis. This causes problems for all the activities done by international donors (including UNICEF). Existing advocacy coalitions, or groups established for this specific purposes should aggressively lobby with local institutions both in north and south of Mitrovica to provide the RAE community with basic access to nutrition, medicine, as well as clothing, food and school material for RAE children, so they could continue to attend education regularly.
􀂙 Residents have complained especially about the access to health facilities and medicine. Depending of their final place of residence, one or many organizations, both international and local, should advocate for easy access to health institutions by the RAE population, and provision of medicine, especially concerning the treatment of lead poisoning of their children.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search