2017 Albania: Evaluation of the “Breaking the cycle of exclusion for Roma children through Early Childhood Development and Education” multi-country project in the Former Yugoslav Republic of Macedonia, Serbia and Albania
Author: Cornelia Rauchberger, Aleksandra Georgievska, Jelena Marković, Irida Agolli Nasufi, José Manuel Fresno, Skye Bain, Salvador Bustamante Aragonés
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as ‘Part 4’.
The multi-country project titled “Breaking the cycle of exclusion for Roma children through Early Childhood Development and Education” was implemented in the former Yugoslav Republic (fYR) of Macedonia, Serbia and Albania between 1 October 2013 and 31 December 2016 and funded by the Austrian Development Agency (ADA), with a total budget of 1,965,000 euros. This evaluation was conducted during the last two months of the project’s implementation and was finalised after the project ended.
This evaluation’s objective was to assess the standard Development Assistance Committee (DAC) evaluation criteria (i.e. relevance, efficiency, effectiveness, sustainability, and impact) of the project. As an end-evaluation which collected data during the last two months of the project’s implementation phase, the evaluation also aims to present lessons learnt from the project, with a particular focus on the system barriers that were aimed to be changed.
Besides UNICEF, its implementing partners and the funding agency (Austrian Development Agency), the project relied on a variety of direct and indirect beneficiaries including: public institutions; government representatives at national, regional, local level; CSOs; parents; and children who were represented by their parents for the purpose of this evaluation and in line with the evaluation’s ethical standards.
The purpose of this evaluation is to assess the ability of the three UNICEF country offices to fulfil their commitments as stipulated in the original project document (2013), its result framework (figure 1) and the requirements specifically envisaged by the Austrian Development Agency (ADA). Furthermore, the evaluation aims:
- To provide quality evidence to inform key stakeholders at national and county level in planning and delivering programmes/services for inclusion of the most vulnerable children in quality universal services.
- To shed light on remaining barriers and bottlenecks that have a significant influence especially on sustainability and explicability of results, especially relevant to early inclusion of Roma children.
The main objectives of this evaluation are to:
- Assess the relevance, efficiency, effectiveness, sustainability and, to the extent possible, impact of the project interventions;
- Identify any unintended outcomes;
- Identify and document lessons learnt in relation to strategies and interventions used to address certain critical bottlenecks; and
- Provide recommendations and guide the strategic planning for improved programmes for Roma early inclusion and education as part of the joint partnership programmes with respective Governments.
The findings and recommendations of this evaluation, as with any monitoring and evaluation activity, are intended to lay the ground for UNICEF and respective stakeholders in the three COs to improve their programming of Roma early inclusion and education interventions as part of the joint partnership programmes with the respective Governments.
Through the use of methodological pluralism, this evaluation triangulates information and provides findings in a convergent logic for the whole project as well as for each of the countries. The four complementary methods follow those listed in the ToRs, comprising of:
- Desk review: to gain an overall understanding of the project’s context, design, implementation and monitoring during implementation as well as identifying crucial information gaps. It was conducted during the inception phase of this evaluation and formed the basis for further developing evaluation methods. It was based on secondary sources (project progress reports, other reports and information on the project, governmental policies, strategies and plans, statistics, etc.) including those listed in the ToRs together with additional relevant documentation submitted by the respective UNICEF. The entire list of documents reviewed for this evaluation can be found in Annex 5 (Bibliography).
- Focus groups: to cover and contrast the opinions of different stakeholders involved in the project. They were held with public institutions, service providers and civil society organisations involved in the project but also with the Roma parents as this is the best way to consult beneficiaries so as to gain their opinion and to understand the effective impact of the project on their lives.
- Structured interviews: to gather specific information on concrete aspects of the project and to validate the consistency of the project. This allowed for triangulating information, views and opinions of different stakeholders that were either involved in the project or able to report on its observed results or effects.
- Structured observation: undertaken during the field visits to directly examine the extent to which the changes, presented in the project documents, had occurred in the field.
Findings and Conclusions:
The objectives of the intervention addressed the rights and needs of most marginalized groups. Adequate alignment with international instruments, standards and principles of human rights and gender equality confirm the project’s aim of reducing inequities, while the intervention’s focus on Roma children and children with disabilities are proof for its conformity with global and national equity-focused development priorities.
Despite the achievement of the intended results and system changes, the project framework lacks specific indicators that would allow to measure the actual effectiveness of achieved results, such as capacity building activities, policies or human rights and gender equality promotion. This lack of adequate indicators can be considered a major weakness of the project.
The project is considered to have efficiently converted the invested resources into the planned results. This conclusion is drawn from the fact that no overlaps with similar interventions nor alternative implementing strategies, that could have achieved the same results, were identified.
Having introduced major policy changes that promote ECD for early inclusion, the project has laid the essential foundations for the continuation of the results of the intervention. Demonstrated system strengthening and higher levels of engagement by the Roma community, public authorities, service providers and parents further contribute to the sustainability of the project.
The project has contributed to the narrowing of disparities between Roma and non-Roma children by ensuring access to pre-school and primary education and ECD health services.
- It is recommended that UNICEF continues and further strengthens its advocacy and technical advice to governments and relevant ministries with the aim of strengthening data collection and management systems that support data disaggregation and other data collection mechanisms in order to facilitate the provision of high-quality, timely and reliable data on children from ethnic minorities and children with disabilities.
- In order to ensure the sustainability of the initiative, the momentum of ECD for the inclusion of so-called vulnerable groups of children must be maintained in the region. Hence, it is recommended that UNICEF ensures the continuation of project results with a clear commitment from national stakeholders who should be engaged in this process.
- As part of the future project design with similar objective, UNICEF should develop a project-specific Monitoring and Evaluation (M&E) framework with clearly defined outcomes, outputs and activities and aligned with the Theory of Change UNICEF aims to pursue. It is also highly recommended that each of these elements (i.e. outcomes, outputs, and activities) include objectively verifiable indicators which not only allow for a sound monitoring of progress and the evaluation of results but also contribute to the knowledge generation during and after project implementation
- Roma CSOs should continue to be engaged in conducting an in-depth analysis of capacity gaps that may hinder Roma parents from enrolling or accessing ECD services for their children and taking into account all potential system barriers.
- For future interventions, the following proposals should be taken into consideration: Further develop mechanisms to work with parents; Further develop identified good practices; Continue developing the capacities of service providers.
An important lesson can be drawn from the system barriers that were targeted by several components of the project. While the project effectively addressed two crucial determinants on the demand side – namely “financial access” and “social and cultural practices and belief” – the evaluation has shed light on a potential capacity gap of secondary duty-bearers (Roma parents) that appears to affect these very same determinants, consequently posing new barriers to the enjoyment of rights by Roma children. One capacity gap relates to the financial instability of many Roma parents which make the indirect costs of ECD services (e.g. transport, material, food etc.) unaffordable (financial access) in those regions where no subsidies exist. The other capacity gap refers to the level of education of parents which was observed to have a direct impact on social and cultural practices and beliefs; or more precisely, low levels of education, especially illiteracy of parents, were reported to weaker parents’ willingness to enrol or access ECD services for their children.
Furthermore, the evaluation was able to shed light on the capacity gaps of service providers, which limit their abilities to address ECD issues in a comprehensive way. While such a capacity gap had been detected among educational staff in the original system barrier analysis (see chapter 2.), the evaluation further identified limited capacities of health services providers who, in cooperation with social welfare personnel, would be essential in the provision of comprehensive ECD services. Hence, the system barrier under the determinant “Access to adequately staffed services, facilities and information” needs to be expanded to “lack of qualified ECD-health service providers”.
Full report in PDF
PDF files require Acrobat Reader.