2012 CEE/CIS Regional: Evaluation of the Roma Good Start Initiative
Author: Sonal Zaveri, Ph.D
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 3' of the report."
Because of limited opportunities, Roma children are often not developmentally ready for primary schooling, are frequently placed in “special” schools, receive inferior education and are marginalized and stigmatized. Multiple deprivations in early childhood compromise normal growth and development and result in, among other things, poor participation and performance in school and the inter-generational transmission of poverty. The Roma Good Start Initiative, an EU-UNICEF Joint Management Project, emphasized Early Childhood Development (ECD) as an entry point for the social inclusion of Roma children. In promoting the survival, development, protection and participation rights of young Roma children, through a variety of research, capacity building, advocacy and communication activities, RGSI incorporated a strong equity focus. Designed and led by the UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States, this project involved the Open Society Foundations (OSF), Roma Education Fund (REF) International Step by Step Association (ISSA), and UNICEF country offices as well as local Roma NGOs and a variety of national government partners. RGSI built on and extended the Roma Early Childhood Inclusion studies, a pre-existing and continuing joint initiative of OSFs, REF and UNICEF. The initiative included six countries: the Czech Republic, the Former Yugoslav Republic of Macedonia, Hungary, Romania, Serbia, and Spain, and lasted 15 months (2010-2012
The evaluation objectives were to: i) assess achievements and constraints against project plans; ii) identify lessons learned; and iii) provide recommendations and possible next steps. The evaluation covered the project duration and was undertaken by an independent expert
The methodology adopted included extensive desk review, phone interviews, group discussions as well as one country visit
Findings and Conclusions:
Relevance. RGSI resonated with the EU’s and UNICEF’s commitment to Roma Inclusion, national plans and international and global rights based initiatives. RGSI responded to various EU social inclusion initiatives as well as prominent international rights-based frameworks, principles and programmes
Effectiveness. RGSI clearly contributed to the expected results: there was better understanding among key stakeholders regarding progress achieved, successful strategies and remaining challenges; better capacity of local authorities and Roma CSOs to strengthen or expand ECD; and better capacity of stakeholders and service providers to include Roma children in ECD and related activities. Contributing factors were the inclusion of Roma voices, multi-dimensionality of RGSI regional and international partners, participation of key and diverse stakeholders and high quality of RGSI activities – research, capacity building and communication materials.
Efficiency. RGSI was supported by the EU (core funding of Euro 424,747) and co-financed by UNICEF (Euro 106,187). ISSA contributed Euro 24, 332. Although costs were underestimated, funds were efficiently managed by UNICEF RO and partners to meet project needs and targets. Targets for communication products and services, research reports as well as advocacy and capacity building events were in most instances exceeded. There were difficulties experienced in conducting and completing costing studies because of the extreme shortage of data on Roma necessary for making valid calculations.
Potential Impact and Sustainability. Impact and sustainability could only be partially assessed as the implementation period was short (15 months) and the evaluation was commissioned as soon as the project was completed. There is some evidence of use and influence of the research and capacity building activities towards policy change. RGSI resonated with other policy initiatives, helped to reinforce them and was positioned at the right time addressing an emerging (and felt) gap (early childhood development and learning among Roma) as a means to addressing inclusion from the very beginning.
Coordination Coordination and consensus-building slowed down implementation but was considered valuable. Sharing and coordination at multiple levels was important for research, capacity building, organizing events and providing feedback on documents. Overall, international/regional partners’ support was extremely valued at the country level; at regional level some partners’ expectations did not always converge.
Results Based Management Close monitoring by UNICEF RO led to completion of most targets; the UNICEF project manager played a key role in coordinating with all stakeholders to monitor progress. Required changes were made in implementation based on implementation feedback.
Rights Based Approach The RGSI goal expressly states inclusion and ECD as an entry point to mainstream Roma children and protection against stigma; evidence from research and capacity building indicate the need and benefits of doing so. The inclusion of REF as a partner and of Roma NGOs and Roma researchers (in RECI) indicate a rights-based approach in design. First person video accounts of Roma mothers’ aspirations for their children sensitized decision makers deeply; this was more “real” than advocacy by Roma NGOs or through print material.
Multiple data sources were used and the desk review started a month before the site visit.
1. Desk review of documents to assess inputs, activities and outputs; analysis of existing international and national policies/priorities; mining emails regarding discussion threads
2. Interviews with key stakeholders, including government representatives, project partners (REF, OSF, ISSA) and Roma civil society organizations as well as thought leaders
3. FGD with service providers (e.g. Roma health and education mediators, members of NGOs, etc.)
4. Field visit to one of the countries where RGSI is being implemented that included face to face meetings, visit to a Multi-functional early learning site and discussion with local authority supporting the site.
5. Virtual interviews with another country mirroring the country field visit.
The suggested sampling size was 10 in-depth interviews with key stakeholders and partners and at least two focus group discussions with beneficiaries/service-providers. The sampling size exceeded targets with 20 individual interviews, two group interviews and two focus group discussions to ensure maximum representation in the sampling. At the country level, Romania and Serbia were studied in depth, with some interviewing in Macedonia. Regional and international partners were interviewed.
Recommendations for regional and international organizations involved in ECD and Roma issues
1. There should be an extension of RGSI to consolidate gains made. Future efforts should be designed to align and synergize with other Roma initiatives - both policy (EU, regional and country level) and programme.
2. At periodic intervals (e.g. five years), regional partners should provide/make available technical support to country level counterparts to repeat the RECI studies to plot change and ensure up to date information availability and use, especially since baseline data is now available for comparison.
3. To maintain momentum and ensure sustainability regarding the importance of early development and learning for Roma, regional partners should support governments to develop and strengthen inter-sectoral policies and systems.
4. Policy changes should be monitored and expert inputs provided as needed to ensure the early development and learning inclusion of Roma children.
5. International agencies should harmonize and align early development, learning and Roma inclusion5. essages and drive the agenda in a targeted advocacy to achieve stronger results.
Recommendations for country decision makers
- In the next phase, build on RGSI success with upstream advocacy and advocate for early development and learning and Roma inclusion at multiple levels for downstream advocacy
- In the next phase countries should have a regional perspective and consultation in order to share data, best practices and lessons learnt with regard to policies and programmes
- Synergy with on-going efforts and identifying gaps in information that are of value, increases acceptance from all stakeholders.
- Stakeholders are able to overcome bias, increase understanding, enhance capacity and take action if supported by high quality research, expertise, advocacy and dialogue with a cross-section of relevant stakeholders including diverse decision makers, NGOs and marginalized voices.
- The right issues, expertly addressed at the right time, involving the right people in an inclusive approach, are important contributing factors to achieve impact. However, policy change is slow and long-term requiring sustained advocacy efforts.
- The voices and participation of marginalized communities (including importantly poor and marginalized women who are key ECD stakeholders) together with those of international expertise are critical to implement a rights-based approach. Early learning opportunities and inclusion of marginalized children not only provide a head start for learning but also protect young children against stigma and discrimination.
Full report in PDF
PDF files require Acrobat Reader.