2018 Romania: Opportunities for Adolescents Formative Evaluation Report Romania
Author: Eugen Perianu (Team leader), Ana Maria Dalu (Evaluator), Alina Boșoi (Evaluator), Mihaela Stoichici Vârlan (Evaluator), Smaranda Witec (Evaluator)
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part ’ of the report, and the executive feedback summary labelled as ‘Part ’.
There are around 2 million adolescents (defined as children from 10 to 18 years old) living in Romania, out of which 47% are living in urban areas. Adolescents in Romania were historically targeted by HIV/AIDS programmes and policies and more recently through youth employment measures . Adolescents’ needs and problems are not clearly identified and therefore national policies, programmes and projects cannot provide an adequate response to these needs. Previous research conducted by UNICEF Romania shows that adolescents face major problems like poverty, social exclusion, and challenges related to education, health and employment. There are several risky behaviours that adolescents are most vulnerable including the use of alcohol, tobacco and illegal drugs, premature and unprotected sex.
The object of this evaluation is the modelling project OPPORTUNITIES FOR ADOLESCENTS, implemented in Romania between June 2014 – December 2017. The modelling project was designed and has been implemented by UNICEF Country Office in Romania, in partnership with a large number of stakeholders, both at central and local levels. The modelling project was designed to respond to the needs of adolescents in Romania, by testing its proposed intervention model at local level (5 cities in Romania- Bucharest, Bacău, Cluj Napoca, Constanța, Iași) via 5 newly established Resource Centres for Adolescents, one in each of the 5 cities.
The purpose of the formative evaluation is therefore to analyze the status of implementation, determine if the modelling project needs any adjustments and collect evidence related to the modelling project’s effectiveness and efficiency in reducing adolescent risky behaviours. The evaluation will also provide information on model’s relevance and will help identify lessons learned and good practices. The evaluation includes an assessment of the extent to which the evaluated modelling project was based on a child rights approach and whether age and gender issues are considered in providing support services to adolescents.
There are two main objectives to be attained by this evaluation:
a. Measure the relevance, effectiveness, efficiency and elements of sustainability of the modelling project in reducing risky behaviours in adolescents in 5 urban areas;
b. Identify the lessons learned and good practices and make recommendations for modelling project adjustment, as well as recommendations on how to promote legislation and policy development related to adolescents (information related to legislation, institutional building, and monitoring and evaluation mechanisms) with a particular focus on the definition of adolescents’ friendly services.
The overall scope of the evaluation is to assess the performance of the current modelling project “Opportunities for Adolescents” and make proposals for improvement or adjustment in terms of design, what has worked and what has not worked, internal and external factors that affected in a positive or negative way the modelling project implementation. The evaluation covers the period of implementation June 2014 – June 2016. The focus is on the five cities where the modelling project is implemented (Bucharest-4th sector, Bacău, Cluj-Napoca, Iaşi, Constanţa) and on the policy development at national level. The evaluation process includes all of the project stakeholders.
The evaluation methods used were: Desk analysis, Concept mapping, Interviews, Focus groups, Survey and Case studies. In total, the evaluation process included 48 interviews at central and local level, 12 focus groups, a survey and 4 case studies at local level.
Data sources were selected based on:
- the evaluation logic (where can the needed information be found);
- need to ensure a complete coverage of all of the modelling project stakeholders, both at central and local levels. In so doing, the evaluators have made sure that all of the perspectives on the modelling project have been captured: of duty-bearers, of rights-holders, of the project managers at different levels, of employees at the Resource Centres for Adolescents, of other stakeholders at national or central level.
Several measures were taken to ensure data quality: data collection instruments underwent an approval process by UNICEF, interviews and focus groups were properly organised, triangulation was ensured. All of the data planned to be collected was collected in good conditions and no significant limitation of access to data or methodological limitations were encountered. The entire evaluation process was however delayed by two changes in the team leader position and by a lengthy process of designing, improving and approving the data collection instruments. The delay has had no negative consequences on the data collection process per se, but the evaluation report will be available much later than foreseen. Nevertheless, all its conclusions, recommendations and lessons learned retain their validity and can still be put to use, in spite of the delay.
Findings and conclusions:
The project was adequately informed by thorough analyses that indicated the problems and gaps standing in the way of realisation of adolescents rights.
Central level activities had a direct focus on vulnerable adolescents: improved legal provisions, improved intervention methodologies for adolescents using drugs and adolescents living on the streets. At local level, activities of the Resource Centres for Adolescents were designed to reach vulnerable adolescents as well, either directly or by referral of vulnerable adolescents to the centres’ activities by other public services.
There are clear benefits for adolescents and parents due to the centres’ activities:
- Adolescents (including vulnerable adolescents that are referred to the centre) have developed their socio-emotional abilities, are well-informed and have a better understanding regarding the consequences of risky behaviours and are more able to make healthy choices and avoid such behaviours;
- Parents have a better understanding of the challenges facing adolescents and their emotional and support needs are more able to provide such support.
Each of the 5 Resource Centres for Adolescents had reached thousands of adolescents by June 2016.
Main constraints identified are: unsatisfactory cooperation between the local partners and the centre and the unsatisfactory involvement of the city administration in the modelling project. The modelling project has been less successful in reaching the most vulnerable adolescents: adolescents living in the streets, Roma adolescents, adolescents that are have an addiction.
The yearly cost per supported adolescent, as average of all the 5 centres, was 36 USD in year 1 of operations and 28 USD in year 2 of operation.
The model is completely replicable in other towns. All of its operating elements are independent of the town characteristics and can be built anywhere else and it is a matter of local priorities, willingness and resources to initiate such a project.
- UNICEF together with the city administration or other key stakeholders from the cities of Bacău, Cluj and Iași should approach their respective county administrations to present the intervention model and its achievements and explore means to replicate it in other towns or rural areas of the county.
- UNICEF should promote the intervention model at county level, in other counties that have not been involved in the modelling project.
- To support all of the potential initiatives at the local level throughout the country, as well as to substantiate its promotion activities, UNICEF should document the intervention model in a Guide published under the aegis of the UNICEF.
- UNICEF should use this evaluation to further influence central institutions
- All of the centres have reached stability with regard to their operating mode. The current volume of activity should be maintained for another 1-2 years, before any attempts to significantly increase resources and activity are made.
- The centres should actively seek in-kind contributions,: public-private partnerships, national programmes, EU-funded operational programmes etc.
- The centres should consider extending the present institutional partnership to other local relevant institutions (town or county level) either public institutions or NGOs, which would strengthen not only the operational network of the model, but also its resource base.
- The centres should continue talks with the National Authority for Child Rights Protection and Adoption related to the licensing process and identify the most suitable option: either to develop a new standard that fits the activities of the centres, or to introduce new provisions within existing standards.
- The website www.adolescenteen.ro should become the key on-line counselling platform for adolescents who have not yet attended the centers' activities, or who prefer to remain at a distance, and for adolescents from other locations throughout the country.
- Centres focusing on prevention of risky behaviours among adolescents are very relevant and are feasible. Simply put, it is needed and can be done;
- A modelling approach can be a very powerful tool to implement novel projects;
- A combination of project activities at policy level and at the operational level is an effective tool to manage a change process in any domain. However, it may also be more effective to focus on a smaller number of activities, clearly related and clearly reinforcing each other, rather than attempting to address many areas, as they are difficult to coordinate and their longer-term sustainability is difficult to support;
- National studies regarding the status of a domain before launching a national project in that domain not only increases credibility of the project plan in front of the project stakeholders, but are key to formulating relevant project objectives and activities;
- The role of UNICEF is very important. As pointed out repeatedly throughout this report, having UNICEF as donor, facilitator and catalyst makes a very big difference to how the authorities and other stakeholders respond to a project idea.
In a strict sense, many of the modelling project actual outcomes had not been planned exactly as they emerged. This is a modelling project, so its plan is, by definition, open to a considerable number of changes in light of the implementation experience.
Full report in PDF
PDF files require Acrobat Reader.