2018 Romania: Summative evaluation of "First Priority: No more 'invisible' children” modelling project 2011-2015
Author: Alina-Maria Uricec, Gabriela Tănase, Team Leader - Irina Lonean, Evaluation Experts - Liliana Roșu Adriana Blănaru Alina Bîrsan Anda Mihăescu Ecaterina Stativă Oana Clocotici
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as ‘Part 4’.
According to 2011 data, children and young people were seriously affected by the economic recession. In this context, in 2011, UNICEF began the implementation of the modelling project initially called “Helping the ‘Invisible’ Children”, later renamed “First Priority: No More ‘Invisible’ Children!”. Based on the theory that children’s welfare in Romania will improve only if and when they, especially those worst-off (‘invisible’), will have enhanced access to social services (education, health and social assistance services), the modelling project aimed to provide a possible solution to increase the impact of social policies on children and their families. In rural areas, particularly the poorest ones, the capacity of local authorities was developed to enable a rapid identification of most vulnerable children and their families, while preventing the escalation of problems such as child-family separation, by delivering the minimum package of services. The project was initiated in 2011 in 96 communes located in the disadvantaged rural areas of 8 counties , promoting the social workers’s outreach activities and an in-depth knowledge of local children’s vulnerabilities and problems. Following the formative evaluations conducted in 2011 and 2013, the project coverage was reduced to 64 communes and later on to 32 communes, while maintaining an even distribution across all the 8 target counties. Starting 2014 it also included a comprehensive working methodology called Aurora, designed for the management of project activities. Aurora uses a data collection tool (a questionnaire) to build a database on vulnerable children and their families and allows for monitoring the dynamics of children’s vulnerabilities and of the services provided.
The overall goal of the present summative evaluation is to assess the impact of the “First Priority: No More ‘Invisible’ Children!” demonstration project in addressing the challenges faced by children and their families from rural areas, particularly disadvantaged communities, in accessing basic services. The evaluation also aims to address the evaluation questions, based on the following OECD-DAC criteria: relevance, effectiveness, efficiency, sustainability, impact and lessons learned.
According to its Terms of Reference this evaluation independently assessed:
- the impact of the modelling project and the extent to which the intervention and all its components have contributed to improving children’s welfare through enhanced access of children and their families to basic social services (education, health, and social assistance services), particularly in the selected rural disadvantaged areas;
- whether the modelling intervention was relevant to the target group, as well as in the national and international contexts;
- how results and evidence generated by the model contributed to improving the impact of social protection policies on the poorest and most vulnerable children and families in Romania;
- effectiveness in terms of results obtained during project implementation;
- the lessons learned, key bottlenecks and good practices;
- how efficient the model was in developing new services and improving the life of children and their families;
- how effective the model was in producing the expected results;
- sustainability of model at the local, county and national levels;
- the extent to which the model could be replicated at national level through the revision and/or development of the normative framework, standards, methodologies, budgets.
The evaluation methodology involved the use of both quantitative and qualitative methods, and of primary as well as secondary data.
The evaluation included a counterfactual analysis of the ‘invisible’ children identified in the 32 communities which were no longer part of the project as of 2012. These children were to receive the services available in their communities but not the services provided under the evaluated project. Data for the counterfactual analysis were collected using a household survey focused on 64 communes: 32 intervention communes and the 32 control communes where project implementation occurred in 2011 only. The sample volume for intervention communes was 428 households, and the one for control communes was 415 households – a random, two-stage and stratified sampling. Data were collected on all household members, namely 4,243 individuals. Maximum margin of survey error is +/- 3.4 percent at 95 percent confidence level, and 4.7 percent at the intervention sample and control sample level (minimum 95 percent confidence level).
To answer the evaluation questions, secondary data analysis was also used: data of the household surveys conducted in 2011 and 2013 during the formative evaluations of the model; the consolidated database created by social workers in 2012; the databases resulting from the use of the Aurora methodology during 2014 – 2016 (including after the completion of the demonstration project); the database of entries and exits into/out of the child protection system, centralised with the support of the GDSACP supervisors, and data on the work of the community health nurses, in the intervention and control communes (Botoșani county), collected with the support of DPH supervisors. Other data sources included documents made available by UNICEF, such as reports of supervisors and of professionals at the local level. In addition, interviews were conducted at all levels of modelling project implementation.
Findings and conclusions:
First, the evaluation shows that identifying children’s vulnerabilities is key to enabling the delivery of services. Most vulnerabilities are not known and cannot be identified without the aid of a tool as Aurora. The least visible vulnerabilities are those connected to domestic violence, risk behaviour and the situation of children with only one or no parent at home. The model created a guide for the community workers in addressing all the identified vulnerabilities, which makes the model highly relevant in relation to the needs of the invisible children.
Second, the model is effective in ensuring the delivery of basic social services and community health care services via community workers. The information and counselling services are more effective than the referral, accompaniment and support services. The more comprehensive counselling services provided by the community counselling and support centres for parents and children within the micro-grant projects were well regarded by all community workers as well as by the children and parents interviewed.
Third, integrating the activities of the community workers was possible due to the design of the working tools used in the model. Social/outreach workers and community health nurses rated their teamwork highly and appreciated the fact that teamwork provided them with a chance to consider a case and approach service delivery from different professional perspectives.
Fourth, the project generated considerable impact on its target group in terms of vulnerability identification, access to social services, including specialised services for children with disabilities, and access to community health care. For service recipients, the access to primary health care (particularly vaccination) increased compared to their previous situation, while risk behaviours and situations of child abuse, violence or neglect decreased. The project has a moderate impact on protecting children from being separated from their family.
– Continue the advocacy efforts to ensure that national public policies cover not only the identification of vulnerabilities but also the standardised assessment of the identified vulnerabilities and the minimum package of services
– Promote the minimum package of services and the component of vulnerabilities identification and needs assessment, by using Aurora nationwide as a modern tool that enables the identification of children’s problems and needs, including those less visible to community members
– Test the possibility of identifying all ‘invisible’ children in the communities, by conducting a comprehensive identification of community households and children. Local professionals can use Aurora to identify all households in a community, by carrying out a community census
– Fine-tune the Aurora application to better serve the case management methodology, by improving the way cases are being managed and the flagging of services recommended in the minimum package of services.
– Develop a new Aurora application module that provides a platform for reporting on the micro-grant project activities, to facilitate the monitoring of and reporting on activities funded through the micro-grants.
– Test the working hypotheses related to the definition of the risk of child-family separation by using data on a larger number of children than the one covered in the demonstration project
– Strengthen cooperation with the Ministry of Labour and Social Justice/the National Authority for the Protection of Child Rights and Adoption and pilot the model in various formulas for comparison purposes, while considering different intervention options to address several social assistance and child care system gaps.
The importance of a systematic use of the AURORA methodology. The model underlined the importance and value of identifying vulnerabilities, assessing the situation of vulnerable children and their families and monitoring this situation. Interviewed community workers and county supervisors unanimously agreed that it was necessary to continue using Aurora, the methodology that enabled the needs assessment as well as the generation of service packages perfectly adapted to the identified vulnerabilities.
Need for early and long-term interventions and for ensuring available specialised services for most serious vulnerabilities. Identifying cases of child violence, abuse and neglect was one of the biggest model challenges. For community workers, identifying cases of child abuse and neglect was a gradual process whereby they first gained the trust of the beneficiaries who gradually started to reveal the problems they were having and began to grasp the serious nature of certain instances of abuse or neglect which some of the communities treated as “normal”. By organising various campaigns against violence as well as counselling sessions, the modelling project helped community workers, key community stakeholders and project beneficiaries increase their knowledge and understanding of the phenomenon. Also, the model registered a limited impact on the adolescents’ level of information about risk behaviours and on reducing the incidence of these behaviours. In all these cases, lessons learned indicate the following: (1) the importance of carrying out prevention interventions and information and counselling services before the vulnerabilities become serious and the risks high; (2) the importance of carrying out long-term interventions, as reducing vulnerabilities involves changing target group risk behaviours; (3) the need to be able to access specialised counselling or recovery services which are often unavailable to disadvantaged rural communities.
Full report in PDF
PDF files require Acrobat Reader.