2011 Romania: Evaluation of Helping the invisible children Project (CBS)
Author: Manuela Stanculescu, Monica Marin
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
The project is part of UNICEF's Community Based Services (CBS) Programme in Romania. The CBS focus is on the preventive approach in social protection system, especially in rural areas. The purpose of the project ‘Helping the invisible children’ is to increase the impact of social protection policies for poor and socially excluded (‘invisible’) children and families. It has employed external or internal social workers in 96 communes from eight counties having multiple responsibilities, including community census (to identify cases of vulnerable children) and, correspondingly, identify and put into practice solutions to their problems with the help of the Community Consultative Structure . At county level the project has relevant partners in the area of services and policy.
The purpose of this evaluation is to collect evidence related to both project effectiveness and efficiency, contributing to the understanding of its relevance and impact. In this respect, there are two main objectives to be attained:
- Measure the project effectiveness and efficiency as well as estimate its potential impact of the project activities, on the one hand, on the local stakeholders (identified vulnerable children) and on the other hand, on the services provided at local level, all based on an analysis taking into account the local specificities determined by geographical, demographic, social and economic criteria.
- Identify and extract relevant elements necessary for the policy development in the prevention area (information related to legislation, institutional building, monitoring and evaluation mechanisms) with a particular focus on the definition of a minimum package of basic social services.
The evaluation framework follows the OECD/DAC criteria on relevance, effectiveness, efficiency, impact, sustainability, connectedness and coherence.
The evaluation used a mixed-method approach, combining quantitative and qualitative research tools. Some of the information has been collected during project implementation (questionnaires completed by the social workers, supervisors’ reports), while most of the information has been produced with the special purpose of this evaluation (synthetic fiche, interviews, focus-groups, case-studies and opinion survey). The evaluation covered all key stakeholders, all counties and all communities. The coverage is exhaustive for the instruments related to database aggregation and analysis (census questionnaires, supervisors’ reports and Synthetic Fiche) and interviews with county stakeholders. When sampling methods were used, they took into account geographical and case-diversity criteria (focus-groups and case-studies) and opinion survey (representative sample based on typology derived from community development and cases of vulnerable children).
Findings and Conclusions:
The project is considered as highly relevant, effective and efficient both according to the stakeholders’ opinions as well as according to the performance indicators. Both sustainability and connectedness should be of special concern in the next phase of the project.
The project is considered very effective by most stakeholders. Due to the short period of implementation, it is difficult to estimate the extent to which the activities carried out by social workers and CCS have attained the final goal. As a solution, we assess the attainment of the project intermediary objectives, which refer to the identification at the local level of (1) ‘invisible’ children and (2) solutions for their support. Concerning the objective on identification of ‘invisible’ children, most stakeholders agree that it has been attained, while on the solutions identified for their support, substantial progress is still needed. This should be the priority of the next phase of the project.
In both economic terms and stakeholders’ opinions, the project is very efficient. The project is perceived ‘like a breath of fresh air, dealing with real problems of real people’. At the same time, the current cost for one child in the protection system is almost three times larger than the prevention cost as resulted from the project.
The project’s problematic areas are sustainability and connectedness. Most communes, particularly the poor (underdeveloped) ones, are characterized by high needs (high number of identified ‘invisible’ children) and low or very low local budgets. The budgetary and staff sustainability need to be addressed. The potential sustainability is significantly higher for communes with project social workers already employed by mayoralty compared to those with external social workers. Also, the likelihood that the Community Consultative Structures (CCSs) will continue to be functional after the project end is highly dependent on the existence of the social worker involved in the project.
The achievement of the project’s objectives and final goal strongly depends on how will be tackled the lack or poor expertise at the local level, the low level of awareness and knowledge on social problems, the generalized tolerant attitudes towards alcohol abuse, violence or school dropout of both population and officials, as well as the widespread poverty.
Activities of dissemination, exchange of good practices and training represent a mechanism for fostering the volunteer spirit, but also for enhancing the capacity of those in contact with the child.
Training activities should address the potential bottlenecks for the second year of project, namely the poor knowledge of social workers employed in the project on specific topics.
Stimulate community ownership needs to be fuelled.
Attitudes, traditions, customs, behaviour and practices: Ideas and solutions to children’s well-being are culturally constructed and, hence, are influenced by traditional beliefs and practices. In the project’s case, tolerance to violence or alcohol abuse, constitute major factors that influence the achievement of the project goal on the long term. Addressing them is a long-term intervention, which however need consideration.
Financial sustainability: Another strong point of the current project is that it has low running costs. Even so, with a low budget at local level, the financial sustainability of the project is at stake, particularly considering that the new legal framework on social assistance announces further budgetary cutbacks in the sector.
Staff sustainability: The main stakeholders agree on the fact that UNICEF’s main focus for the future is to lobby and negotiate with the Ministry of Labour, Family and Social Protection a solution to this problem.
Services sustainability: Preserving the developed services and practices (outreaching and prevention) is crucial for the project. The organizational structure of the SPAS needs to be modified by creating a post of social worker dedicated to these activities. This is difficult but also even more important in the context of the ‘austerity’ reforms of the social sectors.
Monitoring and oversight: The current project monitoring systems have been partially useful for ensuring oversight mechanisms. They should however be updated with the findings from the present evaluation: (i) change the reporting format based more on impact indicators; (ii) request periodic updates on the database of individual cases of vulnerable children; (iii) create a panel of ‘invisible’ children and monitor also the impact of the newly enacted provisions of the Law on Social Assistance in order to make sure their vulnerability does not increase.
The main lesson learned from the project is that at present in Romania, the ‘invisible’ children are a fact. In only 96 communes, more than 3,000 ‘invisible’ children were identified. How many others are expecting to be reached and helped in the other 2,765 communes of the country? The decision-makers at the local and national levels should be made aware of this fact … the ‘invisible’ children need help.
Full report in PDF
PDF files require Acrobat Reader.