2013 Serbia: Final Evaluation Report "Developing community based services for children with disabilities and their families" (April 2010–December 2013)
Author: Camelia Gheorghe and Ozren Runic
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labeled as ‘Part 3’ of the report."
Serbia has made significant progress on a wide-ranging reform agenda, including progress towards achieving the national Millennium Development Goals, and modernisation of the legal framework. The opening of negotiations for EU membership will trigger more intense reforms, especially in public administration and justice. Meanwhile, the country is facing serious economic difficulties. The vulnerable groups of population have been disproportionately hit by the current economic crisis. Among them are families with children, especially those living in the southern Serbia, in rural areas and in the municipalities considered “devastated”. Children constitute 27.6% (160,586) of the total number of beneficiaries of CSWs. The number of children and youth with disability who are beneficiaries of the social welfare system increased from 10,718 in 2006 to 15,573 in 2011 as did the number of other social welfare beneficiaries. This increase put a big pressure on the social welfare system generally and on the CSWs in particular.
Of all children placed in formal care in the last five years (from 2006 to 2011), around 33% were children with disabilities, pointing to the lack of mechanisms and programs to ensure adequate and timely support for families to help them cope at times of crisis and prevent unnecessary family separation.
Data from 2008 indicate that only 33 municipalities (out of 168) financed day-care services for children with disabilities for a number of 1,122 children were included, while home-help services were available to only 125 children with disabilities and their families. Since 2008, the number of services kept increasing, reaching more and more children, young persons and their families. Despite this progress, the number of children currently benefitting from community services is relatively small, while the CBSS for at-risk families with young children are under-developed. The delivery of social benefits and social services varies widely across Serbia.
The purpose of this evaluation is to evaluate the final (end) results and achievements of the Project in relation to the project logframe and Theory of Change. The specific objectives of the evaluation, as per ToR, are the following:
1. Provide feedback to UNICEF Serbia office and its national counterparts on the soundness (defined as relevance, effectiveness, efficiency, sustainability) and impact of the Project approach in developing responsive community services for children with disability in order to reveal good practices and gaps in approaches, and evaluate Project impact following Project Plan, Project Logframe and Description of the Action.
2. Extract general lessons learned and recommendations aimed at further enhancement of the child care system reform.
3. Provide the Delegation of European Union to Serbia with information on impact of their specific support to Child Care System in Serbia
The evaluation covers the entire implementation period (22 April 2010 – 22 December 2013) at both national and local levels and all project components, following the way how the project has been conceptualized in the Social Inclusion IPA Fiche 2008, Description of Action, its Logframe and Theory of Change. The evaluation has been carried out during by a team of experts from Promeso Consulting (Romania), selected following competitive tender.
The evaluation has been carried out in three consecutive phases: Inception/Desk Phase, Field Phase and Synthesis Phase.
In the Inception/Desk Phase, the team reviewed the entire IPA 2008 project documentation and the national policy papers, legislation, strategies and action plans in the area of child protection and social welfare. The desk phase also included extensive review of EU accession-related documents and reports as well as the documentation on reporting to international bodies and human rights conventions ratified by Serbia.
The Field Phase has been devoted to the collection of data from key stakeholders at national and local levels, based on the evaluation instruments developed during the Inception/Desk Phase. The semi-structured interviews with UNICEF management and representatives of all national project partners took place in Belgrade between 16 and 21 September 2013. For the site visits, which took place in the following week, the evaluation team has developed a set of criteria for representative sample selection based on typology of services and service providers, rural/urban regional balance, territorial distribution and development level of municipalities. Out of 10 clusters, 41 municipalities and 52 services the team has visited 2 clusters (Čoka and Aleksinac) and 6 municipalities and services, conducting two discussion groups with local implementation teams, LSGs representatives, service providers and professionals as well as two focus groups with beneficiary parents and families.
In the Synthesis phase, the team has applied the standard evaluation criteria analysis (relevance, efficiency, effectiveness, impact and sustainability) in combination with Human Rights-Based Approach in order to assess the achievement of planned results, draw informed conclusions and lessons as well as to provide meaningful and practical recommendations.
Findings and Conclusions:
The Project is highly relevant for Serbia’s child care reforms and national policies for improving the lives of children with disability. It is in line with Serbia’s Social Welfare Strategy and the Plan of Action for Children while its primary objectives are tied to the implementation of the Social Welfare Law. It is highly relevant for Serbia’s international commitments deriving from the ratification of human rights standards and its status of EU candidate country. The Project addressed the most pressing needs of children with disability and their families as they were identified in the domestic and international. The Project remained relevant over its entire lifetime. UNICEF is seen as best positioned to advocate and promote the children’s rights and get involved in the implementation of such projects.
The Project was effective in achieving its planned results and objectives. UNICEF’s partnership with MoLESP allowed for effective and coordinated development of complex legislation which were drafted and submitted for adoption according to project plans.
Most of the planned Project activities have been delivered in an efficient and timely manner. The project implementation team produced high quality progress reports which were delivered within the set deadlines.
The project had a general high impact level. It made a major contribution to increasing the number of children with disability benefiting from CBSS. The mapping exercise which identified a significant number of children with disability in the targeted municipalities kept hidden and unregistered had a significant impact on the awareness of possibly large number of children in Serbia in similar situation who are in need of identification and outreach.
Most effects and outcomes of the Project are likely sustainable.
The Project had a major contribution to the promotion and realisation of child rights.
The Report provides a number of key recommendations for the Serbian Government and LSGs, UNICEF, EU and other donors, based on the findings and conclusions of the evaluation, as follows:
Strategic Recommendations (S):
S1: Further invest in the development and diversification of country-wide community-based social services to contribute to the social inclusion of the most vulnerable children with disability and prevent separation from their families
S2: Develop a culture of lifelong learning within Centres for Social Work to ensure that managers and staff keep pace with reforms and are empowered to address the needs of children with disability at highest professional standards
S3: Support the formation of a competitive market of service providers to ensure best quality of care at affordable prices for children with disabilities and their families
S4: Strengthen the quality assurance system of the community-based services for children with disability
S5: Improve data gathering systems for monitoring of child care reform and informed policy-making for the benefit of children with disability
S6: Ensure that a more systematic Human Rights-Based Approach is used in the programming and implementation of future UNICEF projects targeting children and their families
Operational Recommendations (O):
O1: Carry out a country-wide identification of children with disabilities for evidence-based policy making and support action
O2: Adjust the approach to respite care to ensure its sustainability
O3: Develop a costing model for various community-based services addressing children with disability to guide local budgeting of services and benchmarking generally
O4: Include clear accountability criteria in future grant agreements with municipalities
The evaluation of the Project led to the following Lessons learnt:
1. Legal (official) commitments of beneficiary institutions at national and local levels are key to successful implementation of initiatives related to social welfare reform.
2. Concept of municipality clustering is very good for efficiency but less appropriate for effectiveness and sustainability.
3. Implementation of local projects in the (so-called) city municipalities has to be carefully planned.
4. Investing in academic and professional research as a back up to main activities brings significant returns.
5. Child care reforms cannot succeed without a strong CSW network.
6. Community-Based Services represent a significant breakthrough for child care, but most vulnerable and neglected children with disabilities could still be in danger.
Full report in PDF
PDF files require Acrobat Reader.