We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2008 CEE/CIS: Regional Thematic Evaluation Of UNICEF Contribution To Child Care System Reform In Central Asia



Author: Oxford Policy Management

Executive summary

Background

UNICEF's regional office for Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS) has contracted Oxford Policy Management (OPM) to conduct an independent evaluation of the contribution UNICEF has made to child care reform in five Central Asian countries—Kazakhstan, the Kyrgyz Republic, Tajikistan, Turkmenistan and Uzbekistan—between October 2003 and July 2007.
Almost all of the child care reform activities in this period have been carried out under the project 'Every child has a right to grow up in a family environment' funded by the Human Security Trust Fund (HSTF), for which the proposal was submitted in October 2003 and which formally came to an end in July 2007. The activities under that project are the central focus of attention, as agreed in the terms of reference. The only other major source of external funding has been the provision of $670,000 by SIDA for the project 'Promoting the deinstitutionalisation process in the Republic of Tajikistan' which took place in Tajikistan from 2004–06. Activities from that source are also included in the evaluation although the SIDA project has already undergone its own separate evaluation so full details are not reproduced in this report. Some activities funded by UNICEF from its own resources are also evaluated where appropriate. A lighter evaluation was conducted in Turkmenistan than the other four countries, reflecting UNICEF's less extensive engagement there.

Purpose
According to the terms of reference the evaluation serves three purposes:
•to fulfil accountability towards the donors, beneficiaries and other stakeholders by providing
information on how funds have been spent and to what extent objectives have been met;
•to increase learning on how best to support future child care reform. This includes the provision
of advice on where best UNICEF should direct its resources in order to contribute most
effectively to the reform process; and
•to inform a regional strategy on child protection in the different countries.
The end of the HSTF project is one of the factors that has determined the timing of this evaluation,
the other being the mid-term review of many of the country offices.

The evaluation has three objectives:
1: Provide the current UNICEF-donor and the beneficiary of the support (Governments of the different countries) with an evaluation of impact, relevance, effectiveness, efficiency and sustainability of the different activities in the project “Every Child has the Right to Grow up in a Family Environment”. Human rights and results based management.

2: Assessed against the progress made so far in the reform of child care system (the set up and changes in the system), and the social-, economic and political context of the country, provide UNICEF country offices with feedback on [the areas of reform so far, significant achievements and areas requiring further development].
3: Inform global and regional strategies based on the experience in the five countries.' (UNICEF terms of reference, section 5).

Methodology
The evaluation was carried out between August 2007 and January 2008, with primary data collection taking place on a 7–10 day visit to each country between September and November 2007. In each country the team collected and analysed secondary data and also carried out primary fieldwork. The fieldwork consisted of interviews, an analysis of strengths, weaknesses, opportunities and threats (SWOT), focus groups and direct observation of sites. Respondents include central and local government officials, development partners, non-governmental organisations (NGOs)—including both UNICEF partners and others active in the area of child protection—and staff of pilot and non-pilot institutions, as well as parents or guardians of beneficiaries, and the children themselves.
Meeting the objectives of the evaluation requires first an assessment of the status of child care in each country, and second, an understanding of the way in which a policy cycle shapes the outcomes that are achieved and their impact on the beneficiary. The study develops a diagnostic framework which can be used as a tool for identifying more precisely the components of a comprehensive policy, and understanding where policy can be improved to better realise child rights. This is shown in section 3. In summary, the policy cycle is represented as a four-stage process of problem identification, policy development, policy implementation, and monitoring and review. This cycle, which is relevant to all policy issues, acts on the specific context of child care policy, which in turn is seen in central Asia to have four components, namely family support services, family substitute services, residential care facilities and overarching governance structures. This divides the whole sphere of child care policy into 16 segments (for example, the development of policy on family support services), using which it is possible to articulate more precisely where UNICEF has intervened and where it is most effective.


Findings and Conclusions

Activities are generally found to be relevant in the light of the governments' commitment to the implementation of the UN Convention on the Rights of the Child (CRC) and their stated positive attitudes to deinstitutionalisation. However, interventions are not always complete: for example, implementation activities are sometimes undertaken without the necessary advocacy for policy development that will ensure their long-term sustainability. Conversely, relevant policy development activities may be undertaken but without following through to ensure that they are implemented. There has been a greater emphasis on family support services than on family substitute services.
While child care reform is relevant to the government's own policies in all countries, there is more practical evidence of this support in Tajikistan than in other countries; in Turkmenistan the window of opportunity for discussion may now be beginning to open, while the remaining three countries are in an intermediate stage. The HSTF programme is consistent with, and therefore relevant to, the operationalisation of UNICEF's global programme in child care reform, and works to support some of the recommendations of the UN Committee on the Rights of the Child in countries where reports exist; but, again, it has not been comprehensive in addressing all priority directions of those reports. Attempts to improve public awareness and achieve better coordination of child care policy have been evident, but support to an understanding of the budgetary requirements and implications of the policy reform, and to monitoring and evaluation, has been largely absent though these are important areas to address.


The situation regarding the relevance of UNICEF's intervention to other partners is split between those where there are few other partners (Turkmenistan and Uzbekistan) and those where it has greater opportunity to engage with development partners and NGOs. In Kazakhstan UNICEF could potentially come to an arrangement of closer complementary activities with NGOs who have expertise in policy implementation, which would free up UNICEF to focus on policy development. In the Kyrgyz Republic and Tajikistan the main consideration is the EU, with its programme of budget support in the social sector. Thus far it has had greater success in the former than the latter.


Effectiveness was considered in relation to the four components of child care policy listed above. In terms of reducing the reliance on residential care, UNICEF appears to have had best success in Tajikistan (though a shortage of reliable data in all countries makes it hard to be certain). In Kazakhstan the number of children in institutions has remained steady while in the Kyrgyz Republic they have increased. Figures are not available in Turkmenistan and Uzbekistan. Factors that have contributed to successful deinstitutionalisation are the attention to the whole policy cycle (including monitoring of children who leave residential care); the availability and awareness of alternative services; public support for deinstitutionalisation; the presence of champions of the reform; and a favourable external environment.
The development of family substitute services is found to be limited, and to have a correspondingly limited effect. Part of the difficulty has been a peculiar sequencing of interventions in foster care by which policy implementation was carried out before there was agreement that the policy should be developed or was even workable. It is of concern that in Kazakhstan some children are being removed from foster care and placed back into institutions since this is the opposite direction to the intended policy. Interventions in adoption have not been widespread. Family support services have been effective to some extent in all five countries. Tajikistan has some innovative examples such as the diversion projects that work with young offenders, and reformed consultation and education services for children with disabilities. In the Kyrgyz Republic, Tajikistan and Uzbekistan there has been success in developing social work as a university-level subject and as a profession; success is also seen in Kazakhstan, where social work was already better established. In all countries except Turkmenistan UNICEF has achieved some important results in supporting the reform of legal and administrative frameworks for child care reform, particularly in drafting regulations and piloting local government structures.


In evaluating the efficiency of the HSTF project the team came to the conclusion that resources could not have been distributed strategically in a manner that might be expected to achieve the best results, since the distribution of the $2.1 million budget for the HSTF project among the five countries appears to have been determined loosely on the basis of their relative population size rather than on the prospects for successful reform or any analysis of the financial capacity of the governments. Country offices had some effect in re-targeting the funds they received to activities that were more of a priority for their own situation. Funding for advocacy was not sufficient to implement the planned activities. Activities were very heavily skewed towards policy implementation but it is not clear that this was always justified given that it is an area in which other actors are also able to contribute.


With regard to impact, it is known that UNICEF has succeeded in avoiding the institutionalisation of some children as a result of its activities in child care system reform since 2003. Exact figures are difficult to establish, but there is anecdotal evidence of these positive instances. In the absence of an effective monitoring system in any country it is difficult to quantify this impact. An important finding is that merely counting the number of children who have been released from institutions does not necessarily reveal accurately whether the impact of the project is successful: an assessment of the real impact of the reform should reflect on the improved well-being of the child. Deinstitutionalisation alone is not sufficient as a goal, since a focus on a quantitative reduction in the number of children in residential institutions risks overlooking the potential negative impact on some children if they are returned to their family environment; and a focus on the number returned to their families may overlook an increased number who have entered the institutions from elsewhere.


There is some indication in the HSTF proposal that the project was intended to be pro-poor, which would suggest that UNICEF might, if desired, choose to support the targeting of its own pilot projects at the poorest section of the population. Discussions with respondents indicate that UNICEF's interventions have not been strongly pro-poor although they have undoubtedly reached many poor families. Sustainability was measured from three points of view: financial sustainability, the development of institutional structures and progress in changing public attitudes. In some instances UNICEF has begun to address the issue of financial sustainability at the highest level, as in Kazakhstan where it has advocated changes in the budget process in the interests of children. In the Kyrgyz Republic UNICEF has successfully used the leverage of the European Unions (EU) budget support programme to strengthen the financial future of its project activities by working with the EU to achieve the inclusion of the planned nationwide rollout of Family and Child Support Departments (FCSDs) into the matrix of conditionality for budget support. However, the financial sustainability of UNICEF's initiatives is not yet fully assured. A frequent response by local and national government counterparts to the evaluation team about this issue was that since child care was important, 'money would be found from somewhere' to continue the activities as necessary. This optimism overlooks the genuinely resource-constrained environment within which most services are provided, especially in countries other than Kazakhstan. Governments in all countries have made progress in developing the administrative structures and legislation necessary to improve the opportunities for children to live in a family. A particular contribution by UNICEF to strengthening the institutional framework in the long run has been the support to university social work courses mentioned above. The evaluation team considers this to be a positive and far-sighted initiative since the expectation is that graduates of the courses will eventually fill positions in central and local government and as active social workers, where they will be advocates for and use the techniques promoted by UNICEF to support the right of the child to live in a family. However, the benefits may take several years to achieve their full impact, and there is a risk that trained social workers may not remain in the profession.


In all five countries there is evidence of a shift in attitudes of some representatives in key positions at central and local government levels and among professionals in favour of family-based rather than residential care. But there remain mixed attitudes among professionals towards deinstitutionalisation and the establishment of community-based services, not so much because of considerations about the welfare of the child—although that sometimes remains an issue—but rather because of the implication of the closure of institutions for the allocation of resources to ministries and the availability of jobs in remote areas. Among the general public some changes in attitude towards child care are apparent in areas where UNICEF is carrying out pilot activities, such as in the FCSD in Issyk-Ata, Kyrgyz Republic and the Parents Education Centre in Tajikistan. UNICEF should now consider how to spread this message further to areas where it does not directly intervene, and to areas where it has not so far achieved significant attitude change. According to a rights-based approach one would expect to find that UNICEF first identifies the rights of rights-holders and the obligations of duty-bearers and assesses their capacity to fulfil them. It should then support the fulfilment of these rights and obligations throughout the programme cycle by designing and implementing strategies to build these capacities, and monitoring and evaluating outcomes on the basis of human rights principles including the right of children to participate in decision-making. The design of the HSTF programme is certainly intended to promote a rights-based approach as it is in line with the CRC. In implementation, too, UNICEF has fairly consistently provided services which enable duty-bearers—such as parents and governments—to improve their ability to support children in a family-based environment. However, the issues of gender, language and ethnicity, which were absent from the project design, have also not been widely highlighted during implementation. In monitoring and review UNICEF has enabled children to participate in some reviews of child care policy. If an effective system of results-based management is in place one would expect to find both that UNICEF measures its own activities using a clear and comprehensive system of monitoring and makes adjustments to planning decisions on the basis of the results, and also that it supports the government in improving the availability and quality of its data. The lack of attention to this essential aspect of programming is a recurrent theme of the evaluation. Within UNICEF's own project the concept of managing on the basis of results has not been widely taken up. The objectives of the HSTF project do not conflict with UNICEF's national and international strategies but it is not clear to the evaluation team whether or how the broader results frameworks have been used during the design and implementation of the project, nor how it was intended that they should be used. The HSTF proposal's own brief set of indicators also have no targets and are ambiguous in definition. In any case, data are not available that would assist UNICEF in making an informed evaluation of the results it has achieved. The use of results-based management processes is underdeveloped at local and national level as it is within UNICEF. Three obstacles are identified: the lack of openness to the discussion of results in some countries; the tendency to base government budgets on previous inputs rather than on results, which reduces the incentive to identify the policy outcomes; and a lack of capacity in monitoring and evaluation.


Recommendations

It is recommended that the organisation identifies its comparative advantage in each country, taking into account the actions and strengths of governments and other development partners as well as its own, and focuses its attention on these areas while ensuring that there are no gaps in the cycle of governments' child care reform policies (such as policies that are developed without a full analysis and understanding of the nature of the problem, or that are implemented but not monitored).
Moreover it is recommended that UNICEF continues to pay attention to advocacy activities among the general public as well as among staff of residential institutions and policy-makers. UNICEF should be clear about what it is advocating (e.g. by making it explicit what type of institutions it wishes to reform, since, for example, some are boarding schools that provide education to children who live in remote areas, and the boundary between these and other types of residential facilities is not always distinct). It should be careful also to ensure that the purpose of the messages it conveys is understood so that advice is not misapplied, with detrimental effects on the well-being of the child (e.g. ensuring that a well intentioned but misapplied deinstitutionalisation policy does not result in children being reintegrated into their families and finding themselves at increased risk).



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2008

Region:
CEE/CIS

Country:
Multi-country

Type:
Evaluation

Theme:
Child Protection

Follow Up:

Language:
English

Sequence Number:
2008/001

New enhanced search