Evaluation database

Evaluation report

2009 Uzbekistan: Summative Evaluation of the Family Education Project for the period January 2005-July 2009



Author: Professor Carol Aubrey and Chris Gittins

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Executive Summary

The Family Education Project (FEP) with its vision of Integrated Early Childhood Development (IECD) was initiated in May, 2003 by the Government of Uzbekistan with technical support from UNICEF Uzbekistan with the broad aim of “family empowerment and increasing the families‟ knowledge of child-rearing practices through community volunteers‟ particularly in the age range of 0-6 years. A high Maternal Mortality Ratio, Infant Mortality Rate, anaemia, Iodine Deficiency Disorders, almost no early learning interventions for under-threes and low pre-school attendance at around 20% had indicated the need for the FEP. UNICEF‟s role was to assist the Government of Uzbekistan through providing technical expertise on Integrated Early Childhood Development, training of trainers and volunteers, development of materials and advocacy for Early Childhood Development at different levels. The FEP resonated with UNICEF‟s commitment to United Nations Convention on the Rights of the Child, the Convention on the Elimination of All Forms of Discrimination Against Women and the Human Rights Advocacy Programme because it places the needs of children at the centre of the programme, empowering families and communities, using a rights-based approach.
FEP was piloted in 13 makhallas (communities) of 6 districts (rayons) in 3 regions - a) Ferghana Region: Kuva District (Rasta and Tashkent makhallas), Uchkuprik District (Yangabad, Gul, Bekmurod makhallas; b) Tashkent Region: Zangiata District (Nazarbek, Katartal makhallas), Yangiyul District (Gulbakhor, Eski Kovunchi makhallas); and c) Republic of Karakalpakstan: Ellikala District (Abai, Ibn-Sino makhallas), Nukus District (Akmagyt, Arbashi makhallas). Volunteers from the community (such as doctors, nurses, teachers and kindergarten teachers, retired professionals, makhalla advisers and others) were trained to implement FEP. After the mid-term review (MTR) the FEP was scaled up to 28 districts of 6 regions to adopt improved child-rearing practices.
2009 is the last year of UNICEF‟s current (2005-2009) programme cycle in Uzbekistan. It is therefore crucial to have reliable, accurate and comprehensive data on the FEP impact, achievements and constraints, and to generate relevant recommendations for further project directions for the next programme cycle (2010-2015). The quantitative impact assessment of the FEP took account of the FEP original piloting in 13 makhallas of 6 districts in 3 regions and sites representing rayons with different levels of intervention. Ferghana and Kashkadarya provinces were selected, ensuring that rayons were involved in the Baseline Assessment (KAP survey), therefore comparison and monitoring of the interim between Baseline and Impact Assessment could be made and a city-village comparison included. The qualitative evaluation that is the subject of this report, focusses on the regions of Bukhara and Ferghana. Sampling size was recommended to be at least 10 families/households, 5 deputy governors, 5 community advisers. It could best be described as a “maximum variation” sample that allowed for the meanings and perspectives of individuals and small groups to be captured through observation, interview and focus groups and which complemented the three-stage stratified cluster sampling of the impact assessment. A qualitative evaluation framework was developed addressing key result areas related to inputs, processes, outcomes and policy against project objectives to identify programme components that worked and those that did not.
FEP implementation used existing governance structures. At national level, the Pediatrics Institute took overall responsibility for FEP training, monitoring, management, as well as co-ordination with the Women‟s Committee (who appoint the makhalla advisers), Ministry of Health, Ministry of Education, local hokimyats and UNICEF. At rayon level the deputy governors and inter-sectoral committees took the lead. At makhalla level, the makhalla committees and makhalla advisers implemented the programme with selected families.
In 2008, district governors‟ offices reported that the coverage had reached 98% of vulnerable families with children 0-6 years. Overall it is estimated that the FEP reached 28 districts of 6 regions. This translates nationwide as 16% of the total population (4,289,641 out of 27,000,000); 14% of the total number of districts (28 out of 1999 districts/cities/towns); 15% of communities (1,417 out of 10,000); 15% of community advisers (1,223 out of 8,000); and for the 5 years of the programme, more than 507,589 families have received face-to-face contact (approximately 10% yearly of all low-income families, that is, out of a total of approximately 5,500,000 families of which about 20% are low income, 1,100,000 families have been reached).
Key findings related inputs and processes of implementation. These suggested that everyone whom we met recognised the distinctive cross-cutting nature of the FEP content and the strategy of using existing government structures, yet welcomed its local and contextualised approach. Families clearly understood the early childhood development (ECD) concept and were able to identify and discuss key messages.
Capacity building through the training of frontline workers and volunteers led to development of a training manual and family home activity booklets. Some bias towards health messages was noted in a number of documentary sources, especially where key messages were obtained from the United Nation‟s Facts for Life (FFL) materials rather than the FEP training manual. FEP materials were revised in 2006 on the basis of baseline data and the existing knowledge gaps of caregivers.
“Scaling up” from pilots to 28 pilot district of 6 regions created a challenge. It was useful in raising awareness of all deputy governors in the 6 regions, yet added an additional level of information cascade and reduced trainers‟ direct interaction with community advisers. This increase in geographical coverage was a potential risk to quality. Quality of training by district deputy governors varied and so too did the experience of trainee volunteers. Five-day training modules became fitted into four days. This has been counteracted to some extent by the extension of “refresher” training and renewed emphasis on interactive methods. Volunteers then used many different entry points to reach families with FEP messages from home visits, to community meetings and events, and service-linked contact through clinic and kindergarten. Monitoring of training activities of volunteers helped to improve information and hence the situation, whilst training took place to support monitoring and evaluation activities.
The content of FEP materials, are highly regarded the materials are in high demand, yet volunteers in one region and the participants in the FEP National Validation Workshop reported that supplies are very limited. Additional FEP publications and related materials have also been produced to support capacity building and social mobilization events. Family Learning Resource Centres (FRCs) appear to have been a useful resource for volunteers as a further source for materials.
Over the period of 2005-2009 there have been many observations that the Early Childhood Development component of the programme needed strengthening, with more attention to age-appropriate outcomes and related activities to stimulate development. By the time of the summative assessment visit, Early Leaning and Development Standards (ELDS) for Uzbekistan were being validated with a view to the preparation of booklets for parents/caregivers and a training module for volunteers.
A range of FEP strategies beyond home visits and one-to-one contacts have been used to popularise and deliver FEP messages through programme communication for behaviour change, though the effectiveness of social mobilisation events and programme communication was less straight-forward to judge. Overall, national entry-points through mass media were judged from UNICEF Annual Reports to have been least successful.
In terms of outputs and outcomes, changes in families were established from the FEP impact assessment that took place in April to June, 2009. Conclusions from quantitative survey data were that the FEP had been very successful in raising awareness and knowledge about safe motherhood, public health services and child health, practice of child nutrition, sanitation and hygiene, psychological and social development of the child. It is recommended that the FEP should continue strengthening these messages, in particular, messages. Pre-school development is regarded as the most neglected area and enrolment to kindergarten in the priority areas remained low at 16,5% of all children who are eligible to attend. This finding is confirmed by all stakeholder groups in the qualitative evaluation with Early Learning and Development Standards seen as providing the basis for development work with trainers, volunteers and parents/caregivers.
Changes in communities through social mobilisation was also reported by stakeholders during consultants‟ visits to the field and changes in use of services observed through a visit to a community health centre and kindergarten where staff were offering an experimental “Saturday School” as a three-month preparation for primary school for children of low-income families.
Overall, UNICEF‟s role with respect to the management of FEP alongside other thematic areas has focused on the policy level, support to service delivery, partnership working and consideration of the extent to which the project corresponds to priority problems of children and women in the country and is strategically well defined with clearly stated expected results. In relation to the total UNICEF Uzkbekistan Country Office budget for 2005-2009, the FEP has been implemented on relatively modest funds that amount to 6,5% of the total budget, with cost per family of approximately $2,5 and per child of $1,2. Monitoring and evaluation of the FEP has provided a means of revealing the status of project progress and informed decisions on future directions and partnerships in the furtherance of Millennium Development Goals targets and principles of the United Nations Convention on the Rights of the Child and the Convention on the Elimination of All Forms of Discrimination Against Women.
In interviews with key partners to review collaboration with UNICEF, there was unanimous support for continuation and development of the project with a focus on Early Learning and Development Standards and innovative models of developmental preparation for primary school. However, whilst acknowledging the need for developments in the preschool sector, donors such as the World Bank are prioritising infrastructure and teaching methodology in basic education, whilst the World Health Organisation, United Nations Population Fund and United Nations Development Programme in principle positive towards the idea of collaboration with UNICEF, saw their priorities lying with reproductive health and primary health care. At the same time, development of Early Learning and Development Standards, curriculum and materials development and training had the goodwill and support of the Ministry of Health and Ministry of Public Education.
Several recommendations have emerged from the summative evaluation.
Recommendation 1 is that the FEP continues through 2010-2015. into a further phase, building on the initial successes. It is an example of a UNICEF programme that has moved from pilot phase to development phase with the support of a range of strategies involving production of materials, training, technical assistance, advocacy and community mobilisation.
Expanded access to the programme could be achieved in two stages. Initially, with a target in stage 1, to reach all the 27% of low-income families nationwide, expanding province by province. After a further evaluation and review of the programme, stage 2 could support each province in delivering the programme to all new families annually as part of antenatal care.
Recommendation 2 is that given the current stage of programme development, a high priority should now be the development of materials and training, using appropriate training methodology, in Early Childhood Development. Both the FEP Training Manual and the Home Activity Booklet should be reviewed in order to achieve a more differentiated approach and a better match between observation, assessment and ECD activities at particular ages and developmental stages for children 0-3 years and 3-6 years. In addition, through communication, advocacy and social mobilization, the programme should target change in the general level of school preparedness or “developmental readiness‟ over the next five years.
Recommendation 3 is that given current resource constraints and continuing high demand for paper resources, it is recommended that more use is made of information and communication technology (ICT) for electronic storage and dissemination of paper FEP materials and that more responsibility is taken for costs of download of these on a local basis. It is recognised that the development of ICT may take time but making use of existing connections, starting with regional centres and expanding as opportunity arises, should be encouraged.
Recommendation 4 is that quality training is continued with national trainers focusing on training trainers at the regional level and providing initial support for cascade training by regional trainers. If further national roll out is to be achieved this will require the development of a fully documented and resourced training for trainers programme with accompanying trainers handbook applying appropriate training methodology.
It is also recommended that all training includes training in cost effective and efficient quality assurance procedures for periodic monitoring of rayon/makhalla training, implementation and behavior change in families (see recommendation 10).
Recommendation 5 is that when the content of the materials is reviewed, the FEP team re-evaluate the key messages one by one in order to distinguish them by likely effect on behaviour change. Simple health messages, for instance that require straight-forward information exchange can then be distinguished from more complex learning to support and facilitate the development of a young child‟s cognitive skill.
Recommendation 6 is that although a variety of training methods has been introduced by experienced national and international trainers in order to make training more interactive, it is recommended that training at all levels is reviewed in order to expand the repertoire of teaching/learning methods and to ensure that training is grounded in adult learning principles.
It is further recommended that training adopts a typology or classification of educational outcomes that will assist participants in distinguishing between learner reaction; modification of attitudes/perceptions; acquisition of knowledge/skills; behavioural change that is identified by transfer of learning to the home and improvements in health or well-being. The aim should be the creation of self reliant and confident parents/caregivers equipped for lifelong learning and ongoing behaviour improvement in the family
Recommendation 7 is that the FEP continues to use a range of entry points (household, service-linked, community-focused and systems-oriented) and a range of strategies (capacity development, advocacy and communication and social mobilisation) whilst recognising that the use of mass communication has been the least successful and could be reconceived under the direction of a high-level national FEP committee (see recommendation 13).
Recommendation 8 is that revised Early Childhood Development material based on Early Learning Development Standards place emphasis on the rights of young children with disabilities to an enabling environment and inclusion in play, recreation and basic pre-school education activities
Recommendation 9 is that a range of different options is promoted with government at all levels for Early Childhood Development/developmental preparation for schooling that includes “Saturday and Sunday Schools”, “Mothers‟ classes”, classes in schools/makhalla meeting houses and involves promotion of family, community and local government mobilisation.  It is also recommended that the FEP project should be utilised to promote greater partnerships between families, volunteers and pre-school teachers who will all need training in inclusive, child-friendly and interactive pedagogies that will empower parents/caregivers and develop skills for lifelong learning.
Recommendation 10 is that when the FEP materials and training are reviewed and particular aspects are developed, the monitoring and evaluation team review whether or not the monitoring scheme can continue in its current form. A decision will need to be made about collecting data on all messages or whether distinctions can be made between participant responses that can be quickly collected from periodic survey and more complex behaviour change in volunteers, families and children that will require more careful observational methods.
Recommendation 11 is that the FEP team consider the development of community-based and self-help models of developmental preparation for schooling since there is an existing infrastructure to be exploited at national, regional and local level and many trained, experienced and enthusiastic people still working in the pre-school sector who would support training and curriculum development, advice through a system of “methodologists‟ and regional training centres.
Recommendation 12 is that current functions and training activities of FRCs (centres and corners) and makhalla centres are reviewed with a view to developing a clear set of ICT strategies and outcomes, with a strategic plan to provide a networked, flexible system offering accessible, relevant, high-quality learning opportunities. An identified set of focus areas that includes developing professional competencies and capacity building to support ICT resources, curriculum materials development (paper and electronic-based) that can be used to support and enhance learning as well as produce materials.
Recommendation 13 is that UNICEF, working with the Ministry of Public Education and the Women‟s Committee, focus on establishing effective leadership and management of the programme at all levels. Committees and working parities with influential chairpersons and including all major stakeholders should oversee the programme and help ensure greater collaboration at national, regional and local level, in addition to promoting the programme and monitoring its impact
The FEP progress over the period 2004-2009 generates a number of lessons that may be used for future programming in the CIS region. Amongst those listed in section 6.6 are:
a) Design of FEP is most successful because it addresses a cross-cutting concern within a policy framework.This is done in a manner that is participatory and reflective in its planning, implementation, monitoring and evaluation. Hence, it is strategically well defined with clearly stated expected results to improve service delivery and improve knowledge and practice in appropriate child-rearing practices.
b) Materials have been developed locally with high quality control. Printed materials have been a cost effective way to reach large audiences though audio-visual digitally produced material could support training in the next stage and serve as a strategy to reach a mass audience.
c) Cascade Training has been inevitable and maintaining high quality participative interactive processes has been a challenge. Refresher training that is cost effective and easily deliverable has been available at local level and has been much appreciated.
d) Monitoring and evaluation has been regular with bi-annual data gathering that has been continuously assessed against baseline data and previous monitoring results. For the next phase, the monitoring may need fine-tuning in order to combine more rapid survey checks across „messages‟ with more in-depth checks of behaviour change in families and the skills of children.
e) Community empowerment through the use of community volunteers is a powerful rights-based model but requires careful management as the FEP relies ultimately on volunteers to transmit Integrated Early Childhood Development messages to the families. Loss of trained volunteers is inevitable in such a programme and the continuous need for capacity building and the generation/retention of a large pool of available volunteers must be recognised.
f) Strategy for FEP. Out of four recommended entry points to the programme, three are working effectively, (face-to-face family visits; health clinic contacts,community development through social mobilisation activities) The exception is raising awareness through mass media.
g) Coverage objectives have been achieved in priority regions. They need to remain under review in the light of risks related to geographical coverage/scaling up strategies versus quality.
h) Sustainability based on quantitative data of the evaluation is believed to be reasonably assured with the FEP contributing to national priorities. Deputy governors are directly responsible and accountable for the reduction of Infant, Under 5 and Mother Mortality Rates in their respective districts.. The community advisers and volunteers are empowered and consider their job as part of their duties and responsibilities as well as their civil duty to assist families to better understand the needs and rights of children.
i) Participatory and reflective evaluation methodology through the regular monitoring and evaluation carried out, involves stakeholders in analysis of local findings and planning and design for future programming.
j) Rights of children. FEP has already made a contribution in its advocacy for Early Childhood Development at different levels. A further contribution can be made in the next phase of the programme in drawing attention to the rights of young children with disabilities to support of families and inclusion in play and pre-school education activities



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search