Evaluation database

Evaluation report

2010 Ukraine: Summative evaluation of the Ivano-Frankivsk oblast project to improve young child health and development



Author: Alexandru Gaina

Executive summary

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:
In accordance with Country Program Document (CPD), UNICEF and the Government of Ukraine, as part of their cooperation programme for 2006-2011, are contributing to the achievement of the following key results in the area of child health and development: a) the majority of children will be born in ‘baby-friendly’ facilities; b) most parents of young children will have increased knowledge of childcare and development.
To reach these goals, the Country Program Action Plan (CPAP) for this period envisages that mother and child health care policies and practices will be reformed in line with WHO/UNICEF recommendations and that a ‘family friendly’ approach will be adopted to complement the Baby-Friendly Hospital Initiative. The ‘family friendly’ approach includes promotion of family support to mother and child during the pre-natal, natal and post-natal periods, promotion of partner attended labour, ensuring free family member access to maternity and children’s hospitals and the creation of a family friendly environment at medical facilities. Special attention will be paid to children living in rural areas and disadvantaged communities.

The project ‘Ensuring young children’s development in rural areas of Ukraine through parenting education’ in Ivano-Frankivsk was designed in accordance with the UNICEF and Government of Ukraine 2006-2011 CPD and CPAP and is implemented in 2006-2011.

At the beginning of the project (2006) UNICEF supported health screening for young children and assessments of parental knowledge. These studies demonstrated that the quality of the primary health care practices existing was poor and that parents and health care workers’ knowledge regarding childcare, nutrition and development was inadequate. Accordingly child health indicators were unsatisfactory: there was high morbidity, a low exclusive breastfeeding rate, a high level of anaemia, high iodine deficiency levels, etc. The role of fathers in child care and development was limited for many reasons: their lengthy absences while they performed seasonal work, lack of child care knowledge and skills, and traditional societal beliefs about gender roles. Poor adult supervision and care at home caused many poisonings, burns, injuries and traumas in young children. At the same time there was no parental counselling or training regarding childcare and development. The majority of villages in Ivano-Frankivsk oblast lack kindergartens, children’s playgrounds or any other safe community spaces in which young children can play under adult supervision.

In July 2008 heavy rains and storms in the Carpathian Mountains led to flooding in parts of Western Ukraine. Ivano-Frankivsk oblast was the most affected region. Thirty people died, 445 settlements were heavily affected, 28,000 houses were ruined, 23,000 hectares of agricultural land were damaged, 1,160 kilometres of roads were washed away, 500 bridges were destroyed and 9,370 people were evacuated from flooded areas. A UN disaster management team assessment concluded that rural communities did not receive adequate support from the government to relieve the consequences of the flood. The Cabinet of Ministers of Ukraine and the Ministry of Health requested that all international partners and UNICEF in particular reconsider existing cooperation plans and focus on assistance in flood-affected areas. As a result, in 2008, a new objective was added to the project–providing flood relief assistance to damaged villages that host project sites. To achieve that, in 2008-2009 UNICEF supported reconstruction of damaged health care centres in villages involved in the project (Kosiv, Yabluniv, Velyky Rozhen, Banya-Bereziv and Rostoky). The office of the UNICEF project implementing partner – the Kosiv Centre for Public Initiatives NGO – was also damaged by the flood. So that it could continue project implementation, the NGO received additional assistance from UNICEF towards rebuilding its damaged premises.

UNICEF piloted two main approaches within the project: partnership with local government and partnership with non-governmental organisations.

That is why there were two main implementing partners, which have simultaneously been implementing quite similar activities:

Ivano-Frankivsk oblast children’s hospital (state organisation – oblast level)
Kosiv Centre for Public Initiatives (non-governmental organisation – rayon level)

The reason for this was to find the most effective operational model: a) a vertical approach – integration into state administrative; and b) a horizontal approach based on community initiatives.

Other project partners were the Ivano-Frankivsk Institute for Post-Graduate Pedagogical Education, the Ukrainian Step by Step Foundation, the Ivano-Frankivsk Press Club and the Ukrainian-Swedish gender project OLEH. At the governmental level the project was supervised by the Ministry of Education, the Ministry of Health and the Ministry of Family, Youth, and Sport.


Purpose/Objective:
The main purpose of the evaluation is to define the project’s results as compared with its main objectives. It was also to see how these results were achieved and whether they are sustainable.
The project ‘Ensuring young children’s development in rural areas of Ukraine through parenting education’ in Ivano-Frankivsk oblast” included three key specific objectives:
Objective 1. Build the capacity of local governments, the health care system, social services and the education system to meet the basic needs of families with children
Objective 2. Improve the quality of and access to primary mother and child health care services;
Objective 3. Ensure that parents have access to counselling on childbirth, childcare and child development and that their children have access to developmental activities.

Methodology:
The proposed type of evaluation is formal and summative. The evaluation process should be based on:
• analysis of existing project-related documents
• analysis of existing national and local policies/priorities
• development and analysis of surveys/questionnaires with a list of questions/indicators against set objectives (quantitative and qualitative data)
• field visits to the oblast, rayons and communities
• meetings with local authorities, involved specialists and families

During the field visits and meetings:
• in-depth individual interviews with key stakeholders and implementing partners
• field visit observations
• structured interviews with project beneficiaries
• focus groups discussions with community advisers, frontline workers/volunteers and families and members

The external evaluation was based on both quantitative and qualitative approaches. An external evaluator carried out the qualitative part. It was based on desk review, focus group-discussions, field visit observations and individual interviews with project beneficiaries, key stakeholders and local specialists. The external evaluator developed terms of reference for the local research agency, which will be responsible for the quantitative part. In accordance with these terms of reference a competitively selected local agency (Kyiv International Institute of Sociology) analysed oblast, rayon and village health statistics (mortality, morbidity, iron and iodine deficiency indicators, breastfeeding rate, traumas and injuries, etc.) and identify major changes and areas for improvement. The Institute also collected data from implementing partners regarding the number and proportion of parents who received training at Schools for Responsible Parenting and Papa Schools; the number and proportion of children born at Baby-Friendly Hospitals; the prevalence of partner attended labour, etc. In addition, via structured interviews with project beneficiaries this agency will also collect data that are not included in the state statistics.


Findings and Conclusions:
The future of Ukraine’s development depends on the health and well-being of its children. Early childhood experience can provide the child with adequate tools to cope, learn and ensure future opportunities. However, especially in rural area efficient access to basic child health and development opportunities continue to remain a serious problem. Fortunately, there are various interventions based at education, health and social service levels, which collectively creates common ground for strategies to promote healthy, positive development for all children. The resulting body of knowledge offers the public, policymakers, and community members an unprecedented opportunity that did not exist before to catalyze the creation of effective, science-based policies and practices to prevent child health and development.
Relevance
The project was designed to improve young children health and development in rural areas of Ivano-Frankivsk and was highly relevant at the start time and even at the time of project end. The project was a timely and appropriate response, given the situation created in rural areas with young child health and development.
The project targeted especially rural, mountain areas, where services focused on child health and preschool development were accessible. The project responded to children’s needs with a comprehensive and integrated interventions and services. Activities were taken across the health and education sectors, and included capacity building for local governments, health, social and education systems to meet the basic needs of families with children; improve the quality and access to primary mother and child health services; and counselling on childbirth, childcare and child development for parents, including access to developmental activity for children. All above mentioned interventions are known to be efficient and effective measures in promoting optimal young child development.
The project was adapted to health, social and developmental services delivery for young children. Approach and design used during the project implementation appropriately incorporated capacity building for early child development stakeholders and services providers at national, oblast and local level. The project included also effective coordination mechanisms and management support systems to address children needs in a harmonized and integrated manner. The project interventions and capacity building efforts were based on UNICEF country priorities (UNDAF- toward institutional reforms that enhance outreach to enable parents and children to fulfil their hu¬man rights; (2) civil society empowerment to enable people to access services and enjoy their rights; (3) health care and health services with a special focus on raising quality and ac¬cessibility, and (4) prosperity through balanced development and entrepreneurship, CPAP and CPD), along with country national programmes, numerous research and development studies. In general terms, the project was timely, appropriate and comprehensive. Poor child health and inappropriate development can lead to developmental retard and lower achievements during all life in terms of health and wellbeing. Focusing approach on health and education was motivated and strategic. Multi-sectoral approach helped in harmonizing and foster relationships among the different sectors and levels of early child development services providers at national, oblast and local levels.
Impact
The project’s expected impact was to ensure young children optimal health and development. In general terms, the impact has been achieved. A series of key outputs can be assigned as project outcomes. These are the rate of breastfeeding, BFH related outcomes, proportion of partnership deliveries, morbidity for some health indicators, parents general knowledge level on child health and development, institutional cooperation, etc.
Breastfeeding rate increased compared with the baseline period before the program; for whole Ivano-Frankivsk breastfeeding up to 6 months increased from 41.2% in 2006 to 51.2 in 2009 year.
The number of baby friendly hospitals increased from 1 at baseline in 2006 to 13 in 2009, respectively, the number of babies born in BFH increased from 5.7% to 86% and the number of partnership deliveries increased from 56% to 80% during the same time period; partnership delivery create support and relaxing atmosphere, increase man attitude towards participation in child development process.
Morbidity for anaemia (per 1,000 population) decreased from 76.4 in 2006 to 58.3 in 2009; similarly diffuse goiter level decreased from 36.6 per 1,000 population in 2006 to 29.5 in 2009; however, general morbidity rate per 1,000 population presented an increasing tendency, from 1715 in 2006 to 1804; these results show that child health requires further concentrated and complex effort over a long time period.
Project interventions positively affected knowledge and behaviours of parents and families in terms of child health and behaviour; and many new achievements and practices in child care are widely implemented, even facing some resistance especially from aged family members.
The general competence level of parents in terms of child care and development increased, due to program activity (average 8.7 out of 10), as well as due to increased general information access and level, including sharing obtained information among community members.
Parents acquire a new sense of their personal competence through child development and caring relations with children; their socio-emotional competence is enriched as they gain new insights and skills relative to their roles in empowering children and enriching; their own growth; main attributes are noted as important outcomes for empowering parents: committing to the role of parenting, enhancing one’s human competence, and developing a family relationship.
Children involved in child development programs on regular basis, presented better results during entrance in elementary school exams; they can communicate easy, have higher cognitive development, can easy learn new information and are better adapted to collective environment.
Parents prefer child development centres vs nurseries because in such centres children receive personal attention and use new approach / techniques in child development; individual consultations are also available (2/3 of all visitors) and general satisfaction level with school of responsible parenting was 9.5 out of 10.
Especially in Ivano-Frankivsk significantly improved relations between medical staff and patients, less complains from patients.
Average length of stay for paediatric patients decreased from 12-14 before the project start to 6-7 days presently.
Decrease in average length of stay in maternity (from 6-8 to 3-4 days), childbirth room became more comfortable.
Psychological training of women in maternity decrease the duration of childbirth, pain intensity; the incidence of birth injury decreased.

Effectiveness
The overall effectiveness of an early childhood program is dependent upon several factors: quality staff, suitable environment, appropriate grouping practices-adjusted to local needs and realities, consistent schedules, funding and parent involvement. Decisions about these factors often need to be made early in the planning and organizing process for an early childhood program. These decisions have important ramifications because they affect the child, the family, the classroom, the centre of development/nursery, and the community.
Key elements of early child development programs include early education and stimulation for preschool children along with support and training for parents and caregivers to improve children’s experiences at home and in the community. The strategy selected for project implementation was complex and cost-effective. Multi-sectoral approach, using medical facility as settings and medical professionals as key message promoters, teaching methods, teaching materials format definitely are among the best options in providing comprehensive services for child health and development in Ukraine. In rural settings, such methods are effective and provide best cost-effectiveness in terms of outputs.
Project activities contributed to institutional capacity building and changing in the behaviour of individuals and communities. In all visited project site we found good cooperation with rayon and oblast level project site, permanent exchange of experience and practice, continuing education for project staff and satisfactory supply with necessary support. At individual level, positive changes occurred first of all in the level of knowledge, changed the mode of acting in normal and stress situation related to child development, increased responsibility towards child health and development, stimulated individuals to search for new information focused on child development, increased cooperation and shared responsibility at family level-including shift in mentality among old family members, positively influenced gender roles. At community level, around 39% of all respondent and 96% of centre participants appreciated positive influence of child development centres on communities, and only 0.5% and 1.3% respectively found no influence on communities. Community collaboration is essential in setting goals and providing focus for early child development program.
Because child development in intimately linked to primary health care for mothers and children, due to integrate approach some specific outputs especially related to maternal/child health (breastfeeding, childbirth, immunization) improved. Similar positive trends were observed for some morbidity parameters, like anaemia, diffuse goiter, and percentage of children born in baby friendly hospitals. However, for significant improvement, more efforts are required at national level, especially in network system configuration, access, clinical protocols, medical personnel and equipment supply.
This project contributes to increasing cooperation level between education, health and social sectors in order to improve child health and development approach. At local administration level, visible positive changes occurred and local authorities increased awareness on young child development issues. In our sample, 39% of all respondents and 96% of child development centres participants answered that such centres presented positive influence on community.
The project significantly contributed to increasing role of fathers in child care and development. According to our survey, training received in Papa School reduced the risk associated with child health and development in 56%, was helpful in child development and in improving child health in 56 and 44 % respectively, helpful in building relation with own child and only 4% responded that training had no influence. Similarly, 56% responded that responsibilities related with child care were shared equally, families became more friendly in 36%.
The intervention was more effective especially in increasing knowledge about child health and development, sharing information among program participants and non-participant, family members, changing stereotypes, and capacity building for community members. Main factors are first of all related with overall socio-economic conditions in Ukraine, availability and accessibility of projects in certain village, willingness to participate in such programs, good image created by project activity, general level of informatization, availability of internet and other media resources, topics for discussion, for parents-sufficient level of knowledge in order to understand child development issues, ability to trust new recommendations and developments, comprehensiveness of material and teaching style.
The overall rate of usage of services is generally difficult to establish, due to changing environment, difficulties in monitoring, follow up and population flows. However, according to informal estimates it range from 20% to 30%, with higher rates for school of responsible parenting and family centres and less for Papa Schools and child development centres. Need to be mentioned that even population located in villages close to project site are often visitors of centres. As usual, school for responsible parenting, child development centres, family centres and Papa Schools are usual located within medical facility, and in rural areas not all localities have such facilities. Additional factors, like employment, school/centre invitation or image in community, parents’ sense of self-efficacy, and beliefs about their need to be involved in their child’s development.
Majority of parents are fully satisfied with provided services. For example, on a scale from 1 to 10, the average satisfaction level with centres of child development was 9.5 (±1.1) and satisfaction with visiting Papa School was on average 8.8 (±1.4) out of 10.
Parents unable to meet their own and their children's basic needs undoubtedly experience stress, frustration, isolation and disempowerment and therefore require support. The poorest and most disadvantaged are likely to attend young child health and development programs at less degree, because of specific social circumstances, e.g. lack of motivation, remoteness, difficulties in understanding health or education messages, negative health behaviour practices (alcoholism, substance abuse, etc). Specific approach to such families, joint effort with social services, understanding local circumstances and finding new solutions in order to reach the unreached is required.
Efficiency
Overall, the project is rated efficient. Majority of interventions were properly synchronised, with high degree of collaboration. Fund disbursement at closing on $313,623.74 reached 100% of total estimated cost. The UNICEF financial output was properly targeted to address project goals. Moreover, after the financial support ended, majority of activities continued, because of high demand among community members and many trainers performed usual activities on volunteering basis. Utilization of manuals, references, DVD, etc was fully maximized. Resources spent on refurbishment created very good impression for participants, along with TV/video sets, PC and availability of literature resources. Intangible benefits are primarily family focused benefits (improved gender roles, equal distribution of duties related with child development, child-family bonding), maternal health benefits (benefits related with breastfeeding, recovery in postpartum period, mental wellbeing), child health (optimal growth and development, increasing advanced communication ability), professional and personal development in parents (absenteeism from work due to child illness, leave benefits), overall family and emotional wellbeing, community empowerment. Very important moment to be mentioned, Papa Schools did not receive any financial support from the UNICEF.
Sustainability
Obtained results at oblast and rayon levels show project gains as very likely sustainable. However, early child development remains not so high on national and oblast level leadership agenda. The Government’s efforts are not sufficient in providing fertile ground for continued utilization of project outputs. The intensive capacity development efforts provided to main stakeholders at all levels have strengthening the institutional system for delivering early child development services. Integrated, multi-sectoral approach, cooperative network, continuing capacity development efforts for project staff and parents, empowering parents, increasing awareness, enabling communities to contribute to child development projects, adequate financing, advocacy for child development, long term vision/planning, continual improvement, based on transparency and accountability, commitment of rayon and oblast officials/administration are among the key sustainable mechanisms for the activities and results that project established. UNICEF provided leadership role in project planning and implementation. Literature, equipment, references, approach methods and advices received from UNICEF provided a solid foundation for project sustainability. We found that existence of regional/rayon level coordination centre significantly facilitate sustainability and project implementation.

Recommendations:
Continuing support is needed for capacity building within governmental, social, education and health services. Specific attention to strengthening the knowledge basis, information dissemination and consolidating partnership aimed to enhance and scale up child health and development.
Promote universal access to evidence-based young child development programs for families from diverse socioeconomic, cultural and ethnic backgrounds. Specific attention for most disadvantaged families.
If evidence-based impact evaluation on young child development project shows positive results, political support can be generated. Where problems are uncovered, program adjustments can be made. Where successes are publicized, additional funding might be mobilized.
Need a more collaborative and supportive approach with beneficiaries to assist in development of similar programmes and projects.
An integrated model is the best approach in young child health and development program. Promote inter-sectoral, interagency, interdisciplinary approach and cross-system collaboration to implement young child health and development programs at national, oblast and rayon levels. Collaborative networks of centres will help in decreasing costs, reduce inconsistencies in training and increase M&E.
Would be useful to introduce commercial components at larger scale in all child development programs. Individual format consultations, home visitation, selling/lending of DVD, CD, and literature could help in collecting additional financial resources.
It’s important to have better knowledge transfer between various project sites when they are happening e.g. through a moderated internet platform – a type of collegiate model of support using something similar to Odnoklassniki, Facebook or Twitter. Wide use of technology, especially 24/7 cell phone counselling would be useful.
Mediatisation of child development programs, using various famous people for propaganda of young child development.
There is a need to have a designed staff to be responsible for implementing and enhancing the M&E component.
There might have been challenges in establishing baseline for key result areas against which the project impact could be measured e.g. incident and prevalence of childhood illnesses at start in comparison with the situation at time of this evaluation or project end, and other areas like mortality and morbidity; rate of breastfeeding, partnership delivery, psychosocial outcomes at start as compared to time of project end etc.
In order to have long lasting effects, such programs need to be sustained over multiple years.
In Ukraine there are no official standards on child development, preschool education, parent education on child care and development and standards for comprehensive basic services for young children. Indicators on child development need to be focused mainly on cognition, socio-emotional, language and emerging academic skills (early notions of reading, writing and mathematical capacity). Also, there is no continuity and sustainability in legislation pertaining to child development.
Development of nursery for children in rural area need to be suggested as major priority for government, regional and local authorities.
Communities need to be selective in locating promising programs for implementation and matching the model to the needs of a particular site and population. According to our practice and experience, medical facilities are the best places for young child health and development programs.
Little recognition by the government of the crucial role of the young child development is evident. Central government has little political will to contribute efficiently to progress of young child development. Specific efforts on oblast and rayon level administration are required. Communities need to play stronger role in activities of child development programs.
The effects of proper young child development and the consequences on society as a whole are widely ignored. At this point, there is poor understanding that young child development is a public health and social problem which requires multi-sectoral, long term, complex interventions rather than a single, short term measure.
Increase awareness among medical and educational community. Workforce development plan need to be elaborated. Such a strategy needs to take into account that the shortage of trained professionals, the difficulties of recruiting and retaining skilled workforce and inadequate financial remuneration. Elements of voluntarism could be implemented in such programs.
Programs on child health and development need to share core elements and content. Teaching positive parenting practices, providing parent education, skilled training, social support, problem solving, using standardized curricula delivered by trained professionals with strong control mechanisms.
For children, especially is important highly trained and responsive caregivers, small class sizes with low child/teacher ratios, safe and adequate physical environments, and age-appropriate activities focused on enhancing the cognitive and socio-emotional development of the child.
Need to accumulate additional evidence for improving programs on child health and development. Comparison between Ukrainian and international practice is required. All programs need to be adapted to local conditions.
Adequate funding need to be provided. Existing funds are not adequate to support the implementation of programs. New resources will be required.
Transparency, monitoring and evaluation, quality control and accountability need to be implemented across all programs
Separation into four different groups infant, toddler, younger preschool, older preschool would be advisable for child development centres
Would be useful to introduce early child health and development module within curricula for medical professionals

Lessons Learned (Optional):
Projects oriented on young child health and development can be addressed within a relatively short time period through complex, multi-sectoral approach. Child development programs require the commitment and participation of various stakeholders at national, oblast and rayon level. Strong national and oblast level support could facilitate implementation of child development programs. Methodological and financial support from UNICEF and international stakeholders is extremely beneficial for project viability and positive effect. Financial component is important issue and sustainable financial support for child development programs need to be ensured.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search