Base de datos de evaluación

Evaluation report

2011 Global: Evaluation of UNICEF's Early Childhood Development Programme with Focus on Government of Netherlands Funding (2008-2010) - Global Synthesis Report



Executive summary

Background:

Propelled by research evidence that investments in well-implemented early childhood development (ECD) interventions can improve children‘s well-being, ECD has increasingly been highlighted as a key strategy in reducing poverty, health and education disparities. This report synthesizes the findings from an evaluation of the United Nations Children‘s Fund‘s (UNICEF) ECD strategies and activities with a focus on the UNICEF-Government of the Netherlands (GoN) Cooperation Programme on ECD. From 2008-2010, the programme provided a 13.5 million (US$) investment toward furthering ECD and advancing comprehensive services, with a focus on sustainable policy development and partnerships to scale up successful interventions.

Purpose/ Objectives:

Under the programme, the GoN funded UNICEF headquarters (HQ) to work with its regional offices (ROs), 10 selected country offices (COs) and country partners in Africa and Asia in achieving three strategic objectives: (1) mainstreaming ECD policies into national plans, policies and services; (2) building the capacity of policymakers, service providers and parents to fulfill duties and claim rights related to ECD; and (3) generating and disseminating knowledge in support of ECD policies and services. In April 2010, UNICEF‘s Evaluation Office contracted with Mathematica Policy Research to conduct the evaluation.

Methodology:

The evaluation scope consisted of two related components: (1) four in-depth country case studies of UNICEF ECD strategies and (2) an assessment of UNICEF ECD strategies at the HQ, RO and CO levels, including cross-country comparisons among the 10 countries funded through the UNICEF-GoN Programme.

The evaluation assessed the status of ECD strategies and activities, and also the factors reported by informants that influenced processes and drove progress toward targeted outcomes. It was a retrospective, theory-based assessment of the processes and results of ECD strategies and activities, employing mixed methods (primarily qualitative but incorporating survey data as well). It drew on logical frameworks for ECD to establish a theoretical foundation for inquiry; processes and outcomes were assessed in reference to those specified in the frameworks. The evaluation relied on data from four sources: (1) a desk review of data and programme documents, (2) executive interviews with key informants, (3) four country case study site visits and (4) an internet-based survey of all UNICEF COs. The evaluation used three analysis methods to address the research questions: thematic framing, triangulation and indicator ratings.

Conclusions:

The evaluation conclusions presented here are based on the findings detailed in the body of the report. Conclusions are presented about (1) the three strategic areas of mainstreaming, building capacity and generating and disseminating knowledge described above; (2) ECD service coverage, quality/efficiency and sustainability; (3) the cross-cutting issues of planning, management, coordination and partnerships, as well as equity and reaching the less reached and disadvantaged; (4) overall effectiveness and relevance/appropriateness and (5) contribution of the GoN investment. Additional conclusions, focused primarily on country-level findings, are included in the body and in the final chapter.

Mainstreaming ECD Policies into National Policies, Plans and Services
Clear communication about the benefits of ECD investments contributed to mainstreaming. Advocacy efforts at all levels that focused on communicating the unique and synergistic contributions of ECD to improving both short- and long-term outcomes for children, families and communities increased excitement and commitment to ECD mainstreaming among government leaders and decision makers.
Involvement of finance ministers in the ECD policy and planning process and costing studies are useful in shaping policy development, advocacy and budgeting for ECD. To engage finance ministers and partners in allocating funds that mainstream ECD into national policies and programmes, UNICEF and country counterparts need ECD-specific costing data and tools for conducting simulations of different funding strategies.
The findings are mixed about the relative benefits of sectoral versus intersectoral approaches to mainstreaming and universal versus more targeted policies. Findings from the four case studies highlight that sectoral and intersectoral approaches have demonstrated successes and challenges. Overall, evaluation respondents viewed intersectoral approaches as desirable for supporting integrated, holistic ECD. Universal scale-up of ECD interventions can produce rapid increases in coverage, but may be associated with compromised service quality and lack of equity in access to services. In countries with more targeted and slower phase-in, ECD service coverage tends to be lower.
Efforts to mainstream ECD messages into other types of interventions are progressing. UNICEF‘S investment in ECD materials that can be added to programmes conducted as part of health and nutrition services provides a model for doing so in other areas, such as water, sanitation and hygiene (WASH); child protection and social protection.
At the UNICEF CO level, decisions about which section the staff member primarily responsible for ECD is assigned to and how the CO approaches coordination of intersectoral ECD activities influence the level of shared understanding, coordination and ability to support country partners in making progress toward mainstreaming. In several of the COs that received UNICEF-GoN funding, an intersectoral ECD committee meets regularly to assess needs and progress. This approach provides an important model for the kind of intersectoral ECD coordination that UNICEF advocates for with its country partners.
Building Capacity for ECD
UNICEF does not use a systematic approach to assessing ECD capacity gaps, implementing capacity-building activities, documenting participation at the individual level and using data to focus follow-up efforts. The need to coordinate and document systematically ECD gaps at all levels (national, subnational and local) is critical to optimizing the investments. Data systems are needed to track participation of ECD service providers and target resources to those who have not received basic training and required refreshers. Similar approaches are needed to target families or geographic areas.
Parent/caregiver exposure to ECD interventions/messages is uncertain because of minimal data, but most interventions are of too low an intensity to support lasting impacts on parent behavior. Research evidence increasingly demonstrates that brief interventions (for example, one-time workshops) are not sufficient to change adult behavior with children (Winton 2008; Winton and McCollum 2008). The evaluation found little evidence that evidence-based adult learning approaches are being used as part of existing ECD capacity-building activities.
UNICEF COs reported that ECD capacity grew over the past four years, but current needs reflect challenges related to resource constraints (too few staff and too little ECD-specific expertise) and bringing additional ECD expertise to the organization. The relatively small number of staff working on ECD issues and limited resources inhibits progress toward targeted outcomes. COs are seeking ECD-specific capacity building and staff with expertise in ECD as well as in the areas of reaching the marginalized and disadvantaged, costing and supporting policy implementation at the national and subnational levels. In addition, the role of ROs and the adequacy of supports they provide to COs in addressing these and other needs are not meeting their potential.
Generating and Disseminating Knowledge for ECD
The efficiency of knowledge generation and dissemination at both the global and country levels is diminished by a lack of coordinated, systematic planning and rigorous evaluations. Insufficient coordination among HQ, ROs and COs in establishing research priorities and planning evaluations detracts from development of a focused research agenda in ECD and results in missed opportunities to leverage resources for more rigorous, longer-term country-specific and multi-country evaluations. Current processes at the country and global levels do not facilitate sequencing of evaluations into formative and summative stages.
UNICEF’s promotion and use of findings from the MICS4 ECD module data are expected to continue to produce substantial benefits to all levels of the organization and to country counterparts. In particular, the resulting summary ECD indicators will facilitate national monitoring and international comparisons of children‘s progress in key developmental domains. Because the module does not include items on infants and toddlers, however, it does not cover the full conception-to-8 age span, which remains a gap.
Current knowledge management practices within UNICEF do not adequately allow public access to findings from previous and ongoing research and evaluation projects. CO, RO and HQ internet pages are challenging to navigate and do not provide a catalog of the studies UNICEF has commissioned or contributed to over time. There is no one-stop location that provides up-to-date information on research, monitoring and evaluation projects in formats designed to meet the needs of diverse audiences.
ECD Service Coverage, Quality/Efficiency and Sustainability
ECD service coverage of center-based pre-primary education has expanded, but coverage and quality are uneven. In all four case study countries, coverage varies across geographic areas and social groups, with children from urban areas and higher-income families typically having more access than children from rural areas and economically or socially disadvantaged groups. Quality of services is also uneven in terms of teacher training, facilities, materials and the number of children per classroom.
ECD service coverage designed for parents of children from birth to age 3 has been limited. Less progress has been made in increasing service coverage for parents of younger children from birth to age 3. In most case study countries, attempts have been made to integrate ECD messages in existing community health services, with varying degrees of success.
Systems are not yet in place to provide adequate training, monitoring and technical assistance necessary for improving quality of programming. All case study countries reported providing some training to teachers and/or community health workers, but problems with training coverage were evident, especially regarding training for replacements and refresher training. None of the case study countries had established a system for assessing ECD quality, reporting and tracking results and using results to improve quality through training and technical assistance.
Planning, Management, Coordination and Partnerships
Greater emphasis in developing and implementing cohesive and well-defined reporting requirements for ROs and COs would yield higher quality data to inform assessments of progress toward targeted ECD outcomes. The reporting questions designed by HQ and advisors (including donor representatives) about the UNICEF-GoN funding and how it was used were not specific enough to ensure consistency in CO and RO reporting. Lack of detailed definitions of the data elements and training on appropriate data sources and collection methods, as well as provision of context for the information, resulted in inconsistent responses and unverifiable data. Evidence of the use of data to make midcourse corrections or share lessons across the ECD network was scant.
Support for ECD from HQ and ROs and within COs is considered adequate but there is room for improvement in both UNICEF’s vertical alignment with COs’ needs and horizontal alignment within COs across office sections. Findings from the CO survey identified needs and gaps that can improve alignment of HQ and RO strategies and activities and make them more useful to COs. At the CO level, horizontal alignment and coordination tend to be better in COs where ECD-related funding is more evenly spread across sections or where staff are assigned specifically to work on improving alignment and integration of ECD activities.
By and large, partnership building for ECD has been successful. An array of partner organizations collaborates with UNICEF on ECD programming at the country and regional levels. A continued focus on sectoral funding strategies may inhibit full participation in supporting holistic ECD among some development partners. There is a clear need for advocacy and support for donor groups interested in ECD and in developing relationships and shared understanding of the links among sectors in regard to achieving shared goals and outcomes for children and families.
Equity and Reaching the Less Reached and Disadvantaged
Case study countries’ capacities to improve access for less reached and disadvantaged children were limited by lack of data and strategies for increasing access. For most of the disadvantaged and marginalized groups, the available data are not adequate for careful monitoring of equity in access, although the available data indicate likely gaps in equity. For example, data are generally not available about enrollment of orphans, but site visits indicated possible inequities in access for this group. Evaluation respondents could not articulate specific strategies for reaching marginalized and disadvantaged children and families and enrolling them in ECD programmes.
Globally, UNICEF’s role in providing leadership in encouraging innovation in advocacy for reaching underserved populations is critical for making progress in this area. The ECD Unit‘s advocacy and leadership keeps ECD staff at all levels and partners focused on developing the tools needed to assess progress, target services and support outcomes. Formal tools and training are needed at all levels.
Overall Effectiveness and Relevance/Appropriateness
Overall, evidence exists that UNICEF’s ECD strategies were effective in meeting targets related to outputs (such as number of parents trained), but evidence of effectiveness in improving intervention quality and outcomes for children and families is scant. Without a more systematic approach to assessing needs, quality of services delivered and outcomes, rigorous assessment of effectiveness is constrained.
Taken together, UNICEF’s ECD strategies and the UNICEF-GoN Programme funding were relevant and appropriate to making progress toward targeted outcomes. Generally, the strategies and activities were adequately aligned with stated goals and the logical frameworks developed for the evaluation. Systemic challenges related to the availability and use of data on national and subnational ECD needs impedes better alignment of UNICEF‘s strategies with identified needs.
Contribution of the GoN Investment
GoN funding in the 10 countries increased awareness of and commitment to ECD among national and subnational leaders, catalyzing efforts aimed at increasing access and quality of services offered to families and children. Country-level investments in increasing the ECD capacity of service providers and parents and getting high-quality training and instructional materials into their hands enhanced progress toward improving children‘s outcomes.
The GoN’s multiyear investment increased UNICEF’s influence, reach and credibility as a partner in ECD at the country, regional and global levels. The funding enabled UNICEF to have a greater role in engaging partners, influencing how funds were spent and leveraging investments.

Key Recommendations:

Key recommendations based on the evaluation findings and conclusions are presented on (1) the three strategic areas; (2) ECD service coverage, quality/efficiency and sustainability; and (3) the cross-cutting issues of planning, management, coordination and partnerships, as well as equity and reaching the less reached and disadvantaged. The intended audience for each recommendation (UNICEF HQ, ROs or COs) is indicated in parentheses. Additional recommendations, focused primarily on country-level findings, are included in the body of the report and in the final chapter.

Mainstreaming ECD Policies into National Policies, Plans and Services
Seek stable, multiyear funding of policy mainstreaming strategies to catalyze the transition from ECD policy development and adoption to high-quality implementation (UNICEF HQ). Focus investments on countries that adopted ECD policies or mainstreamed them within the past two years, specifically providing funds for those committed to trying leading approaches to building infrastructure to support policy implementation. Invest in development of mainstreaming models and test them in formative research.
Identify a summary indicator or small set of indicators for ECD that could be tracked and reported at subnational, national, regional and global levels to focus advocacy and mainstreaming, build awareness and track progress toward critical outcomes (UNICEF HQ). For example, develop summary measures of ―on-track development‖ for ages 1, 3 and 5 that incorporate measures of cognitive, social-emotional and physical development.
Provide training and technical assistance on costing ECD policies and strategies and identify promising practices for involving finance ministers in ECD planning and costing (UNICEF HQ and ROs). Use lessons from recent costing efforts in developing guidance and streamlined costing tools for COs. Consider ways to consolidate costing across sectors to facilitate the costing of intersectoral efforts such as ECD.
Evaluate sectoral versus intersectoral approaches to ECD policy and programme development, as well as universal versus targeted approaches to mainstreaming (UNICEF HQ and ROs). Systematically track differences in approaches at the RO and HQ levels and develop models for testing these approaches against each other.
Promote use of the ECD Resource Pack to inform country counterparts about the benefits of ECD investments and develop versions tailored for targeted stakeholder audiences (UNICEF HQ, ROs and COs). For example, tailor the Resource Pack for use with top policymakers and ministry-level staff to differentiate the value of ECD investments from investments already being made, and do the same for stakeholders at subnational levels.
Compile and disseminate promising practices for mainstreaming ECD messages into other types of interventions (UNICEF HQ). Use findings from implementation research on Care for Child Development to create a how-to manual designed to help COs obtain buy-in for incorporating ECD into existing interventions. As needed, adapt the approach UNICEF HQ took to working with global partners on Care for Child Development to develop modules on ECD that can be added to WASH, child protection and social protection interventions.
Advocate for a consistent CO organizational structure for the ECD focal point that includes a reporting relationship to the deputy representative and provides clarification on the focal point’s responsibilities (UNICEF HQ). This reporting structure would underscore the broader responsibility of the ECD focal points beyond the section in which they are housed and provide accountability to keep cross-cutting efforts moving forward. Consider a similar structure for HQ since responsibilities of the ECD Unit extend beyond young child survival and development.
Building Capacity for ECD
Develop and advocate for implementation of a systematic approach to capacity building that includes assessing needs, implementing evidence-based training, tracking completion of service provider training and parenting education and assessing and evaluating outcomes (UNICEF HQ). Develop the capacity and infrastructure necessary to identify training needs and develop, implement and evaluate capacity-building approaches. Align ECD goals and investments with expected outcomes by targeting specific capacity-building strategies to meet the needs of policymakers, government officials and planners, programme implementers and parents. Adapt the UNDAP capacity development approach to ECD and develop formal training modules designed to meet country and global needs. Evaluate changes in capacity-building infrastructure and outcomes.
Continue to invest in existing resources for capacity building, such as the ECD Resource Pack and ECDVU, and develop new resources to address capacity gaps (UNICEF HQ and ROs). For example, UNICEF COs expressed the need for additional training and guidance in the areas of equity and reaching the marginalized/disadvantaged, costing and finance, quality improvement and training of service providers. Increase the return on these investments by allocating funds to translate and adapt ECD capacity-building materials into more languages and for use by service providers, parents and children.
Invest in developing models for parent/caregiver training based on research evidence about the dosage, content and training approaches that are likely to produce intended outcomes (UNICEF HQ). To ensure that parents/caregivers become engaged, begin by planning interventions that take into account factors that increase and those that inhibit the consistency of participation. Consider including meaningful incentives designed to attract parents and caregivers to training events and activities. Incorporate adult learning principles into training designs to maximize the likelihood that training will produce positive and lasting changes in parent/caregiver behavior.
Hire child development specialists to strengthen the role of the ECD Unit within UNICEF HQ and have dedicated ECD advisors in each RO (UNICEF HQ and ROs). Increase the number of staff at UNICEF HQ with a specific background and focus in ECD who can help integrate ECD with other sectors and provide a greater presence in key planning and decision-making activities. Create or fill RO ECD advisor positions to improve relevance and efficiency in meeting CO ECD-specific needs. Clarify the role of the RO in providing ECD expertise to COs and the region. Provide more technical support to COs on ECD policy advocacy and costing efforts.
Generating and Disseminating Knowledge for ECD
Develop a multiyear, integrated research and evaluation agenda, coordinated across organizational levels and regions, that includes a continuum of formative and summative evaluation to support programme improvement (UNICEF HQ). Create an agenda that describes the state of children; documents the dosage, content and quality of interventions; and rigorously assesses impacts on children and families. Synchronize agendas across organizational levels and regions to address key knowledge gaps and facilitate pooling of resources for larger evaluations. Make findings and lessons learned readily accessible. Finally, develop a system for using research and evaluation findings to inform ECD policies and interventions.
Invest in knowledge management systems that catalog past and current research and evaluation projects at all levels and make them available on public websites (UNICEF HQ). Systematically distribute information about UNICEF-supported research activities and reports outside of the organization through dissemination channels with a broad policy and practice audience. This includes participation at conferences as well as maintaining comprehensive and up-to-date public websites with publications databases that cover previous and current research projects.
Continue to invest in the MICS4 ECD module and to advocate for its use by more countries (UNICEF HQ). Widespread use of the module will help to close the existing knowledge gap about children‘s progress globally in key developmental domains. Consider expanding the module in the future to include items on infants and toddlers.
Work with COs and country partners to fully develop Early Learning and Development Standards (ELDS) for the conception-to-8 age span, use ELDS as the basis for developing training and monitoring systems and evaluate their effects on targeted outcomes (UNICEF HQ and ROs). For example, ELDS can serve as the basis for defining quality in ECD interventions and for developing monitoring processes and tools for assessing the degree of adherence to ELDS. Gaps identified through monitoring can inform ongoing training and technical assistance. Use ELDS to inform curricula and training materials for staff and community volunteers who deliver ECD services and programming. Evaluate ELDS efforts to identify successes and challenges. Develop guidance on how to maximize the contribution of ELDS to achieving improved service delivery systems and outcomes for children.
ECD Service Coverage, Quality/Efficiency and Sustainability
Advocate for increased funding levels and intersectoral donor groups to increase sustainability of ECD strategies and interventions (UNICEF HQ, ROs and COs). Involving finance ministers in ECD planning and informing finance and other ministry-level staff about the benefits of holistic ECD interventions contribute to sustainability of ECD interventions. To reduce turnover (an inefficiency related to training resources), governments may need to shift from unpaid or minimally paid community volunteers to more paid staff over time, requiring additional funding allocations to scale up and sustain service quality. UNICEF is in a strong position to advocate among donors about the need for a holistic, long-term approach to ECD interventions and engage intersectoral donor groups for ECD.
Advocate for investment by country partners in initiatives to improve the quality of center-based ECD interventions, especially in countries in which coverage has expanded rapidly (UNICEF HQ, ROs and COs). As noted above, rapid expansion of centers may result in less focus on quality. To ensure that new facilities are safe, healthy and equipped with appropriate materials, provide technical support for a parallel expansion of teacher training systems and monitoring and inspection systems. Use ELDS as a starting point for developing standardized monitoring tools and collecting consistent information about each center.
Advocate for increased access to holistic services that reach children ages 0 to 3 and their parents (UNICEF HQ, ROs and COs). Highlight the need to address gaps in services for the youngest children and encourage partners to expand services for parents of children under age 3. Evaluate and disseminate effective service models, including interventions in which parents and children participate together. Involve multiple sectors—including health, WASH, child protection and social protection—in promoting holistic ECD. Provide funding for effective interventions across the relevant sectors.
Planning, Management, Coordination and Partnerships
Prioritize development of results frameworks for holistic ECD (UNICEF HQ, ROs and COs). At all levels, specify and define measurable ECD outcomes. At the CO level, establish or refine logical frameworks for ECD that reflect activities across sectors. Measure progress toward the expected outcomes identified in these frameworks over time. Encourage UNICEF COs and country partners to use monitoring results to identify and plan for course corrections as needed in areas for which sufficient progress is not being made. Share well-specified results frameworks with all partners to guide activities and ensure that all are working on a common set of targeted outcomes.
Take steps to improve the quality and efficiency of reporting on specific investments by donors (UNICEF HQ). Work with donors to specify measureable expected outcomes and clarify reporting requirements at the start of each funding period. Provide consistent training on the measurement system and reporting expectations to funded ROs and COs that includes an overview of the measurement strategy, its purpose and goals and expected uses of the data; definitions of data elements; identification of appropriate data sources and measures; a system for reviewing data quality; and how to use the data for programme improvement.
Encourage distribution of funding for ECD more evenly across sections in UNICEF COs (UNICEF HQ). As an advocate for intersectoral approaches to ECD, COs serve as models for country-level intersectoral collaboration. Instead of concentrating ECD funding and interventions within a few CO sections/sectors, allocating funding for ECD more evenly may reinforce the importance of intersectoral coordination on ECD. Intersectoral committees of CO staff can be tasked with the responsibility of increasing coordination among sections on ECD policy and programming at the country level.
Human rights-based approach and Reaching the Less Reached and Disadvantaged
Allocate substantial resources to improving access to ECD interventions for the less reached and disadvantaged (UNICEF HQ, ROs and COs). Given that the majority of COs are not certain about funding for current and future ECD strategies and activities, initiatives focused on increasing access require reallocation of existing ECD funds or allocation of additional resources. Clear goals and indicators of progress, strong leadership and effective partnerships are needed to secure the resources required to address the need for reliable data to inform intervention targeting, development of strategies and approaches to increasing access and engagement and retention of children and families who are disadvantaged and may remain unreached by new strategies.
Develop a set of strategies to increase access to ECD interventions for disadvantaged and marginalized populations (UNICEF HQ). Identify and disseminate strategies to increase access, including a set of outreach approaches and incentive strategies. Identify successful models from other sectors to inform development of a set of evidence-based interventions for increasing access to ECD programming. Examples include the use of community volunteers to identify orphans at the village level and assist with their enrollment in ECD services. Develop and test creative incentive strategies. Consider strategies such as financial incentives for construction of ECD centers in areas with high concentrations of disadvantaged and marginalized populations, higher rates of compensation for ECD teachers to staff those centers and incentives targeted to parents to encourage enrollment.
Develop tools and data sources for monitoring access to ECD services among disadvantaged groups (UNICEF HQ, ROs and COs). Most case study countries did not monitor access to ECD for specific marginalized and disadvantaged groups. Provide technical support to country partners in establishing systems for collecting these data, such as levels of enrollment for orphans, very poor children and children from specific minority ethnic groups. Once data systems are in place, set targets to monitor progress.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Búsqueda