2002 Global: A Review of Evaluations of UNICEF's Education Activities 1994 - 2000
Author: Chapman, D. W. ; UNICEF NYHQ
The evaluations examined in this review were completed over the last six years. The underlying education projects typically had been underway 2-4 years at the time of the evaluation. Initial design of these projects occurred in the 1-2 years preceding their start. Hence, the evaluations reviewed here reflect thinking from 5-10 years ago about what interventions and strategies represent effective education development. In the rapidly evolving world of UNICEF, that is an eon in time. Over the last 5-10 years, UNICEF thinking about the focus of its education work and its strategies for accomplishing this work has developed enormously. Little of the current thinking can be expected to show up in a review of evaluations of those earlier projects. Nonetheless, findings from the work undertaken during this period can inform future education and evaluation work within UNICEF.
Purpose / Objective
The objectives of the review were to:
(1) identify the range of strategies used by UNICEF Country Offices to extend access and to strengthen the quality of education;
(2) suggest the extent to which UNICEF-supported activities (projects and programs) were successful in achieving their intended ends;
(3) identify implications and lessons from these projects and programs in the context of strategic planning and programming of UNICEF education work until 2015;
(4) provide an overview of challenges and key issues that remain, and offer specific recommendations for the future; and,
(5) assess the contribution of project and program evaluation in UNICEF education activities.
Overall, 411 studies from 93 countries (or regional offices) were made available for this study. To be included in this review, each study had to meet two criteria. First, there had to be a clear UNICEF involvement in the evaluated activity and/or in the evaluation itself. Second, the study had to be evaluative in nature and broadly comply with UNICEF's policies, procedures and methodologies. Studies that reported baseline data, situation analyses, and general descriptions of educational needs in a country were eliminated from further consideration. A preliminary screening of these studies identified 185 studies as meeting these criteria.
Interviews were conducted with specialists of the Education Section at UNICEF Headquarters. Findings were also considered in light of relevant research and international experience reported in the international development literature.
Key Findings and Conclusions
There is a wide range of views within UNICEF about how to best extend access, improve quality, and address the education needs of children. UNICEF's education activities, designed in the early and mid-90s, did not reflect a strong consensus about what interventions were most likely to promote wider education access or improve quality. In fact, an important strength of UNICEF's education work has been its ability and willingness to undertake such a variety of interventions, based in large part on the judgment of those in the field -- country staff, government officials, and local collaborators.
It should be noted that, given the experience of the last six years, a broader consensus about effective practice is now developing, a topic discussed more fully in the main body of the paper. Since the time frame in which the activities reviewed in this study were designed and implemented, UNICEF has progressively developed more coherent goals and strategies for its support and advocacy work. This change has been consistent with the whole-child and rights-based approach to development and in line with the Convention on the Rights of the Child. UNICEF-supported projects and programs in education still show a high degree of diversity in terms of strategies and implementation modalities on the ground which, in turn, reflect the different realities and choices across regions and countries. Diversity also results from the fact that UNICEF often attempts to innovate. However, it should be noted that, given the experience of the last six years, a broader consensus about effective practice is now developing.
Early Childhood Development:
Findings suggest that early childhood education can make an important positive contribution in the lives of children, but that success is not guaranteed if projects are not properly designed or implemented. Success depends on effective implementation of activities and the support of key stakeholders, particularly parents, teachers, and government officials. The evaluations provide some evidence that children who participate in early childhood education activities demonstrate higher levels of learning in primary grades 1-4, but the differences in achievement tend to be small. Teacher and parent reactions to these activities were mixed, supportive in some countries, not supportive in others.
Country Officers supported a wide variety of activities to reach excluded children reflecting the diverse contexts. Some approaches worked well, some did not.
- Stipend programs generally were effective in getting more children in school. One program that provided stipends to offset lost income to the family due to a child attending school instead of working was quite successful. Subsidies given directly to individual children in another country did not seem to improve school participation rates, but did appear to reduce absenteeism and lower dropout of students, once they were in school.
- Activities aimed at reaching out-of-school youth were generally regarded as successful. Sometimes, however, results were either mixed or hard to determine. In some countries, these activities encountered resistance from the community.
- Several of the projects were successful in raising community awareness about the value of enrolling children in school, the problems children face in gaining access to school, and the risks associated with child labor. However, increasing community awareness did not always translate into increased community action and, even when communities did participate more actively, initial increases in participation did not necessarily endure.
- Increased children's knowledge of risks (e.g., HIV/AIDS) does not always lead to the intended changes in behavior.
- The success and eventual impact of UNICEF-supported activities depends as much on the quality of the implementation as it does on the initial project design. Even well-designed projects fail if poorly implemented. However, it appeared that monitoring and follow-through in ensuring effective implementation was sometimes weak.
- Seemingly well-designed projects encountered unanticipated difficulty when they misjudged local factors that worked as disincentives. Similarly, some activities encountered unanticipated negative cross-impacts, in which solving one problem created other problems, some more severe than the initial problem being addressed.
Across the key areas of UNICEF education work, its activities to promote girls' education reflect the most consistent planning, the most uniform framework of action, and the most systematic evaluation efforts. UNICEF's multi-country African Girls' Education Initiative (AGEI) has provided a structure that has helped Country Offices clarify goals and objectives, select strategies, and monitor progress.
A recent desk review of UNICEF-supported girls' education activities provides a useful review and analysis of UNICEF's work in girls' education. Among the main findings were the following:
- Community participation is recognized in most interventions as a key to ensuring relevance and acceptability of girls' education. However, lasting impact on girls' participation is limited where local initiatives are not accompanied by attention to national policies in education and related sectors.
- The physical accessibility and safety of the school, its sense of psychosocial security, and the quality and relevance of its pedagogy are fundamental elements in determining whether and how any child, and especially girls, will participate.
- Teachers' attitudes and behavior are central in promoting girls' participation in education.
- Lowering the financial and personal cost of attending school is a key factor in promoting education. Programs that reduce the direct costs of school attendance (e.g. elimination of school fees) or target subsidies to girls (scholarship programs) seem to be effective.
While the evaluation of girls' education activities is arguably the most systematic and cumulative area of UNICEF education work and girls' education activities appear to have considerable impact, evaluation findings are, nonetheless, mixed. The main findings of the present review were that:
- While raising community awareness of the importance of girls' education was an important component of many of the activities, changing attitudes and behavior was not easy and not always successful. The impact of some activities was limited by the failure of the project designers to adequately anticipate and address the complexity of incentive systems that operate. For example, in one country, efforts to increase the proportion of female teachers encountered unexpected difficulty when women, once trained, did not want to be separated from their families or be assigned to rural areas. Community-participation activities in support of girls' education in another country encountered resistance when parents did not want to pay for children of other families.
- Efforts to improve girls' education sometimes had negative cross-impacts. For example, in one country where school fees were abolished, girls' enrollment increased. However, school budgets declined when Government did not compensate schools for those lost fees. The loss of resources at these schools threatened the quality of instruction.
- In some countries, design and implementation of activities suffered from lack of coherence and poor communication among partners. Several evaluations highlighted the need for stronger coordination among UNICEF, government and NGOs.
- Effective interventions are not necessarily less expensive. Some activities found to be effective had higher unit costs than less effective alternatives. As long as the increase in outputs is disproportionately greater than the increase in cost, such activities should be encouraged.
There was substantial diversity in the strategies that were undertaken to improve the quality of education. While a broader consensus is now forming, the range of interventions pursued over the last six years provided a useful opportunity to test a wide array of strategies.
- Teacher training had a positive impact on changing teachers' pedagogical practices, raising student achievement, and improving enrollment, though those impacts were not automatic or assured.
- Multi-grade teaching, in which one teacher teaches several grade levels in the same classroom, was frequently effective in raising student achievement.
- Student-centered teaching was a relatively popular intervention but the impacts on teaching practice and on student learning were mixed. It was frequently too complicated for teachers to implement effectively.
Restoring Education in Emergency Situations:
UNICEF's role in emergency situations has evolved from an emphasis on providing supplies to displaced persons to an emphasis on preserving and restoring basic structures of their formal education system and helping countries make the transition from emergency conditions to functional schools. Among the findings were that (1) UNICEF was generally effective in delivering intended inputs (supplies, materials), (2) UNICEF provided a longer-term continuity that was much-needed during the emergency, but that (3) UNICEF was somewhat weaker on contingency planning and disaster preparedness.
Country-wide, Multi-country Studies:
These evaluations were among the most useful for purposes of this review because they looked across wider sets of activities and emphasized synthesis. These studies found that stakeholders and participants usually held positive views of UNICEF activities, that the actual success of the activities in achieving their intended objectives was quite mixed, and that limited success in achieving those impacts often was due to factors external to activities themselves.
- In several countries, UNICEF activities have clear goals but lacked well-defined objectives or criteria for success. The evaluations of these activities reflected the lack of clarity about what was intended to be accomplished. For example, in one case, the Mid-Term Review concluded that the success of programs could not be determined because of lack of clear criteria for success, lack of initial baseline data, inadequate indicators, and poor measurement. In another country, ambitious project goals were not well aligned with the limited resources available to the project. The goals were not translated into realistic objectives that might have been more easily addressed.
- The confusion between UNICEF and Government (or NGO) partners about roles and responsibilities was frequently cited as a limitation on the overall effectiveness of UNICEF work.
Studies highlighted the effectiveness of school clusters and the need for UNICEF Country Offices to publicize their accomplishments more broadly within the countries.
A. Conclusions on UNICEF-supported Education Activities
Widespread positive regard for UNICEF work:
One of the most consistent findings across activities was that the individuals and governments involved in delivering, receiving, and otherwise supporting UNICEF field work believed that UNICEF projects were doing good things for their country and for them. Participants, counterparts, and other stakeholders express widespread positive regard for projects. They like what UNICEF does, believe it is important, and support the effort. That positive regard is not always supported by evidence that the activities are meeting their intended objectives, but the positive regard serves as one indicator of organizational success.
Governmental partner agencies and project participants liked working with UNICEF and held UNICEF-supported education projects in high regard. However, where projects did encounter difficulties during implementation, it was often attributed to problems in coordination within UNICEF or between UNICEF and Government. This was not necessarily due to personalities. More often, it was attributed to incompatibilities between personnel systems, accounting regulations, procurement procedures, and information flow.
Some of the UNICEF-supported projects and programs showed weaknesses in their design. In particular, the components of some activities seemed scattered and of dubious importance. These components may each have been included in response to the pressure of some interest group during the design phase, but distract from the overall coherence of the activity during implementation. At the same time, outcomes are sometimes over-promised. Seemingly effective activities appear to fail when they cannot deliver on over-stated promises. Activity designers may believe they need to promise over-stated outcomes to justify the requested investment. Actions taken to 'sell' a project during the activities' approval process may inadvertently raise the stakes higher than is reasonable, given actual project activities.
Undocumented design changes:
During implementation, project staff may introduce alterations in project design and intended outcomes that they believe are necessary accommodations to get past unanticipated hurdles or design oversights. These changes are often the product of subtle negotiations among UNICEF and country partners and result in trade-offs that are often undocumented. One risk is that, in their attention to initial objectives, evaluators sometimes fail to evaluate the activity that was really implemented.
Roles and relationships:
One of the threats to the success of UNICEF-supported education projects was the confusion that developed among project partners and key personnel about roles, relationships, lines of authority, and locus of responsibility. Projects failed when people in leadership positions did not do what they were supposed to do or did not do things other people thought they had agreed to do.
Differences in addressing access and quality:
In general, activities to extend access have had more success than those aimed at improving educational quality. Extending access often involves changing structures (e.g., building schools in new locations, lowering school fees, providing textbooks) aimed at changing school-going behavior through the creation of incentives and the reduction of barriers. Changing school quality often focuses more on getting people to behave differently through training and persuasion. Changing individuals' behavior through persuasion is generally the harder proposition.
Sustainability of results:
Evaluations gave little attention to the issue of sustainability of effects resulting from UNICEF-supported projects and programs. Some conclusions do, however, emerge from this review:
- Evaluations of community participation, teacher training, continuous assessment, and restoration of education in emergencies all observed that positive outcomes, once achieved, were difficult to sustain. This lack of sustainability often seemed to stem from inattention to incentives. Activities were generally designed to promote the best interest of the child without sufficient attention to the best interest of those who were expected to implement the activities.
- Findings suggest that more thought should be given to what design characteristics are most likely to promote sustainability. Some evaluations suggest that actions undertaken to facilitate implementation in the short-term (anchoring a project in the community) may, in the long run, work against sustainability. Sustainability comes from building project activities into a more durable infrastructure, such as the government bureaucracy.
Going to scale:
The goal of most development projects is that once they demonstrate their success on a small scale, similar activities will be developed at a larger-scale, perhaps even at the national level. This move, from pilot to full-scale implementation, is described as going-to-scale. Although the evaluations reviewed in this study do not give much attention to this dimension, there is evidence that the dynamics of going-to-scale are more complicated than is widely realized. Pilot projects often operate under highly advantaged conditions that can never be replicated on a wider scale. When a project goes to wider implementation in settings that are less advantaged, participants tend to be more average in their professional abilities, incentives tend to be less attractive, materials tend to be less available, and supervision tends to get stretched.
B. Conclusions on the Conduct of Evaluation
Emphasis on evaluation for mid-course correction at the project and sectoral program level: UNICEF evaluation practice is mainly geared toward mid-course corrections at the individual activity or sectoral program levels. The evaluations that were reviewed seldom made any explicit reference to how they fit into the Integrated Monitoring and Implementation Plan (IMEP) or, in a broader sense, into the overall Country Program. Evaluations seldom indicated how they were intended to connect to the UNICEF's Mid-Term Review or the Country Program Evaluation. It appears that evaluations, to a large extent, were designed and implemented in an ad-hoc manner and not as a tool for strategic Country Program management.
Confusion in selecting evaluation criteria:
Weaknesses related to the design of projects and programs (see above) lead to a lack of clarity concerning evaluation criteria. A number of the evaluations expressed confusion over what criteria were to be used in judging the success of an activity. In other cases, program designers failed to specify the criteria of project success, leaving it to subsequent activity managers or evaluators who may not have had as clear an understanding about what the activity was intended to accomplish.
The dominance of affect as an indicator of project success:
Of the evaluations reviewed, many were unable to document changes in behavior or performance. When that happened, there was a pervasive tendency to diminish the importance of those behavioral changes in favor of attitudinal data, typically showing that participants held favorable attitudes toward the activity. When evaluations did report attitude, the definition and measurement of attitude tended to be rather casual. Attitude measures in a number of the studies were little more than measures of generalized affect of questionable validity or meaning.
Limited data on outcomes and impact:
Attention to overall outcomes in terms of capacity building and impact of UNICEF-supported education activities on the lives of children seem to be underrepresented in the evaluations reviewed in this study. Relatively few evaluations actually assess the extent to which education activities achieve their broader goals or objectives. A number of evaluations report disappointing or no discernable outcomes or impacts.
The attribution of impact to UNICEF-supported activities is difficult and often impossible. Improved access and quality of education are often the result of a variety of factors that are not directly related to UNICEF support, including implementation decisions of local partners and the macro-economic and social environment in which the project operates. Factors such as economic growth, poverty reduction, monetary and fiscal policies, and government spending can have an enormous impact on access to and quality of education and provide a more or less favorable environment to individual project activities.
Issues not fully addressed in evaluations: With notable exceptions, little attention was given to issues of cost, efficiency, sustainability, or going-to-scale.
- Cost: Inadequate funding was cited in several cases as an important factor contributing to implementation problems. Beyond that level of observation, cost was seldom reported or discussed. Few evaluations reported aggregate cost, unit costs, or opportunity costs associated with the interventions being evaluated. Few of the studies included a cost-benefit analysis. In only 1-2 studies, was the distinction made or any attention given to the interplay between investment costs and capital costs. Research and international experience suggest that failure to attend to recurrent costs is a frequent reason that projects are not sustained. External funds are invested in training systems, buildings, and initial production of materials without adequate regard for the recurrent costs associated with keeping the training going, the buildings clean, and the printing presses running after the external monies end.
- Efficiency: The efficiency of UNICEF-sponsored education projects was virtually never discussed in the evaluations that were reviewed. This is probably due to the absence of cost data (a necessary component in the consideration of efficiency).
- Sustainability: While several of the activities had sustainability as a goal, it was seldom addressed in any of the evaluations. This is largely because (a) most of the evaluations were conducted early in the activity, when the evaluation questions were concerned more with level of implementation and (b) a persisting confusion about what aspects of an intervention should be sustained (e.g., specific activities, organizational capacities, or positive impacts). One of the threats to sustainability was normal staff turnover within the Country Office and government ministries working with the activity. Finally, due to the frequent failure to clearly document (or even consider) recurrent costs, many projects lacked the necessary financial support needed to continue.
- Going-to-scale: Given the difficulty many seemingly successful pilot projects have in replicating their success on a wider scale, attention to ensuring the preconditions for going-to-scale is widely considered to be an important element in the evaluation of projects that are intended for larger delivery. It was surprising, then, that going-to-scale was not widely addressed in the evaluations reviewed for this study.
Linking evaluation findings to future agendas:
Many of the key development challenges of the next decade will be different from the past decade. Across the UNICEF activities reviewed in this study, there was little attention to emerging issues such as HIV/AIDS, the impact of new technologies, globalization, and the complex issues arising from decentralization. Widespread attention to these issues has arisen more recently.
Coping with new challenges:
Over the last decade, there has been important elaboration of needs and issues, new focus, and shifts in the strategic thinking about UNICEF's role in education development. It is widely regarded as one of the most effective international agencies working in the area of education development. Still, as world conditions change, the struggle to remain relevant and effective continues.
Consolidation of achievements:
UNICEF will need to balance its attention to new issues with its commitment to sustaining and consolidating the gains it has already made. Many of the education issues of the last decade will continue to be prominent in the next ten years. A critical issue over the next decade will be countering the risk of eroding the gains that have already been made in education access and quality, even as new issues are added to the agenda. It will be important to ensure that existing programs and systems are working before shifting investments to new agenda. This effort is particularly threatened by issue fatigue and the backlash effect. Issue fatigue occurs when the novelty of a problem wears off and attention shifts to new and more intriguing issues. The backlash effect occurs when some constituencies feel that certain issues are getting too much attention and that the concentration of concern for those issues is allowing inequities to emerge in other areas.
Design and implementation of education activities:
UNICEF-supported education projects and programs are usually well-considered by governments, NGOs and other partner organizations. There is sometimes room for improvement, however, in how projects and programs are designed and implemented. The role of each partner involved in these activities often needs to be clarified. Conditions for sustainability and possible scaling-up need to be more critically assessed. Greater clarity is often required about expected outcomes, particularly with respect to national capacity building and impacts on the lives of children. Individual project and program activities should be clearly related to situation analyses and overall Country Programs in a human rights perspective and in terms of results-based management.
Evaluation challenges for the future:
There is a need to strengthen the role of evaluation in the Country Program cycle and in overall strategic planning and programming. The evaluation effort could be strengthened by more attention to outcomes, behavioral changes, cost, sustainability, and the conditions necessary for going-to-scale. Additionally, evaluation could benefit from a shift away from a focus on individual activities toward more of a focus at programmatic and policy levels.
Full report in PDF
PDF files require Acrobat Reader.