Base de données d'évaluation

Evaluation report

2004 Global: Changing Lives of Girls: Evaluation of the African Girls' Education Initiative



Author: Chapman, D., N’gra-zan Coulibaly, C.N., Coyne, B., Emert, H., Diallo, A., Hickson, K., Issayas, S., Kyeyune, R., Lexow, J., Lokkesmoe, K., Osei, J., Back, L.; UNICEF NYHQ

Executive summary

Background and Context

The African Girls’ Education Initiative (AGEI) has been the centrepiece of UNICEF’s effort to promote girls’ education.  Started in 1994 with funding from the Canadian International Development Agency (CIDA), it was continued in 1996 with funding from Norway.  The initiative was grounded in the premise that by targeting girls, the programme would reach a major proportion of the population of children that has been denied access to education.  Over the course of the program, the Norwegian Ministry of Foreign Affairs provided US$ 45,097,576 to support AGEI activities in a total of 34 countries, two Regional Offices and UNICEF Headquarters.  Specifically, during AGEI-1 (1996-2000), Norway provided $22,722,072 that was used to support work in 18 countries.  During AGEI-2 (2001- April 2004), Norway provided another $22,375,504, used to support continued work in the original 18 countries and initiate new work in an additional 16 countries.

Purpose and Objectives

This external evaluation was intended to assess the effectiveness of AGEI activities on increasing girls’ access, persistence, achievement, and overall treatment in school and the extent that AGEI fostered and promoted changes in national and organizational policy in ways that support the education of girls.  The evaluation was intended to yield information useful in (a) designing and implementing girls’ education programmes in other countries and regions, (b) strengthening the capacity of UNICEF and its partners in the design and delivery of girls’ education programmes; and (c) contributing to the achievement of Education for All (EFA) goals and Millennium Development Goals (MDG). 

Methodology

The evaluation examined six dimensions of the AGEI: (a) relevance; (b) role, design and focus; (c) effectiveness, (d) efficiency, (e) sustainability, and (f) adequacy of programme support.  Evaluation data were collected through: (a) a comprehensive review of project documents, (b) six country studies illustrating different types of context and outcomes and more or less successful experiences (Botswana, Burkina Faso, Eritrea, Ghana, Guinea, Uganda), and (c) interviews with key UNICEF personnel involved in the design and implementation of AGEI.  To the extent possible, information was validated and cross-checked with other sources. 

The documents reviewed included funding proposals, yearly technical reports (YTRs), country level evaluations, and regional and global consolidated reports.  Special attention was given to examining change between beginning and end of each AGEI phase. The annual consolidated reports and the regional reports were found to be particularly helpful.  The YTRs and evaluations were of somewhat more limited usefulness, due to uneven quality, missing data and, frequently a lack of analysis and reflection.

Findings and conclusions

Relevance:  This evaluation found that AGEI was a relevant programme.  Girls lag behind boys in school access, retention, and learning achievement across much of Africa, disparities that are recognized by governments and the larger international community as serious constraints on the social, political, and economic development of the continent.  AGEI activities were directly aimed at reducing these constraints and improving gender equity.

AGEI was essentially based on a gender equity approach, which aimed at reducing gender discrimination and promoting social/economic self reliance of women through policies and programmes that increased access to basic education and training of girls.  AGEI did not reach the level of a gender equality (empowerment) approach, which extends beyond access as well as quality issues surrounding basic education. While the focus on equity was appropriate for AGEI, given the social context at the time AGEI was launched, efforts being supported through projects such as those associated with the UNGEI would benefit from broadening the focus to give more attention to issues of equality.

Another aspect of relevance was that AGEI played a tremendous role in shaping UNICEF's education agenda since the late 1990s.  It contributed to the definition of UNICEF's organizational priority of girls' education and to clarifying the division of tasks with UNESCO. A notable step in the process has been UNICEF's role as lead agency in the United Nations Girls’ Education Initiative (UNGEI).

Role, focus and design:  The evaluation found that the AGEI played a useful role in raising the prominence of girls’ education as a gender equity issue in many countries.  It contributed to sparking a sustained discussion about the importance of girls’ education and the best ways to encourage greater female school access, persistence, achievement across many countries.  Governments in several AGEI countries had finalized (or were close to finalizing) national policies aimed at encouraging the educational access and achievement of girls.  While these policies were the result of governments’ own commitment and the encouragement of the wider international community, UNICEF was often singled out by government and development partners as a particularly strong voice in encouraging government action.

While there was wide agreement across governments and development partners about the main factors constraining girls’ participation in schooling, there was considerably less agreement about what interventions were most appropriate in addressing these constraints.  In practice, a wide variety of relatively discreet activities to promote girls’ education were undertaken under AGEI ranging from small scale models (e.g. girls’ clubs) to community action and policy development.  This wide focus seemed appropriate, given the uncertainty about the relative effectiveness of alternative strategies.

AGEI-1 suffered from weak programme design.  The logic linking particular interventions to their intended consequences was not always clear.  Proposals were sometimes over-promised, especially in AGEI-1, offering overly ambitious claims about what they could accomplish.  Reports tend to focus on inputs and outputs, on one hand, and on trends in girls’ education (e.g. GER and NER), on the other. There was still relatively little attention at the level of outcomes.  In some cases, there was an assumed cause-effect relationship between AGEI outputs and positive trends, which was not substantiated by clear evidence. 

Government and UNICEF often had not conducted formal evaluation of seemingly effective activities in ways that would have provided objective evidence of their success.  They frequently expressed strong beliefs about the effectiveness of particular interventions, but lacked the systematic documentation of those outcomes and attributable impacts.  While evaluation of AGEI activities was generally weak, there were some good examples of strong evaluation. These included Breakthrough to Literacy (Uganda) and CHILDSCOPE (Ghana).

Effectiveness:  UNICEF country staff and government collaborators give little attention to systematically assessing the outcomes of AGEI activities and strategies. As a result, claims about the relative effectiveness of the interventions implemented through AGEI were limited.  It was difficult to determine whether programme objectives were met due to weak design of activities, a lack of valid baseline data, and inadequate performance monitoring and evaluation. 

For the sake of convenience, the evaluation clustered activities within a life-cycle approach. Acivities in early childhood education emphasized a comprehensive and integrated approach rather than the narrower focus on pre-school education.  Relatively successful, albeit isolated, projects were developed in Burkina Faso, Cameroon, Chad, Kenya, Liberia, Togo and Uganda.

Girls’ enrolment, retention and learning achievement in formal basic education were promoted through such activities as feeder or satellite schools, girls’ hostels, school feeding programmes, provision of construction materials and supplies for infrastructure, curriculum development, instructional materials development and distribution, girl-to-girl tutoring, and teachers’ training.  In most initiatives, some degree of community participation was sought for the construction of schools, for the promotion of girls’ enrolment, and the management of schools. Virtually all AGEI countries were involved in this type of experience.

Non-formal education and “second chance” schools literacy and vocational training focused on children who had never been to school or who had dropped out of school.  Their success seemed to depend on the quality of the teachers, the status given to this form of education, and the opportunities available for students to transition into either formal schooling or employment. AGEI supported non-formal education in Burkina Faso, Guinea, Mauritania, Niger, Nigeria, Sudan and Uganda.

Some activities worked better than others. Strategies that appeared to UNICEF field staff to be more successful included the provision of water and sanitation (e.g., Benin, Burkina Faso, Burundi, Comoros, Ethiopia, Gambia, Guinea Bissau, Kenya, Mauritania, Mali, Niger, Nigeria, Somalia, Tanzania, Uganda), food rations for girls (e.g., Benin, Burkina Faso, Eritrea, Gambia, Guinea Bissau, Guinea, Mali, Senegal, Togo), early childhood programmes (e.g., Burkina Faso, Uganda, Togo, Cameroon), girls’ education clubs (e.g., Uganda, Ghana), girl-to-girl tutoring (Benin, Guinea), Breakthrough to Literacy (e.g., Uganda), second-chance school options (e.g., Guinea, Uganda) and, in some locations, community school construction (e.g., Burkina Faso, Eritrea, Guinea). 

Some AGEI activities were judged by UNICEF field staff to be less successful than had been anticipated.  For example, girls’ education clubs (e.g., Botswana), construction of hostels (e.g., Eritrea, Botswana), playground construction (e.g., Botswana), and one-shot materials distribution (various).  AGEI sometimes encountered unanticipated consequences, both positive and negative.  Several countries reported community backlash at special treatment of girls.

Similar interventions and strategies may be relatively successful in one country and less promising in another. Across countries, several factors determine whether or not specific interventions were successful.  The most important factors were cultural context and the prevalence of gender stereotypes.

One constraint on effectiveness was that the AGEI at its various levels was principally viewed as a female initiative that resulted in not gaining the full participation, support and involvement of men and boys. A critical gender analytical framework with adequate tools and methods were not developed in any consistent way.  While there were several efforts undertaken that addressed gender issues, many of them were implemented on a relatively small scale and were not part of a systematic or systemic approach aiming at the realization of girls’ and women’s rights in a comprehensive fashion.

Efficiency:  Efficiency is understood as the relationship of cost and effectiveness.  Determining efficiency requires good data on both costs and outcomes.  Few government or UNICEF offices had clear or reliable information on the costs of the activities they believed had been successful.  Similarly, few had objective evidence of outcomes of AGEI activities.  Lacking objective data on outcomes or cost, the ability to determine efficiency was limited.  The evaluation found UNICEF country and government staff frequently had an inadequate understanding of the difference between costs and expenditure, investment and recurrent costs, and direct and opportunity costs.  That said, some of the more effective activities were also the most expensive. Examples include COPE and ABEK in Uganda and CHILDSCOPE in Ghana. This has implications for the chances of sustainability and replicability of experiences as well as their mainstreaming.  There was a paradox related to community participation.  Poor communities were often expected to contribute their labour and local materials to the building of schools and sometimes also to the payment of salaries of teachers, while affluent communities (e.g. in urban areas) had all school costs covered by government.

Sustainability, replicability and mainstreaming:  The likelihood of governments or communities sustaining activities initiated by AGEI beyond external support appeared to be low.  In most cases, AGEI adopted a classical project approach providing major external support for investments (mainly infrastructure, training, equipment) against a “soft” commitment from governments and communities to cover recurrent costs and future investments (amortization of investments, operation and maintenance of infrastructure, teachers’ salaries).  In practice, many of these commitments are not likely to be honoured, due to a lack of resources and a lack of commitment to girls’ education and women’s empowerment. 

Three other findings of note were:  firstly, the use of a “demonstration project” approach to promote mainstreaming seldom led to wider adoption or implementation of the activity without continued external funding.  Secondly, there was sometimes a lack of linkage between government plans and successful field activities.  This was seen in BTL in Uganda and CHILDSCOPE in Ghana.  One reason was that neither central government nor communities were able to use evidence of activity success to shape national funding streams in ways that would sustain the activity.  Finally, the replicability and mainstreaming of activities aimed at girls’ education was influenced by the changing nature of development funding.  In the move to sector-wide planning, sector-investment plans, and multi-donor budget support, some government and UNICEF personnel are concerned that it will be harder to maintain specific agendas in support of girls’ education.

Adequacy of programme support: The framework agreements of both AGEI-1 and AGEI-2 were relatively general and overall programme guidance was open allowing for creative proposals from the field. At times, there was ambiguity across levels of UNICEF about who (and what level) was responsible for the design of AGEI activities.  Some Education Officers in the field expected Headquarters and Regional Offices to provide more guidance than they actually received regarding effective practices. 

Lessons Learned

While the AGEI will be completed at end of 2003/beginning of 2004, results of this evaluation offers lessons that may be useful in further programmatic efforts on behalf of girls’ education, such as those supported through the framework of and in partnership with the UNGEI. 

Gender equity vs. gender equality:  The emphasis on girls’ access, retention and achievement in education is necessary but not sufficient.  Future large-scale girls’ education initiatives need to balance improving gender equity in school access, persistence, and achievement with promoting equality in female participation in leadership and decision making roles throughout the education structure. 
Influencing the discourse at the policy level:  The AGEI demonstrated that the strategic allocation of relatively small amounts of money across a large number of countries to support relatively modest efforts can do much to raise the prominence of target issues and shape policy discussion at the national level.  This approach is most valid when combined with systematic efforts to assess the relative success of the various strategies being tested.  AGEI was relatively weak in that assessment, a point discussed later.

Nonetheless, AGEI did much to change the shape of policy discourse across sub-Saharan Africa in ways that helped move the issue of girls’ education from relatively obscurity ten years ago to one of the top education priorities of many countries (and of UNICEF itself) today.  This accomplishment is particularly impressive, given the relatively small amount of funding actually available to each country through AGEI.

Managing for results:  In AGEI, few country offices attended to whether the desired consequences of activities actually occurred.  Future efforts similar to AGEI would benefit from more attention to outcomes. 

Developing realistic estimates of likely outcomes results-based framework:  Interventions sometimes appeared to fail not because they were inappropriate, but because they could not achieve inappropriately high goals. 

Clarifying criteria of success:  Experience in AGEI highlighted the recurring dilemma UNICEF country staff and government counterparts faced in deciding when an activity that is not yielding results should be discontinued or just given more time to demonstrate its worth.  AGEI provided information useful to future projects about establishing realistic goals and objectives and establishing clearer criteria for when activities should be continued or terminated.

The efficacy of demonstration projects:  A widely used strategy for promoting education reform is to demonstrate the effectiveness of an intervention on a small scale in a way that is highly visible to a wider audience, thereby creating a local demand for wider implementation of the intervention.  AGEI showed that the clear demonstration of the effectiveness of an activity is not enough, by itself, to spark adoption on a wider scale.  In AGEI when countries used a “demonstration project approach”, few, if any, of the interventions composing those demonstrations were adopted and replicated without external donor assistance.

The role of description in replication:  Successful replication of effective activity depends on having a clear description of the characteristics of the intervention and its programme of use.  YTRs were uneven in this regard, though the majority fell short on description.  Experience in AGEI highlighted the need for better project description as part of project monitoring and evaluation.

Over-reporting; under-informing:  In future UNICEF projects of this type, country reporting requirements could be reduced significantly even as useable information could be increased, by a rethinking of the reporting mechanisms.

Rethink monitoring and evaluation mechanisms:  Monitoring and evaluation of AGEI activities was weak.  The strong personal beliefs held by many Government and UNICEF staff about the effectiveness of AGEI activities were not necessarily grounded in objective evidence.  Informal evidence suggests that a number of strategies employed in AGEI were effective in promoting girls’ access, retention, or achievement.  However, in many cases this was based on individual observations, word-of-mouth, and relatively informal evidence.  Formal evidence of the effectiveness of those practices is largely lacking.  Few Country Offices had actually undertaken formal evaluation of those components in ways that would have provided objective evidence of their success.  This represents an unfortunate loss of information for the field and an important missed opportunity for UNICEF.

Attention to cost and aspects related to efficiency:  The lack of cost data in AGEI constrained sustainability and mainstreaming of seemingly effective practices.  Future initiatives of this type need to collect more complete cost data, including direct and indirect costs and costs at the individual, community, district, and national level. Sustainability and mainstreaming require that governments understand the financial consequences of the interventions to which they are being asked to commit.  Another lesson from AGEI is that effective intervention may be expensive. Where expenditure data were available, they indicate that effective strategies tended to be expensive.

Attention to sustainability, replicability and mainstreaming:  Future initiatives similar to AGEI would benefit from a closer linkage between central government actions in support of girls’ education and the wide variety of innovative strategies being promoted at the community and school levels.  There were examples in AGEI where successful strategies at the community and school level did not have much visibility at the national level or had visibility and, for other reasons, were not adopted by government for wider dissemination.

Complementarity of formal and non-formal basic education:  AGEI demonstrated successful “second-chance” strategies for reaching youth who had already dropped out of school and older youth who had missed early opportunities for schooling.  At the same time, the AGEI experience highlighted the problem some governments have in knowing how to effectively integrate non-formal and formal structures.  Even though COPE was widely recognized as effective, government was reluctant to extend the model using government funds.

Overall Impact of AGEI

The changes that have occurred in girls’ participation, retention, and achievement over the course of the AGEI programme are the result of many factors in addition to AGEI both within and outside the education system.  Consequently, increases in girls’ access cannot be attributed specifically to AGEI activities.  Credit must be shared with national governments and other development partners who have also been promoting girls’ education.  It can be concluded, however, that AGEI was a significant force in making a widespread and meaningful contribution to improving girls’ education across Sub-Saharan Africa using a multi-country approach. 

Note: Summaries of country case studies are included in the main report.  Full text reports of these case studies in PDF format can be viewed by clicking on the links below.

Country Case Study: Botswana

Country Case Study: Burkina Faso

Country Case Study: Eritrea

Country Case Study: Ghana

Country Case Study: Guinea

Country Case Study: Uganda

 

Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2004

Region:
Global

Country:
Inter-regional

Type:
Evaluation

Theme: 
Education - Girls

Partners:

PIDB:

Follow Up:

Language:
English

Sequence Number:
2004/810

Recherche