Author: David J Clarke
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report.”
The programme design was based on UNICEF’s prior experience in the education sector in Myanmar. The core components of Early Childhood Development (ECD), Child Friendly Schools (CFS) and Life Skills Education (LSE) had all been part of UNICEF’s education programmes and had already been piloted and refined in the local context. Thus, in essence, it involved a scaling up of interventions that had already been developed and were familiar to the Ministry of Education.
CFS, which originated as a concept in the late 1990s, had already been adopted as a flagship strategy in the EFA National Action Plan 2003-2015 to improve access to the quality of basic education. LSE had been introduced through the Schools-Based Healthy Living and HIV/AIDS Prevention Education (SHAPE) project in 1998, which was focused on HIV prevention in both formal and non-formal education. SHAPE enabled the introduction of LSE as part of the core curriculum in primary education. The non-formal education component was adapted for the Extended and Continuous Education and Learning (EXCEL) project. Support for ECD had been initiated in 2000 in the areas of parenting programmes and pre-school education.
The main purpose of this evaluation of the UNICEF Education Programme in Myanmar 2006-2010 is to assess its performance in terms of relevance, efficiency, effectiveness, and suggest any needed modifications for further programming. The evaluation report attempts to identify key lessons learned and good practices documented through delivering the project interventions, as well as to suggest future directions that would contribute to the design and development of a second phase of multi-donor support from 2011.
The methods included a thorough document review of the UNICEF Education Programme. Based on this and the terms of reference, (See Annex 1), a conceptual framework was developed with accompanying methods involving qualitative research including for focus-group discussions and key informant interviews. A rapid situation analysis of the education sector was conducted to inform the review.
Meetings and focus group discussions were held in-country with key counterparts including UNICEF staff in Yangon and in the field; key counterparts and implementing partners (e.g. Ministry of Education and local NGOs) and various stakeholders involved in the programme (See Annex 2, for a list of persons consulted and Annex 3 for a bibliography). Discussions were held with donor representatives,
Field visits were made to a selected sample of target beneficiaries, to observe, and conduct interviews/in-depth interviews/focus group discussions with stakeholders at township and school levels (e.g. teachers, students, and members of parent teacher association). The visits were conducted in Mon State, Ayaryarwaddy and Yangon Divisions.
Findings and Conclusions:
Key findings of the evaluation
The findings in the evaluation are organized in terms of the following criteria: relevance; coverage; effectiveness; efficiency and sustainability. These issues are addressed in separate sections of the main report.
4.1 The design of the Education Programme was highly relevant to addressing the key issues in Basic Education
The main strengths are considered to be as follows. The programme:
• Was aligned with Government Policy on Basic Education and highly valued by MoE;
• Addressed key barriers to equity in ECD and primary education;
• Addressed the poor quality of primary education;
• Attempted to builds capacity in some key functional areas of the Ministry of Education e.g the EMIS; and
• Developed models of good practice which can be taken to scale (e.g. ECD, EXCEL).
4.2 The programme had a number of critically important design limitations which need to be addressed in any subsequent phase of support for education
These include the following:
• Inclusion of programmes targeted at 0-3 year olds. These are not normally part of an education sector policy framework or programme. While interventions have very robust child development justification, they must be considered as on outlier in the overall education programme strategy and more properly should be within a holistic mutli-sectoral response to supporting child development under the direction of the Ministry of Social Welfare;
• Implemented in a context without a comprehensive sector plan. The Education Sector lacks a clearly defined and costed 3 year-strategic plan which could have been developed with the participation of the MoE;
• Inadequate M&E arrangements. While there is strong importance given to M&E in the original proposal, there was a lack of a defined and consistent set of programme indicators which would be used to track progress and ensure accountability. Multiple steps were taken to address M&E during implementation. However, the arrangements put in place were too complex for UNICEF to manage and as a result a great deal of the data obtained was not analysed. There was an overreliance on a large-scale survey to measure changes in school practices without any triangulation using qualitative research methods.
• Lack of an exit strategy. There is no clear exit strategy to ensure that dependence on UNICEF provided school supplies does not emerge in schools and in poor households and communities.
4.3 The purpose of the programme and the 3 main outputs were achieved: however, sustainability is major issue.
• The programme increased equitable access to quality early childhood education and primary education with extended opportunities for all children, especially those in disadvantaged and hard to reach communities. This is reflected in improved ECE enrolment and primary school survival rates in targeted Townships;
• Some school improvements were achieved through Child Friendly Schools implementation;
• Access to non-formal education, including life-skills based continuing education (EXCEL) and non-formal primary education (NFPE) was increased. Models for both EXCEL and NFPE were developed.
4.4 The ECD programme achieved multiple positive impacts on both the demand and supply sides of early childhood education (ECE)
• Increased community awareness of and demand for ECE services;
• Increased participation in parental education;
• Increased participation rates in ECD/ECE. 125,000 children benefitted from ECD in 109 Townships;
• Increase in school-based ECE provision;
• Improved social and communication (life skills) and school readiness of children participating in the ECE programme;
• Increase in equity in ECE through the development and use of learning materials in minority languages, promotion of inclusive ECE and increased participation of children with disabilities;
• Increased participation of poor and disadvantaged children. It is estimated by UNICEF that 70% of children attending the ECD programme were from poor households;
• Capacity in delivering effective ECE has been strengthened, including though the setting of national Minimum Standards;
• Increased space for civil society participation through support for NGO implementation of ECD/ECE;
4.5 Delivery of quality and equitable ECD/ECE still faces important challenges
• Lack of clear MoE policy on ECE and ownership;
• ECD financing is dependent on community/household contributions;
• ECE coverage rates are still low, particularly of children in remote and disadvantaged rural areas. Much depends on NGO availability;
• Effective targeting of the most disadvantaged, vulnerable and at risk children;
• ECE teacher attrition rates are high; and
• Need for better disaggregated data on ECD (by gender/school/Township) for planning and M&E.
4.6 Programme support for primary education improved the school intake and survival rate in most schools
• Endline data showed that 62% of primary schools improved their survival rates, while 30% saw a decline. The survival rate went from 62% at baseline to 76% at endline;
• The Apparent Intake Rate (AIR) showed a significant improvement at endline (from 0.62 to 0.76)
4.7 There was no significant change in repletion rates.
• Most schools have an automatic promotion policy in Grades 1-3;
• Grade repetition is still being practised in some schools.
4.8 Capacity strengthening has taken place at Township and school levels in primary education, but needs to be more systematic and evidence-based
• Pilot programmes such as Language Enrichment Programme (LEP) and the Township Education Management Information System are promising developments but need to be rigorously evaluated before any scaling up is planned;
• In-service training has been delivered to in CFS; School Improvement Planning; Child Centred Approach (CCA);
• The quality of pre-service teacher training has been addressed only in the area of life skills education;
• Effective school management remains a challenge despite the introduction of school improvement planning, School Management Information System (SMIS) and CFS;
• No systematic capacity building needs assessment has been undertaken to improve the quality of basic education service delivery;
4.9 School environments have been improved, but many schools remain inadequate for effective teaching and learning
• School sanitation has improved through the construction of 600 latrines, the provision of potable water and resources to repair basic infrastructure (roofs in particular);
• New models of primary school construction have been piloted in Cyclone Nargis-affected areas. These schools can also function as cyclone shelters and offer a higher standard of classroom and school environment than is currently the provision in most schools;
• School infrastructure can be poor. Many schools only provide a single hall type of construction with no partitions in which all classes are taught simultaneously in the same classroom structure. This practice facilitates multi-grade teaching if teachers are absent or lacking, but is not conducive to supporting good quality teaching. Some classrooms are very dark;
• Regular school maintenance is a key issue;
• There are many multi-grade classes in rural schools. Multi-grade teaching needs to be strengthened through appropriate teacher training.
• There is a need for MoE to rethink how it provides funds for school infrastructure including the critical question of how much it contributes.
4.10 The provision of essential school supplies has supported greater equity in primary education service delivery, but targeting of additional support for the most disadvantaged children is still required
• Poverty is deeply entrenched and school levies are pervasive. The school supplies are not enough to relieve the total cost burden for poor families. It may amount to only 10%-20% of the burden that is reduced. The supplies need to be continued and possibly enhanced in terms of better targeting of exercise books and pencils to poor children;
• There is a need to explore the need for targeted school feeding and nutrition;
• Equity needs to be ensured between schools under the responsibility of the MoE and Monastic schools;
• Greater support is in the early years of learning required for speakers of languages other than Myanmar in literacy skills acquisition;
4.18 Programme interventions have had limited impact on the quality of teaching and learning
• Current interventions are having an effect on individual schools though only a limited impact on the system. Research is needed on why some schools are better able to adopt CFS principles than others;
• The wide variation between the performance of schools within a township indicates that there is a need to strengthen educational management particularly at township and school level to support the progressive adoption of CFS principles in the classroom;
• The construction of many schools in hallway arrangements militates against effective teaching and learning;
• No correlation was found between CFS interventions and learning achievement. This implies a revisiting of the CFS approach with a much stronger emphasis on school/teacher effectiveness;
• Many children experience learning difficulties at school. Mathematics and English are the subjects that were considered to be most difficult to teach and to learn. There is a need to explore how to improve teaching in learning in key subjects.
• CFS is too complex for some teachers and they find it difficult to apply it in practice. There are some large classes (40+). Classroom management is a challenge and noise levels can be high.
• In some schools there is the persistent heavy use of choral repetition and whole class activities;
• Some teachers are resistant to change;
• Teachers are not used to doing lesson plans; they prefer to read the textbook.
• SIPs tend to be narrowly focus on infrastructure, water, fencing and furniture and less concerned with school effectiveness and the quality of education in the classroom.
4.12 Community participation in primary education remains limited in scope
• Improving school-community relationships as envisaged by CFS is a low priority for most schools;
4.13 Learning outcomes were improved but they remain very low in literacy and numeracy
• Improvements were made in many schools in Myanmar language learning and literacy skills acquisition;
• Attainment levels in primary level Myanmar language and mathematics are very low;
4.18 Support for Life Skills Education in primary education has been effective
• The LSE sub-project has had a national impact at primary level through curriculum development and teacher training.
• LSE seems to be liked by students. Students are benefiting from the primary school LSE curriculum in a number of ways. It helps increase knowledge and skills that are health-related; strengthens communication skills, critical thinking and problem solving; increases cooperative and interpersonal behaviours and reduces bullying. Knowledge acquired is shared with parents. Teachers have learned how to apply child-centred teaching methods, improved their knowledge on health issues and strengthened their own life skills;
4.18 Teacher education and support in life skills education needs further strengthening
• The main challenge is ensuring that teachers are skilled and motivated to teach the subject as intended;
• Some teachers do not follow the steps in the Teachers’ Guides. They may use more traditional teacher-centred methods instead. As a consequence students may not be able to practise the skills in the classroom. They lack the skills to use visual aids effectively. They have difficulty with sensitive topics such as SRH and sexuality. They may not give LSE a high priority, even though it is a core subject. In Education Colleges, the approach that is contained in the Teachers Guides is not used.
• There is no specific LSE teacher training manual or materials. There is a lack of a specialized department for life skills education. The Teacher Training Colleges are weak at sharing their experiences with others.
• There is a need for further investments in teacher education, supervision and support for LSE within a broader framework for support to improve the quality of teacher education.
4.16 The EXCEL programme is innovative and a very promising form of continuing education.
• EXCEL manages to reach and retain one of the hardest groups to reach: out of school adolescents.
• The effects of the programme on the lives of participating children are multiple. They include increases in knowledge, self-esteem, psycho-social skills and connectedness. The programme is a blend of Continuing Education (CE), LSE and social protection.
• There is strong demand for the programme to be lengthened in duration and strengthened in terms of CE and vocational training.
• Multiple community impacts are reported, including a strengthening of social capital at village level.
4.17 The EXCEL Programme needs further development to meet expressed needs of the target group
• Demand for EXCEL exceeds supply. There are many children out of school;
• The programme content needs to be broader and include more Continuing Education and links with local vocational training opportunities;
• Some children are hungry and would benefit from targeted nutrition interventions.
4.18 The programme outcomes provide a platform for policy dialogue and strong foundation for a second phase of support to ECE and Basic Education
• It is critically important that the results of this programme, strengths and limitations be discussed with senior MoE officials as well as key policy issues concerned with equity, quality and efficiency in the basic education sub-sector.
4.19 Sustainability is in very much in question
• There is some evidence of emerging dependence of UNICEF provided supplies. School staff and parents do not wish to lose the current provision.
The following strategic recommendations are made:
• Evaluate pilot interventions such as LEP and TEMIS as a matter of urgency. Develop a scaling up or revision strategy in line with the findings;
• Support MoE in developing a costed and detailed strategic plan for UBE/UPC 2012- 2015;
• Develop an exit strategy for existing support to CFS schools, especially in the field of supplies and focus more strongly on sustainability;
• Continue to strengthen existing interventions in to address equity and quality in primary Education. Undertake assessments to provide better understandings of equity, quality and capacity issues;
• Systematically review the approach to CFS implementation in the light of the endline findings;
• Undertake evidence-based policy dialogue with MoE on ECD (3-5) and NFE to develop a more comprehensive sector policy framework and review the findings of this evaluation;
• Focus more on strengthening primary school effectiveness and improving learning outcomes (especially literacy and numeracy skills);
• Take a strategic approach in partnership with JICA to strengthen the quality of primary teacher pre-service education;
• Investigate ways of improving targeted interventions in ECD and primary education to better cater for the needs of the poorest families;
• Consider making further developments to the EXCEL programme to increase duration and CE/vocational skills development; and
• Develop a detailed and technically sound M&E plan as part of the next phase proposal;
• Improve the effectiveness and inclusiveness of donor coordination mechanisms for any future programme phase of support.
PDF files require Acrobat Reader.