Evaluation database

Evaluation report

2010 Cambodia: Evaluation of the Accelerated Learning and Multi-Grade Teaching Programmes



Executive summary

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

(A summary of the 12 sets of Recommendations from this evaluation is included as Annex 8. Most have relevance for the revision of the CFS Master Plan)

INTRODUCTION
  • The two programmes, Accelerated learning and Multi-grade teaching were introduced in Cambodian schools in response to two associated problems: a gap in enrolment and overage children.
  • MGT has been applied since 2003 and now operates in 10 provinces or municipalities (Kompong Thom, Stung Treng, Kompong Speu, Prey Veng, Otdar Meanchey, Svay Rieng, Kampot, Takeo, Kandal and Kep). A/L has been applied since 2006.
  • The evaluation took place from 16th March 2010. Over the following 2 month period at least 100 stakeholders were consulted in 4 locations and at central and decentralised levels of the Ministry.
  • Central records for MGT and on overage enrolment, net enrolment and gross enrolment ratios (NER and GER) were examined from 2004/5, as well as survival rates and numbers of “complete” schools (i.e. schools offering teaching of grades 1 to 6). Central data on A/L is sometimes confused with that on equivalency. However provincial records are accurate and readily available.
MoEYS Guidelines
  • The situation with regard to salary incentives and local arrangements for teacher deployment in Cambodia is governed by practices that have become widespread rather than by regulation. In October 2009 the Ministry issued a set of Guidelines whose effect was to bring these practices back within existing rules.
  • Flexibility has been applied to class-size norms for primary schools (officially 50 students). This has allowed an increase in double-shift teaching from 5,431 schools in 2004/5 to 5,993 in 2008/9. Between 2004/5 and 2008/9 the numbers of MGT and contract teachers have also risen (MGT from 518 teaching in 665 schools to 1172 in 941 schools; contract teachers from 740 to 2,214). There were more A/L teachers in 2008/9 than in 2004/5 also (the programme was started in most provinces in 2006), but national data is not available. Data are still not kept separately for A/L and the Non Formal Education (NFE) equivalency programme except at provincial level.
FINDINGS
  • The evaluation findings are presented and discussed in the report below under the three headings: Relevance, Effectiveness and Sustainability.
RELEVANCE
  1. National Priorities
  • A/L and MGT are mentioned in several policy documents: The ESP 2006-2010 lists MGT as one of the strategies to achieve the outcome and targets of the Equitable Access policy. A/L is not mentioned in the ESP 2006-2010, nor in ESSP 2006-2010. The ESSP 2006 -2010 states that there will be an Action Plan for introduction of multi-grade teaching (p45).
  • The ESSP 2006-2010 also includes a reference to plans for the development of the Child Friendly Schools (CFS) programme. MGT occurs briefly in the CFS Master Plan as does A/L. MGT is included in the Education For All (EFA) National Plan of Action (NPA) 2003 – 2015 and is linked to double-shifting in smaller schools. The EFA Mid Decade Assessment (MDA) refers to MGT as an effective strategy for addressing teacher shortages and incomplete schools in rural areas.
  1. Size of the Problem
  • The size of the problem was estimated through a data set with figures from MoEYS EMIS records between 2004/5 and 2008/9. EMIS were especially helpful in extracting data on MGT from their records. These show a steady rise in the number of schools, classes and teachers using MGT since 2004/5. Both Government and Contract teachers have increased in number.
  • The total number of MGT classes in Otdar Meanchey is currently thought to be 143, over double the number (68) recorded by EMIS for 2008/9. In most cases newly appointed MGT teachers have not yet received training and there is a need for immediate support and also a long-term strategy.
  • In Stung Treng things are different. The number of classes recorded at central level for 2008/9 was 114. The POE reported the same number for this year, and this lack of change seemed to be confirmed from visits to schools.
  • The potential size of A/L programmes nationally is judged from the number of overage students in primary schools. EMIS data on the population at age 6, age 6 new intake in Grade 1 (from which the %age of correct age enrolment can be calculated), primary GER and NER were examined to indicate the size and persistence or gradual elimination of the problem.
  • Central records for overage enrolment suggest little change since 2004/5, just over 20% of primary population. An indicator of elimination of overage enrolment is low and falling GER. At 120% for the whole Kingdom GER is not yet low. The trend for the last 3 years in Kg. Speu and Prey Veng is downwards, though not significantly. In remote provinces including Stung Treng the recent trend is in the opposite direction.
  • Figures for A/L programme schools, teachers and classes are not recorded centrally. There is confusion with Re-entry and Equivalency programmes. 10,684 students (3,766 girls) are enrolled in “Equivalency Programmes/ Accelerated Classes” according to NEC. There were 656 Equivalency or Accelerated learning teachers (235 females), (Unofficial translation, NEC Report, March 2010, p60). It is not possible to establish student numbers in the separate A/L programme.
  • Budget implications of plans for expansion of Re-entry Equivalency and A/L programmes need to be examined.
  1. Links between programmes
  • Policy priority was gained from analysis of the broader context. The weakness of linkages to strategic plans is cited in a Mid-Term Review (MTR) of UNICEF Education Pilots and Early Childhood Development Initiatives, 2006-2008, in 2008: “pilot projects have not maintained as close linkages as needed” (p3).
  • A/L and MGT are associated with ESP, ESSP and EFA plans, but more as options, not as key strategies and with little focus on recorded outputs/comes. Similarly the CFS programme provides a broad context for A/L and MGT, but the links are not strong either conceptually or in terms of management oversight.
  • NEC reports still focus more on inputs than outputs and plans for “scaling up” of the A/L programme (p118) risk double counting.
  • At provincial and district level there was a much greater sense, at least in most areas, that these two programmes are making a contribution to an improved education system. Until recently A/L was positively associated with education reform in most areas. MGT is also understood clearly as a long-term strategy for providing education in sparsely populated areas of Otdar Meanchey. From reports, MGT is more associated with serious challenges in the system in Stung Treng. In terms of quality assurance, nowhere was there an explicit link between either programme and curriculum outcomes or student achievement standards.
  1. Persistence
  • The 2003 EFA National Plan of Action 2003-2015 considered MGT a temporary solution. The POE Director in Stung Treng also sees MGT as a temporary, and regrettable, strategy. POE in Otdar Meanchey, on the other hand, expects to include MGT in its programme strategy for the foreseeable future.
  • The number of MGT classes this year has jumped in some areas and It is hard to separate the long-term need from this temporary reaction to tightening of the Ministry norms. However, there may always be some communities where it is uneconomic to provide a complete set of teachers for a population of below 300 children.
  • A “complete” primary school is one that offers all 6 grades. If a school is complete it may not need a MGT programme, drop-out may be reduced, and overage enrolment may fall also. The number of complete schools has risen steadily in most provinces since 2004/5. With more complete schools the need for MGT and A/L programmes should be reduced.
  • EMIS figures also show a rise in survival rates to grade 6 except more recently in remote areas. Survival for girls is also steadily improving, except in remoter areas, including Stung Treng. Nationally rates are still very low (around 60%), meaning a large number of students still drop out and may be eligible for re-entry or A/L in the future. The need for such programmes is likely to remain for particular populations and it is important to identify where these populations are and to plan support and monitoring at school, district and province level.
  • Age 6 intake figures at grade 1 also show a steady rise for girls as well as boys and broadly for urban, rural and remote areas. In this sense, the need for A/L classes, at least for grade 1 students, is gradually being reduced. Grade 1 enrolments into A/L are also falling though the impact of the new Ministry Guideline has undoubtedly been felt by the A/L programme and may result in making the programme less attractive to students. Although at the moment the teachers in these classes were both energetic and dedicated to the improvement of opportunity for their community, there is a strong risk that the situation will deteriorate.
  • To summarise a complex situation, there are long-term trends which suggest the continuing need for MGT classes in small rural communities. It is important to identify these and to separate theme in terms of planning and support from those affected by the immediate impact of MoEYS Guidelines.
  • The need for A/L classes seems to be diminishing. In this sense the programme is remarkably successful. However, there are immediate concerns thrown up by the Guidelines where DOE staff and school principals need help and flexibility in supporting some excellent teachers.
EFFECTIVENESS
  1. Are the programmes effective?
  • Education staff at all levels (teachers to central department staff) say that A/L is a success, but MGT is problematic. This is despite the fact that support for both in terms of training and materials is similar, class sizes are similar and the demands on the teacher are not so different. The financial incentive (+60% vs 0%) definitely favours the MGT teacher, but apparent characteristics of schools and communities do not appear to differ markedly, except in Stung Treng where several schools are in ethnic minority areas.
  • Measures of success cover inputs and programme characteristics rather than results, though in some respects these are impressive. Measures of success for A/L included the fact that students enrolled in large numbers and that the drop-out and failure rates (in the end of year test) were low, although figures for enrolment, retention, promotion and completion were not emphasised by any of the respondents as measures of success.
  • A serious problem in terms of any measure of education quality or effectiveness is that there is currently no reliable standard for comparing regular and multi-grade or accelerated classes. Reliable end of year testing is technically difficult without a precise and explicit standard to aim for. The most commonly used standard still seems to be whether or not the teacher or student has “finished the book”.
  • Several teachers and principals were asked how students from A/L and MGT programmes compared with students from regular classes. Answers were that A/L students could perform better since they were older, more motivated, etc. but those in MGT classes were inevitably weaker than in regular classes. However, several felt that low quality was certainly not inevitable.
  1. Quality or Access issues?
  • When the evaluation asked this question the most frequent reply was that MGT and A/L are programmes that aim to increase enrolment, so they are access programmes. And quality is commonly shown by promotion and drop out rates.
  • The distinction between quality, not simply efficiency, and access is important because the response to the problem will be different depending on the different understandings and measures of success will differ also.
  • In the case of A/L the programme largely addresses access and efficiency, and from measures of achievement discussed above (enrolment and promotion) it seems to be successful. However, the low priority apparently given to the basic skills of reading, writing and numeracy are worrying. If children are unable to read with understanding by the end of grade 3 they must be at risk of failing or dropping out. This is surely a priority quality target.
  • The same issues are true of MGT. If the programme is understood as a strategy for providing access the need is for training, incentives, materials, classroom organisation, timetabling, etc. Success is measured by enrolment, survival rates, falling drop out, etc. If quality is an important criterion and measure of success then promotion and survival are also important, but so is student achievement on basic skills. In general problems and solutions facing the MGT programme were closely tied to the general situation for education in the province
  1. Management or Technical Response?
  • Is the current response largely management or technical and which is more appropriate? The selection and preparation of teachers for A/L were identified consistently as key components in programme success. A/L training is short and largely involves an examination and discussion of materials. TTD has provided 6 days of training to over 1700 MGT teachers, and teachers are provided with a salary incentive
  • In the schools visited, relations between the principal, the teachers and the community were good and seen as critical in Kg Speu and Prey Veng. In Otdar Meanchey the school cluster seemed to play a key support role. Community involvement in the school in Stung Treng was more difficult and less settled.
  • Any school programme depends on provision of basic infrastructure and a strong relationship to the community. These non-technical criteria are essential before any programme can operate let alone one with special characteristics. In this sense management issues come before technical ones.
  • Both programmes have materials developed with UNICEF support. The student materials in A/L classes were, by contrast to those in MGT programmes, seen in use in all classes visited. Given the state of affairs in the MGT schools visited it would seem that the MGT Teacher’s Book deals in more detail than appropriate with conceptual issues. A more pragmatic approach may be better.
  • The management of A/L programmes was impressive though the broad numbers alone do not reveal the whole picture. Nevertheless, one can see that the programme is being actively managed at local level, at least in terms of access and support.
  • Technical support for both programmes is provided by Trainers at POE and DOE levels. Each district has identified a District Training and Monitoring Team (DTMT). The Team is supported under the CFS programme and MoEYS is making use of FTI funding to further develop capacity.
  1. Monitoring and Evaluation
  • The lack of a precise and widely understood objective for each programme makes them harder to evaluate. There is a need for an M&E framework. This is not yet in place, though as shown, data and monitoring are available, in rather fragmented form at various levels. National provincial and district data, at least for the A/L programme suggest positive progress Results for MGT are more mixed and less easy to determine.
  • The development of a central Annual Operational Plan (AOP) is a major step forward and still a fairly new development. This is essentially, the M&E framework to be applied to the two programmes.
  • The difficulty is that macro-level planning is disconnected from more detailed plans at sub-national levels. The outputs for these programmes and activity areas are very general and do not translate easily into provincial level inputs, or, more importantly, into outputs or results.
  • Field visits confirmed the mismatch of demand with trained teachers. A more explicit strategy for retaining trained staff in schools where they can use their skills was discussed with POE staff in Otdar Meanchey and Stung Treng. Central Ministry data suggest that the training is not always being made best use of.
Sustainability
  1. Measures of Sustainability
  • To achieve mainstreaming, “Emphasis should be given to aligning with existing MoEYS structures” (UNICEF MTR p4). The set of questions in this area of the evaluation were designed to test ownership by MoEYS, harmonisation with other DP programmes, alignment with MoEYS structures and measurement of results through MoEYS systems and according to MoEYS criteria.
  • Ownership and alignment should be seen in terms of local management and support systems, i.e. planning, reporting and performance monitoring. This was more evident in the A/L provinces, which is a cause for concern. The need for MGT programmes, is likely to be more durable and systems in Otdar Meanchey and Stung Treng (and other provinces) should be strengthened as a matter of priority to increase sustainability.
  • It has been argued above that explicit measures of student performance in these special programmes should be against the same standards used for regular classes. A focus on results will also support Cambodia’s Harmonisation Alignment and Results (HAR) Plan of Action, 2006.
  • UNICEF does not see either A/L or MGT as an externally managed project. It is a measure of success that none of the MoEYS respondents in the field saw either programme in this way either. Clarification of the role of central MoEYS staff in terms of performance oversight through data monitoring and the occasional provincial visit would be beneficial.

  • There was no evidence from discussion or visits of other projects or programmes running parallel A/L or MGT activities.
  1. Links to Policy & Practice
  • Both programmes are quite well integrated with MoEYS policy and practice, though clearer policy goals and more explicit management objectives at local level would be welcome. Broad links to better student achievement is also needed to guide teachers and students, and to tell parents and education staff what students are learning.
  • The Child-Friendly Schools (CFS) programme, provides a logical “home” for the A/L and MGT programmes. Both are included as specific sub-programmes under the broader umbrella of CFS, though the extensive CFS materials do not do more than mention the programmes and give few details as to how either programme should operate. CFS aims to provide a framework and not a separate set of activities. This should allow the A/L and MGT pilots to link with other broad CFS activities such as teacher training. Teaching ideas for regular classes are often as applicable to MGT or A/L classes.
  • The A/L teachers interviewed did not find the planning of their lessons a problem, perhaps because of the materials, many MGT teachers said they needed more help. It is not at all clear what would be useful unless more specific textbooks were available. CFS and other teaching materials are often badly stored and neglected. Several respondents clearly found the amount of material already distributed overwhelming. A storage system and a means of listing what materials have been received is a priority.
  • The popularity of the A/L sets comes partly from the fact that they can be given directly to the children and seem to require no additional planning or preparation. On the other hand it remains unclear whether the children are actually learning what is required by the curriculum.
  1. Proportionate and Organic Response
  • MoEYS is committed to continue and to expand the programmes. However, the size of need is difficult to determine. The policy base for both programmes is also not strong. Clearer policies and performance measurement are needed since without evidence the argument for expansion is weak and without policy the programmes will not be sustained.
  • At school and district level the presence of overage students in most classes is common. Moreover, some school directors and DOE staff who heard informally about A/L have asked for details of the programme and organised training for themselves without outside support. Thus there is some evidence of capacity to solve problems locally.
  • It is not clear why some trained, prepared and experienced teachers should find MGT teaching as demanding as they claimed. In Stung Treng the overwhelming assessment was that MGT is an unsatisfactory system.
  • Equally it is hard to explain why a teacher in an A/L class close to a town centre that had dwindled to a manageable dozen students be so discouraged, while a rural area counterpart suddenly obliged to teach over 50 students in the same class, half A/L and half normal grade 1, with no incentive payment should remain motivated and effective. School, district and POE leadership may be involved.
  • A professional commitment to the children in the community was one striking component of the most positive schools in Kg Speu and Prey Veng provinces. Teachers know their communities and are aware of providing a service to the families there.
  • “Participation by children, families and communities” is the 5th Dimension of the CFS programme, and this participation seems to be a strong characteristic of a positive school environment. To achieve this situation the CFS materials can provide some ideas, but can also appear daunting and almost present a barrier if they are seen as programmes that must be followed in every detail.
  1. Curriculum & Standards Training
  • TTD will spend 5003.8m riels on training and implementation of the Basic Education Curriculum and Standards this year. Other departments also plan to spend budget on related activity. Value for money would be shown by a substantial impact on teaching and student performance across all programmes. However they need to be implemented with an emphasis on results, i.e. measurement of students achieving the expected standards.
  • From school visits and in discussion with POE and DOE staff there is still no explicit link between current school programmes and the Standards. Awareness of the 2006 Curriculum and the Student Achievement or Curriculum Standards at POE and DOE level was minimal, and while the Standards posters were displayed in some schools, they were not claimed to be in use and no connection was made to the A/L or MGT programmes.
  • The Standards are able to provide a guide for teachers and principals to the student achievement they should be aiming for in MGT, A/L and regular classes. In addition, using these Standards would make it possible to compare performance of students in regular, MGT and A/L programmes. Without this it is not possible to judge the effectiveness of spending on training programmes or the success of special programmes such as MGT or A/L. Student “performance” simply means being able to read and write confidently by the start of grade 4, an essential target for all students.
  • The School Self Assessment (SSA) component of the final CFS Dimension attempts to institutionalise reporting on children’s performance to the local community through a “school report card” or Evaluation Tool. What is new in this Dimension is the inclusion of student performance against the Curriculum Standards, and the sharing of this information with the community.
  • From observations, implementation of CFS has not yet reached this point of development. In addition, before thinking of sharing information on performance with parents, there is currently little sense of school performance on the part of principals and teachers internally. References to standard learning achievement must surely be the next step in linking all school programmes to a national standard in the most basic of skills.


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search