2002 MYN: Evaluation of Training Activities Supported by the Myanmar-UNICEF Country Programme
Author: Khin New Oo; Svendsen, L.; Doyle, B. A.; Seymour, J. M.; Dessallien, C.
The previous Myanmar-UNICEF Country Programme (1996 - 2000) and the current Country Programme (2001 - 2005) have identifed and supported capacity-building activities, particularly training, as one of the key interventions that will lead to improved services and support to children and women in Myanmar. An important portion of the Programme resources has been dedicated to financing training materials preparation, trainer experts, venues, monitoring, etc.
Given the centrality of training in the Myanmar-UNICEF strategy, the level of efficiency and effectiveness of training activities is a critical determinant of the Country-Programme performance. This is why Myanmar and UNICEF jointly decided to undertake a participative utilization-focused evaluation of training activities supported by the Country Programme. This evaluation is the first of the major evaluations to be undertaken during the first half of the 2001 - 2005 Country Programme period, as approved by the Executive Board in September 2000.
The results of this evaluation will contribute to organizational learning within UNICEF Yangon, with partners in Myanmar, and, recognizing that training activities have been adopted as a capacity-building strategy in almost all country programmes, within UNICEF globally. It will contribute to an improvement in the effectiveness and efficiency of systems, programmes and services for children and women.
Purpose / Objective
The overall objective is to assess the effectiveness and efficiency of training activities supported by the Myanmar-UNICEF Country Programme, and to provide ideas and recommendations that will contribute to improving training programmes particularly with regard to their relevance, purpose, processes and the monitoring systems to measure the results (including output, outcome and impact).More specficially the evaluation will:
- assess the relevance of training as an operational strategy for capacity-building
- assess the process of planning training activities, including training materials, processes, participants, trainers and methodology
- assess the contribution of the training activities vis-a-vis the objectives of the respective project and of the Country Programme as a whole (i.e. assess the effectiveness of the training)
The evaluation used complementary methodologies including:
- a desktop review of existing documentation, including annual reports, mid-term review (1998), ProMS output documents, Programme Plans of Actions for various years, related studies and evaluations, reports from other organizations
- focus group discussion with the stakeholders on the process and results of training activities
- key informant interviews including UNICEF Senior Management, Sections Chiefs and Project Officers, Government officials, partners and donors. Interviews were also undertaken with trainers and trainees in various locations throughout the country, including those currently involved in training, and with those who have completed training activities and are now utilizing those skills (after various time periods)
- observation of training activities in various parts of the country, supported by various programmes
- review of training materials, including the process followed in their development, their appropriateness and effectiveness
The evaluation focused on 6 key training components in Health and Nutrition, 5 key training components in Education, 5 in Water and Sanitation, 1 specific training component for Children in Need of Special Protection, and 1 related to Facts for Life information and communication. The selection of training components to be evaluated was based mainly upon three criteria: importance to the project objective, size of budget allocated, and number of trainees involved.
Key Findings and Conclusions
An impressive volume of outputs was produced by the training activities during the 1998-2001 period. Nearly a million beneficiaries received training in the last 3 years, including tens of thousands of health workers, tens of thousands of teachers, parents and school managers, thousands of water and sanitation professionals, and hundreds of thousands of volunteers. This record shows the importance of training in the Myanmar-UNICEF Country Programme.
In terms of resources, it is estimated that 15% of the Country Programme budget is spent directly on financing training activities. But, this percentage is far from representing the efforts that UNICEF staff and partners dedicate to training activities. With hundreds of training action to prepare, implement and monitor every year, it is estimated that training activities mobilize between 40% and 50% of the total time and efforts engaged in the Country Programme.
Training activities are very diverse, ranging from acquisition of the basic elements of job qualification (one or several months), to understanding and mastering new concepts or specific skills (5 days) and to information delivery (1-day advocacy).
The implementation process of the training activities were satisfactory. Managemetn suffered from some manpower shortages and other constraints and it appears that monitoring activities were sometimes too weak. It is estimated that with the same budget, an additional 20 to 30% in efficiency could be obtained with improvements in preparation, focus, implementation procedures and monitoring. These improvements differ from one sector to the other and are detailed in sector reports.
The overall efficiency (quality of processes) appears to have been good. Implementation was satisfactory, although monitoring was weak for some programmes. Modalities of training, instructional techniques and the materials developed for training showed a good choice of instructional technique and good quality training material, with some examples of excellence (participatory and communication workshops). The unit cost of the training activities has been estimated good or very good in all projects and programmes.
For the main objective of the training activities, change in participants' knowledge, skill and attitude, which is the first level of result (or outcome), the overall result is fairly good.
For the other level of results, change in individual's performance and in organizational performances, the evaluation found two distinct patterns. The impact of training activities is good in the Water and Sanitation sector, where other key factors influenced a very successful change achieved in sanitation coverage in recent years in Myanmar. The impact of training is very low in the other sectors because conditions of institutional change and sustainability were not met in the Health and Nutrition sector, in the Education sector and in the CNSP project.
Regarding relevance, the evaluation confirmed that there was a clearly identified need to improve the performance of service providers, and that improving their skills would contribute to overall improvement of the service. Training was therefore deemed appropriate and relevant as a part of the strategy, but not sufficient in itself to increase capacity building. Other influences related to the capacity of service, such as institutional constraints (e.g. midwives being overburdened, poor equipment), structural constraints (e.g. classroom size), low wages, motivation problems, recruitment difficulties, vacant posts, etc. must also be addressed in order to have a sustainable impact on capacity.
These recommendations can be synthesized along 3 main axes:
- More adaptation (contents, methodologies, models) to the context for acceptance and effectiveness
- More professionalism (logical framework, planning, monitoring, result management, integrated capacity building approaches) for accountability and impact
- More understanding and commitment from the counterparts on the need for a multi-dimensional approach to capacity building to improve basic services offered for women and children
To improve efficiency and effectiveness:
- Monitor and evaluate systematically all the training components. For each training component, determine a small number of performance indicators, monitor these indicators regularly and manage for results.
- Focus training on skills that can be applied; incorporate more active practices in the training activities; reduce theoretical presentations.
- Extend the duration of some training courses when it is needed, to reach the learning objective (for instance, at the township level of training in the cascade of teachers' training); conduct refresher training when sustainability needs to be improved.
To improve effectiveness and impact:
- Reinforce the planning process by drafting robust logframes, determine expected results related to time schedules, identify steps if necessary, fix intermediate reachable targets, select good indicators.
- Be explicit about the working model that is promoted through training (e.g. child-centered model); communicate the model to make it more convincing; adapt the model to past experience and readiness of beneficiaries to accept new ways of performing; break down the model and apply it step by step, when necessary, for acceptance.
- Improve the understanding of the multi-dimensional capacity building process among partners and UNICEF staff; work with partners on capacity level indicators (for instance, develop and introduce a proven Criterion Referenced Test for standard four students and develop and standardize it with better indicators and data-gathering methods for the assessment of primary school internal efficiency) and on management information systems.
- Strengthen inter-sectoral training collaboration to create more synergy in specific sectors (e.g. WATSAN and Health and Nutrition).
Full report in PDF
PDF files require Acrobat Reader.