2009 Timor-Leste: Evaluation of the UNICEF Education Programme in Timor-Leste (2003-2009)
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
In September 2009, UNICEF‟s Evaluation Office in New York contracted with the American Institutes for Research (AIR), through its International Development Division (IDD), to conduct an independent evaluation of UNICEF‟s work in the area of basic education and youth literacy, life skills and participation in Timor-Leste. UNICEF‟s education programme, “From Emergency Responses to Sustainable Development for Children and Adolescents in Timor-Leste,” is at a critical juncture as it nears completion, and UNICEF and its partners seek to evaluate the efficacy of the myriad of interventions sponsored during the initiative in order to consider further collaboration.
This executive summary briefly describes the evaluation approach and presents the main findings and recommendations from the evaluation. Detailed results and recommendations are presented in the full evaluation report.
In response to the numerous challenges facing the education system following Timor-Leste‟s independence in 1999, the primary objectives of this education initiative have been to: (1) assist in the development of a high-quality, basic education system through curriculum development and reform, and capacity-building efforts at all levels, including resource gap-filling; (2) assist in advancing the Government of Timor-Leste‟s goal of universal access to education for all Timorese children through institutional capacity building and modernization of education management and information systems to address barriers to access; and (3) build the capacities of youth through initiatives targeting literacy, life skills and participation. Projects within the overall initiative can be split between those that have targeted upstream beneficiaries (e.g., teachers, school administrators, district education officials and parents) and those that have targeted downstream beneficiaries (e.g., youth and adolescents or pupils). The projects targeting upstream beneficiaries have primarily focused on increasing the Ministry of Education‟s (MOE) capacity to increase pupil enrolment and achievement by improving the capacity of teachers, school administrators, and district and national officials to effectively complete their duties. On the other hand, the majority of those projects targeting downstream beneficiaries have tended to do so directly. For example, youth and adolescent-focused sub-projects sought to improve youth participation, youth literacy and HIV/AIDS awareness through direct stakeholder trainings and classes.
Consequently, the objectives of this evaluation were to determine: (1) the relevance, appropriateness and coherence of the UNICEF Education Programme; (2) the effectiveness and impact of the initiative in relation to its objectives; (3) the efficiency with which outputs and activities have been delivered; (4) the connectedness of UNICEF‟s efforts to those of other actors; and (5) the initiative‟s sustainability. The evaluation, though summative in focus (and therefore “backward-oriented”), also sought to uncover potential areas of institutional learning so that the initiative might improve moving forward.
American Institutes of Research (AIR) had envisioned using a mixed-methods approach that relied both on primary and secondary data sources to measure the performance of sub-projects and the overall education initiative against criteria set forth in the Terms of Reference (ToR). Due to a lack of quantitative data sources, the evaluation relied heavily on qualitative sources as a means to evaluate the initiative as a whole. Thirty-four national-level interviews were conducted, including interviews with members of the Ministry of Education (MOE) and other government officials; implementing partners, local and international non-governmental organizations (NGO), civil society organizations and other donors; consultants that had worked for or currently work for the initiative; and UNICEF‟s Country Office, UNICEF‟s Regional Office and Headquarters/New York. UNICEF‟s Evaluation Office hired a local research firm to conduct focus groups with pupils, teachers, school administrators, community members, youth and district education officials in five of Timor-Leste‟s 13 districts. Within each of the five districts, two sub-districts and four school catchment areas were randomly selected to participate in focus group discussions and structured interviews. These districts represent diversity in UNICEF‟s overall level of programme saturation (or “dosage”), as well as geographic, linguistic and religious variation within Timor-Leste. In addition, the evaluation team conducted several direct observations during its data collection mission, including observations of a national training of trainers, preparations for an annual joint planning session, and a life skills training course for youth. The evaluation team also conducted a focus group with teachers and interviewed a school director outside of Dili.
In addition to these primary sources of data, AIR reviewed and systematically analyzed a rich body of secondary data sources. These secondary data sources included progress and annual reports submitted to the Swedish International Development Cooperation Agency (SIDA) by UNICEF, internal monitoring reports, planning documents, independent evaluations of UNICEF activities, scholarly articles, national laws relating to education and youth, curricula, conference reports, reports from other partners and expenditure data from UNICEF‟s accounting system. The primary purpose of this review was to substantiate and triangulate the perceptual data gathered at the primary level, as well as to extend the results of the primary-level analysis. For example, it was very difficult to locate and speak directly with beneficiaries who had participated in sub-projects implemented during the first phase of the initiative (2003–2006). A desk review of project documents and other secondary sources provided the evaluation team with critical information to help evaluate the first phase of the initiative. In addition, the secondary data sources provided a context in which to situate the findings from primary-level data collection activities. Finally, AIR conducted an extensive analysis of cost and output data in an effort to complete a cost-effectiveness analysis.
Based on the foregoing analysis, a number of recommendations emerged. These are presented below. The recommendations are listed in order of priority; Recommendations 1–3 are considered most critical by the evaluation team.
Recommendation 1: The Education Programme and the Adolescent and Youth Participation Programme in the Country Office, in close collaboration with the Planning, Monitoring and Evaluation Sections in the Country Office and East Asia and Pacific Regional Office (EAPRO), should create robust systems of monitoring and evaluation, including the development of a results-based accounting system.
Recommendation 2: The Education Programme and the Adolescent and Youth Participation Programme in the Country Office should provide, in every program and activity, continuing and intensive mentoring and professional development based on the needs and demands of stakeholders at the community, district and national levels.
Recommendation 3: Senior management within the Country Office should convene a task force to increase the efficiency of procurement, logistics and operations, and financial management procedures.
Recommendation 4: UNICEF‟s Country Office should ensure that sub-project goals are aligned with overarching initiative objectives for Timor-Leste and create systems through which to clearly communicate the initiative‟s mission to all staff, partners, consultants and beneficiaries.
Recommendation 5: The Planning, Monitoring and Evaluation Section of the Country Office, in close collaboration with the Education Programme and the Adolescent and Youth Participation Programme, should ensure that programming responds to the needs of sub-populations such as girls, pupils with disabilities and difficult-to-access communities.
Recommendation 6: The Education Section within the Country Office, alongside the Planning, Monitoring and Evaluation Sections of the Country and Regional Offices, should systematically identify and incorporate good practices into programming to scale-up interventions more effectively.
Recommendation 7: The Chief of Education within the Country Office should support increased coordination between the education sector and other relevant sectors to leverage and enhance synergies.
Full report in PDF
PDF files require Acrobat Reader.