“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
This evaluation reviews a project funded by the European Union (EU) and implemented by UNICEF between October 2007 and March 2009 to “improve access to quality education for the Iraqi refugee children as well as the host community children.” To achieve this goal, the Education Support to Iraqi Refugee Children (ESIRC) project was structured around three components: (1) to improve/rehabilitate school buildings and sanitary facilities; (2) to provide educational supplies and furniture; and (3) to enhance pedagogical/learning environment by developing schools’ capacity to integrate Iraqi refugee children.
The purpose of this evaluation is threefold: (1) to assess the overall impact of the ESIRC project, including the preparation and design phase, based on the traditional UNICEF evaluation criteria of relevance, efficiency, effectiveness, impact, and sustainability; (2) to identify/document lessons learned (as they relate to the ESIRC project good practices, strengths and weaknesses, opportunities and constraints;) and to provide recommendations; and (3) to provide recommendations to enhance the effectiveness of the second phase of the ESIRC project, planned for April 2009 - March 2012.
Drawing data from multiple sources, this evaluation used several quantitative and qualitative tools and strategies. Specifically, our methodology included: a review of project documents; a survey of key beneficiaries (principals, teachers, students and parents); and an interview of key UNICEF, EU, and MoE staff involved in the planning, design, and the execution of the project.
Overall Project Outcomes
Overall, the number of Iraqi children who benefited directly or indirectly from the project varied between 2007 (9.770) and 2009 (10.104), with a significant increase in 2008 (15.084) (see Figure 4). These official numbers reveal two observations: first, the project’s goal of providing access to 100.000 students throughout the country, was not reached. Second, we observed a sharp decline (-33%) in Iraqi children’s enrollment in 2009. To elaborate upon the first observation, the goal was based on an inaccurate estimation of the total number of Iraqi students residing in Syria, resulting in a major difference between the project’s goal and the total number of its beneficiaries. (In fact, the estimated number of Iraqis living in Syria could never be verified.) In addition, two other major reasons for this decline were mentioned by the interviewees: the short timeline of the project’s execution (18 months) and the inability to mobilize and educate the targeted community about the project’s goals and activities. Regarding the second observation, the sharp decline observed in 2009 was due to several reasons. Among them were Iraqis returning to their home country, lack of interest in school, and expiration of residency permits.
In terms of effectiveness, several indicators from the survey and the interviews reveal that UNICEF was generally effective in delivering intended inputs (supplies, materials). Overall, this approach of using the presence of Iraqi children as a momentum for school improvement was revealed to be effective and well-received, and lessened the potential negative perceptions of the host community vis-à-vis the influx of new Iraqi students.
Regarding supplies and rehabilitation, the project’s accomplishments were impressive. Successful cooperation between UNICEF and the MoE helped to establish standards for rehabilitation and supplies, providing a systematic approach to school rehabilitation. Based on self-reported data, 4,059 teachers, 96,046 Syrian students, and 9,692 Iraqi students have benefited from the distribution of school supplies and the rehabilitation of schools. Hence, it’s safe to conclude that the UNICEF goal to increase the enrollment of IC through the school rehabilitation and improvement program was partially attained. Self-reported data indicates that double-shifting schools, because of their higher usage levels, exhibited the greatest amount of stress on their physical plants, such as urinals and water taps. This raises the issue of the ongoing maintenance of these facilities. The lack of a maintenance program is likely the main reason behind this rapid deterioration of rehabilitated schools. The implementation of a systematic maintenance program based on established standards should prevent much of this deterioration from occurring in the future.
Schools’ Supplies and Material
Overall, the data indicates that the participating schools reported benefiting from supplies that included (1) school and classroom equipment such as an electrical water cooler, heaters, curtains, etc., (see Table 8 above for details) and (2) teaching materials such as math and science teaching kits and recreational kits. However, some interviewees reported that the wish-list items provided in 2007- 2008 didn’t really contribute to the IC enrollment. This situation was corrected in 2009 to focus on learning and teaching materials such as science and math kits. However, the project provided an opportunity for schools to acquire items that would be otherwise impossible to acquire (copiers, computers, etc.). In fact, it contributed to the establishment of a culture of standards for school supplies and materials based on a systematic process of needs identification, purchasing, and delivering to the different beneficiaries. According to several interviewees, the project initiated a shift from a more traditional centralized approach to a participatory approach which was more responsive to the needs of the schools.
From August 2008 to March 2009, 27 workshops were conducted. Following a train-the-trainers model, the project focused its activities on a few staff members (principal, one counselor and three teachers) at each school, with the assumption that follow-up training sessions would be conducted. Overall, the activities were well-received by principals and teachers, who reported a high level of satisfaction. Of those who attended the sessions, 32% indicated that they benefited from sessions on understanding the psychosocial environment, whereas a little more than 20% reported benefiting from sessions dealing with active learning methods and extra-curricular activities, which is a positive outcome for the overall training. The MoE should follow-up on the train-the-trainers model and ensure that all teachers benefit from the training sessions, particularly the one-third of the surveyed teachers who didn’t attend any of the training sessions.
In this document, efficiency is understood as the relationship between the project activities’ cost and the overall project outcomes. This understanding requires great amount of financial data disaggregated by project activity. Lacking this type of disaggregated data made it not possible for us to conduct a thorough efficiency analysis. Our ability to link expenditures to project activities was limited as well, particularly when other funding streams were used in supporting some of the project activities (supplies and materials). Additionally, this lack of financial data constrained our understanding of the overall project sustainability. However, in spite of this lack of financial data, our interviews revealed that resources were applied appropriately to achieve project outcomes. This perceived efficiency is supported by the accomplishments of the project in terms of schools’ getting rehabilitation, supplies, and materials, and by the overall satisfaction expressed by the MoE senior leadership towards the project outcomes. However, upcoming projects should give closer attention to all of the cost aspects related to efficiency. UNICEF should provide guidelines to estimate the full cost of project activities, including non-budgeted items such as UNICEF and MoE staff time. More importantly, financial data should be collated against project objectives and tracked throughout the project lifecycle.
Project management and coordination
Keeping in mind the initial HR shortage within UNICEF, the project was well managed, overall. As the project advanced and additional resources were brought on board, project activities were kept on track and advanced as planned. In terms of coordination, the project served as a framework for coordination among various stakeholders involved in education. As one the interviewees pointed out, “the project provided a coordination platform that didn’t exist before.”To ensure sustainable ownership, this coordination platform will be coordinated by the MoE. However, it’s important to draw attention to the fact that this platform needs to be used more effectively, particularly in the documenting and sharing of best practices and experiences among all of the stakeholders.
Project monitoring is probably the weakest chain in the ESIRC project. Lack of documentation and history of the project posed several challenges during our evaluation. For this reason, a monitoring framework with clear guidelines and documentation should be designed and implemented. This framework should cover all project activities (rehabilitation, supplies, training, CFS) and should be linked to specific objectives and targets. In addition, a clear reporting system should be established for all stakeholders, indicating report type, frequency, audience, and feedback. To facilitate this project, a project document repository should be created and maintained within the UNICEF Education sector, and should include all project documents. A naming convention should be established and implemented, in order to facilitate file tracking, versioning, and retrieving.
Partnership and Collaboration
As noted before, although the partnership and collaboration between key stakeholders was strengthened, beneficiaries’ involvement was very limited. The role of the Iraqi community in the project should be clearly defined and their participation and involvement in the design and execution of the project should be encouraged.
PDF files require Acrobat Reader.