Evaluation database

Evaluation report

2012 ESARO, Kenya: Building a culture of resilience



Author: Professor Sultan Barakat and Professor Frank Hardman

Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."

Background:

As part of continuing efforts to ensure that education services are sustained in the wake of natural and man-made disasters, UNICEF and its partner organisation, Save the Children (SC), undertook a capacity building training and institutionalisation programme, a component of the global Education in Emergencies and Post-crisis Transition (EEPCT) Programme. This initiative, implemented with the support of the IASC Education Cluster, aimed to “…build and strengthen sustainable national emergency preparedness and response (EPR) capacity in the education sector in the Eastern and Southern Africa Region (ESAR) holistically and strategically.” Since its inception in May 2009, the training programme has managed to reach more than 2,800 frontline responders from national governments, local and international NGOs, civil society organisations, and UN agencies throughout ESAR, despite many challenges.

Evaluations of the capacity building strategy were conducted in five countries in ESAR—Burundi, Comoros, Madagascar, Malawi, and Rwanda. Through a mixed-methods evaluation strategy that included a desk review of open source documents, academic literature, and reports provided by UNICEF ESARO and country offices, in addition to interviews, focus group discussions, observations in the field, and an online survey, this evaluation sought to assess the knowledge and practices of the training participants in order to provide a comprehensive account of the impact of the UNICEF/SC capacity building strategy. It also explored the level of institutionalisation of EPR/DRR practices into the education sector policy and planning at the national and sub-national levels. Finally, capacity gaps and challenges to institutionalisation efforts were identified, principal lessons learned were outlined, and corresponding mitigating actions were recommended. Key elements for future capacity development strategies, as well as a logical framework and a road map for the way forward, are also included in this report.

Purpose/Objective:

The Eastern and Southern Africa Region is vast—reaching from Eritrea in the north to South Africa, with a population of approximately 370 million. With the emergence of South Sudan as an independent state in July 2011, UNICEF now has 21 country programmes throughout the region. This large, populous, and diverse region experiences some of the highest recurrences of emergencies in the world, with natural disasters such as repeated droughts, floods, severe storms, and cyclones representing most of the emergencies. Experts predict that it will be one of the regions of the world most affected by climate change, with some countries experiencing prolonged drought, intensified because of higher temperatures and reduced rainfall, while others suffer from increased storms and higher ocean levels. Additionally, conflict, political unrest, large-scale population displacement, economic crises, and disease epidemics combine to create complex emergencies and further impact on an already vulnerable people. 

Methodology:

The five countries included in this evaluation varied significantly in the level and types of natural disasters they experience on a regular basis. While not representative of ESAR as a whole, the most prevalent natural disasters afflicting the five country case studies were droughts, earthquakes, floods, storms, and, in the case of Comoros, volcanoes.

The impact of natural disasters and political and economic turmoil on children and the education sector is immense. Nearly 9 million children are out of school in ESAR, many of them due to emergencies such as conflict, socio-political crises, and recurring natural disasters. These emergencies regularly disrupt schooling for millions more. Building greater resilience into the education sector at the national, sub-national, and local levels is imperative if education services can continue to be provided to learners throughout emergencies and crises.

Findings:

The evaluation focused on assessing the impact of the capacity development strategy on training participants on three levels: knowledge, practices, and institutionalisation. The evaluation found that, despite the enormity of its task and the relative lack of resources, the capacity development training and institutionalisation strategy has managed to train a critical mass of practitioners, decision makers, and key education stakeholders in its initial phase and lay the foundation for building a culture of resilience throughout ESAR; however, there remains room for improvement and strengthening the achievements made by the strategy.

While it was difficult to isolate the impact of the UNICEF/SC training programme on participants’ knowledge, particularly at the policy level, the evaluation determined that the training participants demonstrated increased knowledge of emergency preparedness response/disaster risk reduction (EPR/DRR) principles, but that this knowledge progressively diminished at successive levels of rollout. The provision of basic EPR/DRR manuals and visual learning aids proved to be a key element of increasing knowledge retention and assisting with the dissemination of the training’s EPR/DRR training message. Greater collaboration with relevant civil society partners, particularly, at the local level, would increase the grassroots EPR/DRR knowledge base, which was found to be low compared to the national and sub-national levels, where actors had directly participated in the training initiative. Finally, the evaluation determined that insufficient attention has been paid to emergencies linked to conflict displacement, political unrest, and health epidemics, both in the training and in subsequent activities.

Training participants did demonstrate improvement in their EPR/DRR practices in many cases; however, there were inconsistencies in their practices across different countries and levels. The participants who responded to the online survey and took part in interviews during the five country visits indicated that the information received in the UNICEF/SC training assisted them in the development and strengthening of their EPR/DRR skills and practices.

The evaluation found that there have been encouraging steps taken to include EPR/DRR practices in the education sector policy and planning to varying degrees in the five country case studies. A particularly crucial achievement of the capacity building strategy was closer co-ordination with relevant actors on Education in Emergencies (EiE) which strengthened the impact of the training and institutionalisation efforts in countries where it occurred. A major challenge to the long-term sustainability of the institutionalisation efforts was the weak financial, technical, and operational capacity of the ministries of education and disaster management agencies in each of the five country case studies. Also negatively impacting the implementation of EPR/DRR best practices was the fact that the information gathering and dissemination systems were extremely weak or non-existent in all five country case studies. The fact that follow-up activities and monitoring & evaluation mechanisms were not put in place also weakened the impact of the institutionalisation efforts. There is a lack of awareness amongst the general public, national government representatives, and the donor community of the importance for prioritising EPR/DRR initiatives in ESAR. To mitigate this, a comprehensive, multi-sectoral strategy that is responsive to the different country contexts and diverse capacities and needs at the local, sub-national, and national levels is required.

CAPACITY GAPS

A key task of the evaluation was to identify capacity gaps that, if addressed, would strengthen the effectiveness of the capacity development programme. The information gathered in the five country visits showed that the capacity development efforts have not yet reached all key stakeholders, particularly those at the grassroots level, weakening the training impact at the subnational level. Additionally, staff turnover and a lack of follow-up have resulted in low levels of knowledge retention. The teaching and learning aids that were so instrumental in further dissemination of the EPR/DRR knowledge and practices are lacking in most of the countries visited, and the initiatives have not addressed some of the key emergencies, particularly related to conflict, and in some cases, there has been insufficient contextualisation, resulting in a low level of knowledge, skills, and buy-in from key stakeholders. There is also a lack of baseline data, making it difficult to attribute the impact of the UNICEF/SC training and institutionalisation strategy. The lack of rigorous monitoring & evaluation also creates challenges in measuring the effectiveness of the EPR/DRR capacity development strategy. In addition, the evaluation found that while good progress has been made on incorporating EPR/DRR practices into education sector planning and policies, they have yet to be translated into action, which can be a factor of the lack of resources that the relevant national government ministries and agencies are experiencing. Co-ordination mechanisms also need to be strengthened and weak information gathering and dissemination systems severely limit the prospects for viable and sustainable EPR/DRR programming. Co-ordination is also lacking between EPR/DRR interventions, which are often ad hoc, resulting in overlaps and gaps. Finally, one of the largest capacity gaps is the lack of awareness of the importance of Education in Emergencies programmes amongst the general public, national governments, and the donor community.

Lessons Learned and Recommendations:

The lessons learned draw from the information from the findings and capacity gaps, offering actionable recommendations to address each gap.1 The first lesson concludes that EPR/DRR capacity development training can contribute significantly to improved emergency response; however, when the cascade training model employed by the EPR/DRR capacity building training is used in isolation, it offers extremely limited potential to improve EPR/DRR practices at the grassroots level.

Additionally, teaching and learning aids must be provided to schools to ensure that children benefit from the capacity building initiatives. Future initiatives should also take into account the need for strategic collaboration with community-based groups, which offers further potential to develop and strengthen long-term sustainable EPR/DRR capacity, as broad and multi-sectoral collaboration offers the best chance of successful EPR/DRR capacity development. Future capacity building strategies should be situated within a multi-sectoral and comprehensive EPR/DRR strategy to ensure maximum impact, sustainability, and value for money. Future programmes also need to incorporate the following key elements: gathering baseline data at the beginning of future training initiatives to assist with monitoring & evaluation efforts; the inclusion of all crises relevant to ESAR and individual country context; a variety of follow-up activities in conjunction with the training encourage longterm sustainability; and rigorous follow-up and monitoring & evaluation mechanisms. The evaluation’s findings made clear that strong national institutions are essential to the long-term success and sustainability of EPR/DRR programming and that in order to ensure that knowledge gained is implemented into relevant education sector policy and practice, guidance and technical support is required.

Conclusion:

The study found that, in many cases, significant progress had been made towards improving individual training participants’ knowledge and practices, and that promising steps had been taken in many target countries towards institutionalising EPR/DRR within education sector policy and planning. Crucially, co-ordination between key EiE partners had been significantly strengthened as a result of the training. Further, policy and planning within organisations targeted had been subsequently improved to incorporate key elements of the EPR/DRR capacity development training. Despite such marked achievements, the evaluation also identified several challenges to the success of the initiative across the five country case studies. In some countries, such as Madagascar, programme impact had reached the vital community level, and had led to tangible improvements in the disaster preparation and response of local Ministry of Education staff, teachers, and learners. Such cases are nascent examples of the EPR/DRR capacity development programme contributing to what this report terms ‘building a culture of resilience’—a combination of ten key elements identified by this evaluation as necessary for promoting successful EPR/DRR programmes.

Despite the challenges, the achievements of the EPR/DRR capacity development and institutionalisation initiative are particularly significant in view of the relatively limited financial, technical, and material resources allocated to the programme and its broad scope of implementation. These commendable achievements must now be expanded upon and supported with additional resources, as outlined in this evaluation, to ensure that programming contributes to building the culture of resilience necessary to ensure that all girls and boys, including disadvantaged groups, can benefit from their right to education before, during, and after crises. Through UNICEF’s role of policy advocate and technical advisor, the organisation is well-placed to champion Education in Emergency programming with the goal of supporting national governments and other stakeholders in the region in their efforts to build a locally owned and led sustainable culture of resilience.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search