We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Afghanistan: Summative Evaluation of Child-centered Disaster Risk Reduction Project



Author: Nadeem Haider, Asmat Gill

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

The object of the evaluation is a joint venture of UNICEF and Save the Children Afghanistan titled: ‘Child-Centred Disaster Risk Reduction (CCDRR) Project'. This is one of the pilots that stemmed out of the ‘Strategic Partnership’ between UNICEF and Save the Children International (SCI) Afghanistan Offices.

The CCDRR project was conceived and implemented with the aim to strengthen the capacities of communities, relevant government institutions (particularly those responsible for disaster management), civil society organizations (CSOs), teachers and children (in schools) to understand, mitigate, and prepare for natural disaster risks. The goal of the CCDRR project was, “to build the resilience of communities in Afghanistan to natural disasters through a community-led child-based approach to disaster risk reduction (DRR)”. The project had three Intermediate Results (IR # 1-3) or components, each focused on enhancing capacities of communities, governments (including CSOs), teachers and children at schools.

The CCDRR project was implemented in 12 districts of Balkh, Jawzjan, and Saripul from November 2012 to March 2015. The community and school based risk management component was implemented in 120 disaster prone communities and 52 schools. The target groups included communities (men, women, boys, girls and older persons), teachers, education managers, students at schools, and staff of relevant public departments and CSOs at national, provincial and district levels. The project benefitted over 23,840 persons directly and another 155,400 individuals indirectly.

UNICEF Afghanistan provided the funding and oversight. The actual implementation of the project was carried out by SCI Afghanistan. Afghanistan National Disaster Management Authority was the lead government partner, together with Ministry of Education.

Purpose/Objective:

The principal purpose of this summative evaluation was to have an objective assessment of the overall effectiveness of the project strategies and activities with respect to improved resilience of the Afghan communities, especially children. Moreover, it was expected to inform country strategies, national and international DRR policies and practices for both the SCI and the UNICEF. Furthermore, it was commissioned to distil and document key learning for the stakeholders.

The evaluation was commissioned to demonstrate commitment to internal and external accountability, learning principles and values, including meeting commitments made to the donors. The objectives of the evaluation included: i) assessment of project achievements; ii) objective assessment of coordination mechanisms, monitoring, evaluation, accountability, and learning (MEAL) processes; iii) coherence to UNICEF and SCI programming priorities & principles; and vi) value addition of UNICEF-SCI strategic partnership. Furthermore, it expected the consultants to list lessons learnt, good practices, and use findings and learning to offer actionable recommendations.

The evaluation followed The United Nations Evaluation Group (UNEG) and Organization of Economic Development and Cooperation (OECD-DAC) evaluation criteria including impact; effectiveness; efficiency; and sustainability. It looked into UNICEF programming principles and priorities i.e. equity, gender equality, accountability, human rights based approach (HRBA), and assessed the alignment to UNICEF and SCI country strategic plans. The evaluation has complied with the UNEG principles, norms, and ethics.

Methodology:

The mixed method approach used for field data collection that enabled to address the method/technique related limitations, biases, and facilitated data corroboration for validation and meaningful triangulation. The evaluation did make use of secondary data and information made available. It was used to complement the primary information. Recognizing children as most significant respondents’ group, this evaluation made use of ‘child friendly techniques’ to engage with children. On that count, it could be referred to as a ‘child-centred evaluation’. Extensive secondary sources review was undertaken, during which 434 internal and external documents were reviewed. Multiple quantitative and qualitative methods were used for primary data collection. The primary data collection entailed multiple methods including a representative household perceptions based survey (post-KAP); key informant interviews (KIIs), focus group discussions (FGDs), and field observations. The post-KAP covered 402 households (with 45% female respondents) in 10 out of 12 districts (2 districts were dropped for security reasons). Purposive sampling technique was used for application of qualitative methods. The qualitative methods were applied to explore “Positive Deviance” and the associated factors (enablers/dis-enablers). In total, 50 KIIs, and 13 FGDs were conducted at all levels. Out of 150 FGD participants, 60% were children (boys and girls), while remaining were adults, both male and female. The primary data collection had been carried out in two phase of field visits, during which capital and respective provinces were covered. The survey was administered through a team of field enumerators (50% females), and were trained in conducting interviews and recording information. The training session was followed by pre-testing of survey questionnaire.

Findings and Conclusions:

The project was and continues to be highly relevant to the Afghanistan development and disaster risk context. The UNICEF-SCI strategic partnership is unique in carrying huge potential to capitalize on the comparative strengths and advantages for meaningful contributions for children and women in Afghanistan. The prioritisation of CBDRM/SBDRM approaches adds further to its relevance. These appear even more relevant to a context such as Afghanistan, featuring low levels of development, chronic insecurity, limited outreach, and capacities within public agencies, and evolving disaster risk context. Furthermore, it will enable and empower communities to take meaningful actions at local level for prevention, mitigation, preparedness, and response.

There are noted design deficiencies such as unrealistic or ambitious result expectations, and limited focus on addressing/advocating the underlying causes for disasters and disaster risks (referred to as macro-development agenda). Moreover, the project design demonstrates gaps around strategic engagement with government for sustainable capacity development, and in-apt reading of the governance context to facilitate creation of sectoral coordination mechanisms.

Ample evidence is available to suggest that project leveraged organizational learning and resources from other projects. This is both encouraging and enabled quality and consistent project delivery. The demonstrated successes with social mobilization, in particular with having representation and participation of women and children in community forums and project activities, enabled inclusiveness and create equitable results. The project implementation could not leverage the strategic partnership fully, as is evident from limited intra-/inter-organizational coordination. Effective coordination could have helped in resources maximization and relatively more responsible exit.

Recommendations:

  1. Undertake sector-wide assessment of the efficacy and impact of CCDRR/CBDRR/SBDRM models and approaches applied by all actors, to guide future programming and donor/government prioritisation.
  2. UNICEF-SCI to draw on organizational strengths fully for creating meaningful results for children. Intra and inter-organizational coordination/collaboration mechanisms need to be streamlined.
  3. Prioritize DRR integration into thematic work (CP, WASH, Education, Health) over standalone CL/CCDRR Projects, unless absolutely necessary. (UNICEF and SCI Senior Managers/ Thematic Units).
  4. Draw balance between micro and macro level initiatives (macro may imply research, advocacy, etc.) to contribute to desired impact/change (UNICEF and SCI thematic leads).
  5. SBDRM must graduate to comprehensive school safety from existing school preparedness (centric) approaches, and may need to be seen as integral and/or complementary to the CBDRM approaches.
  6. Government capacity development must consider local context, prioritise institutionalization of new practices (including creating adequate technical, financial and material capacities) for sustained change.

Lessons Learned:

Social Mobilization: The social mobilization process must take stock of and capitalize on the prevalent community organizations or forums (instead of creating new ones) to minimize friction and disruption (as noted for having CDC representation in the CERTs).
Geographic Targeting: Evolve and apply layered approaches (featuring methodical assessments and stakeholder consultations) for geographic prioritisation, to achieve effective and equitable targeting.
MEAL Framework: Develop and rigorously apply performance benchmarks for delivery of acceptable (quality) and consistent inputs and outputs (delivery). 
Capacity Development: Undertake deeper assessment of partners/governments current capacities and commitment, and use them to inform the development and execution of exit plans featuring gradual transition to these partners for continuity/sustainability of project outputs e.g. community forums and mitigation projects. 
Development Needs Assessment: Assess local development needs and priorities (such as village development plans) and strategize dovetailing the project inputs (soft and hardware support) to the extent possible with local priorities to create ready acceptance and harness support for other project inputs. 
Thematic Integration: Leverage available organizational resources and context knowledge e.g. training & communication materials, trained staff, awareness of key influencers (like Mullah), and others, to maximize results and prioritising (in terms of resource allocation) inputs that are being done first time. 
Communication Strategy: Plan and implement a well thought-out and adequately resourced communication (including knowledge management) strategy and plan for greater visibility, profiling and effective knowledge management. 
 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
Afghanistan

Region:
ROSA

Type:
Evaluation

Theme:
Emergencies

Partners:
Save the Children

Language:
English

Sequence #:
2016/001

New enhanced search