Evaluation database

Evaluation report

2011 Haiti: The Use and Impact of ECD Kits - Post Earthquake Haiti 2010



Author: Lisa Deters

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

 

Introduction

 

On 12 January 2010 an earthquake measuring 7.0 on the Richter scale struck the West Province of

Haiti, the poorest country in the Western hemisphere with a legacy of political instability and

natural disasters, with the epicenter less than twenty kilometers southwest of Port au Prince, Haiti’s

capital, triggering a large scale international relief operation in response to the tragic and massive

destruction. A relief to development operation that is currently ongoing beyond the one year postearthquake

in 2011.

This report aims to provide a better understanding of a specific niche response, the ECD response,

in post-earthquake Haiti 2010. It concentrates on the UNICEF ECD Kit, a dominating ECD

response, examining the efficiency, effectiveness, impact, relevance or appropriateness,

sustainability and the overall quality of the kit and its use. A primary objective of this report is to

identify the capacities and capacity gaps in the usage of the kit while an overarching focus examines

the overall potential of the ECD Kit via such a mass distribution of the kits during the first phase of

an emergency to assist the Ministry of Education (MoE), education authorities and relevant

stakeholders to provide and include ECD during the first six months to a year of an initial response.

During emergency situations young children are considered among the most vulnerable. Early

childhood experts, as cited in the world’s leading medical journal The Lancet 2007, advocate for

appropriate support for young children founded on evidence-based knowledge about the impact of

early experiences on long-term cognitive, psychosocial and neurological development. Therefore,

early childhood care, education and overall development are emerging as essentials in emergencies

reflected in the increasing support of early childhood programs and services by leaders in

emergency response as evidenced by UNICEF’s response with an unprecedented distribution of the

UNICEF ECD Kit in Haiti.

UNICEF distributed the ECD Kits through their partners through two time periods: first from 25

January 2010 through 4 April 2010 1,566 ECD Kits were distributed followed by a distribution of

1,051 kits from 16 June through 28 October, 2010. In total, there are 2,617 ECD Kits that are

accounted for as having been delivered to partners. This distribution of a large quantity of ECD

Kits, demonstrated the commitment spearheaded by UNICEF to include ECD on the emergency

agenda following the earthquake disaster and it presented a unique opportunity to investigate the

issues related to ECD investments and interventions in emergency situations.

The report evidences many positive and increasingly dynamic developments within ECD in

emergencies. There has been a growing recognition by a range of stakeholders of the importance of

ECD in a humanitarian response, as evidenced through the formation of the ECD Working Group

(ECDWG). The report also identifies an important base of existing capacity and expertise across the

agencies interviewed for the study.

An emerging finding from the study is a lack of a shared understanding of the common elements of

an ECD in emergencies (ECDiE) response. At the country level, the overall low staff capacity was a

challenge. This was reinforced by three factors: that most agencies involved in the education were

focused on primary education provision; that there was often insufficient coordination and

programme planning across all ages and stages and phases of an education response; and that

agencies have tended to focus on immediate and short-term results. Though in a context with

limited resources, developing capacity cannot just be about individual agencies bringing their own

capacities to scale, rather, focus must be on leverage between stakeholders, including with the

relevant sections of the ministry of education (MoE).

 

Notwithstanding, major capacity constraint with regards to capacity building and accountability are

identified. Due to various methodological constraints it has not been possible to quantify these gaps.

Many of the constraints are associated with larger global challenges for ECDiE. The report seeks to have

relevance beyond ECD advocates and provide practitioners with practical feedback and support.

 

Brief research aims

 

The research intended to explore the emergency relief responses for young children, in particular

the heavily distributed UNICEF ECD Kit, while maintaining an overall focus on the implications of

ECD in emergencies recovery practices and any emerging policies.

1. Evaluate the use of the ECD Kits (in example: efficiency, effectiveness, impact, relevance

and appropriateness, sustainability).

2. Explore the overall implications of focusing on young children in the early phases of an

emergency.

3. Investigate the international and national responses towards young children post-earthquake.

The research study used the UNICEF ECD Kit as an entry point to support two areas of interest, a

focus on the kits as well as an investigation regarding the ECD investments and interventions in

emergencies in general.

The study was grounded in a holistic perspective, acknowledging the value and interplay of

institutions or governments, communities and families in child development, to address the use and

impact of ECD Kits and concomitant programs, services and supports for young children and

families.

The study’s design ensured the principles of ethical research by building local capacity through the

training and use of local data collectors. Through the use of the local data collectors, the goal

remained for 200 sites, the majority with the ECD Kit, to be assessed by the open-ended

questionnaire. Also included was the use of stakeholder interviews to gain insights into what other

programs, services and supports were supporting young children. As proposed in the study

proposal, thirty stakeholder interviews were desired; however, this number was halved with sixteen

interviews formally occurring.

 

Data collection

 

The field research was completed in two phases. The first period was for three months in Haiti from

13 October 2010 through 15 January 2011 followed by a shorter field visit for two weeks at the end

of July to beginning of August 2011. The data collection focused on three methods:

1. Open-ended questionnaires on the ECD Kits (goal – evaluation of 200)

2. Semi-structured interviews (with local and international representation)

3. Focus group discussions (with local communities)

Despite the challenges of conducting research in a fragile context, with the logistical support of

UNICEF HCO, the data collection period resulted in the completion of 204 open-ended

questionnaires, 16 semi-structured interviews with stakeholders in ECD and three focus groups

comprised of the facilitators and guardians of young children with exposure to the ECD Kit.

The open-ended questionnaire targeting the UNICEF ECD Kit had been designed in collaboration

with the Consultative Group on ECCD (CGECCD), UNICEF HQ NYC ECD in Emergency Unit

and the UNICEF Copenhagen Supply Division input. Additionally, it was reviewed by some

UNICEF HCO sectors involved in the ECD Kit distribution to the partners, predominantly nutrition,

child protection and education as well as the M&E sector of UNICEF HCO. The stakeholder semistructured

interview questioning route was also reviewed in a similar fashion.

Data via the open-ended questionnaire was collected in the greater Port au Prince (PaP) area within

the twenty communes that were affected in the West Department: Cabaret, Cite Soleil, Croix des

Bouquets, Tabarre, Carrefour, Grand-Goave, Petit-Goave, Leogane, Gressier, Delmas and Petion-

Ville. The greater Port au Prince area was chosen predominantly for a pragmatic reason; although,

ten communes were heavily affected in the Southeast Department with the worst damage in Jacmel,

due to time constraints with the data collectors by the time that they were identified, hired and their

short-term contracts were settled upon, they were employed for five working days to collect the data

and given their allotted transport budget of 750 gourdes per day by UNICEF HCO, they had neither

the time nor the funding to cover the Southeast Department inclusive of Jacmel as it would have

required overnight stays incurring extra costs and time. Therefore, the UNICEF partners who had

received ECD Kits and were working within the greater Port au Prince area were included and

dominate the study.

In an effort to build local capacity and to complete the scope of the work, one research assistant and

ten data collectors were hired locally. UNICEF HCO identified and recruited the research assistant

and all ten data collectors, of which the ten data collectors were sourced through the snowball

effect. As the research assistant, Mr. Rhodner J. Orisma, is himself a doctoral candidate at a

university in Florida as well as a professor at a Haitian university instructing in methodology

courses, Mr. Orisma assisted UNICEF Human Resources (HR) with identifying potential data

collectors supplementing their snowball effect in identifying qualified and experienced data

collectors. Through his contacts, two of the data collectors were identified. The ten data collector

roster includes four females and six males: Samuel Revolus, Renan Anglad Charles-Cius, Guersley

Chery, Jacqueline Baptiste, Mirlene L. Percinthe, Rodady Gustave, Martine Delisca, Patrick

Laurenceau, Marie Sonide Dorilma and Reginald Hellande Jeune. All ten data collectors claimed to

have experience in data collection either from university coursework or previous positions with

international non-government organizations (INGOs) working in Haiti. The new learning from this

study was regarding the briefing on the ECD Kit and ECD in general. The data collectors were each

given a specific area within greater PaP for which to be responsible for covering; although, due to

their difficulties in accessing and locating sites due to movements, transitions and closures of

certain programs since almost a year post-earthquake, the collectors predominantly targeted their

goal of completing twenty open-ended questionnaires and in doing so some moved around into

departments with a great concentration of partner programming for the kit usage and easier to move

around such as Carrefour, Delmas and Petion-ville.

This study heavily relied upon the collaboration with the locally hired research assistant for

translation and interpretation needs. The research assistant served to translate the data collection

instruments for this research and training materials into French and Haitian Creole. The assistant

acted as an interpreter during the full day training on qualitative research theory and practice as well

as on the implementation of the questionnaire for the ten data collectors.

Once the data collectors were hired, a full day training was what the timeframe permitted to ensure

sufficient time for them to complete the data collection before the lead researcher’s date of

departure. It was led by the lead researcher at the United Nations Stabilization Mission in Haiti’s

(MINUSTAH) Log Base where the UNICEF HCO offices are currently located, provided the ten

data collectors with a comprehensive briefing on the UNICEF ECD Kit and an overview on

qualitative research inclusive of the validity and reliability of open-ended questionnaires,

appropriate documentation practices in terms of photography and personal observations, use of key

informant interviews and the ethics of research stressing consent, confidentiality, reciprocity, doing

no harm and dignity in documentation. The training also provided the collectors an opportunity to

familiarize themselves with the main data collection tool, the open-ended questionnaire, and an

opportunity for practice and role-play.

After which, the data collectors each were each assigned twenty open-ended questionnaires to be

responsible to complete within their terms of reference over the course of five working days

scheduled over 17, 20-23 December 2010, some completed them in five consecutive work days and

others due to the challenging circumstances of informal and formal preschools and other

educational centers and programs for young children being closed for seasonal holidays in

December 2010, spread the five days over a few weeks to gain access to certain centers and

preschools once opened again in January 2011.

On the 27 of December 2010 the first data was submitted and the research assistant, employing the

assistance of two individuals who were privately sub-contracted by him, began the work on the

English translation of all of the 202 questionnaires from Haitian Creole and French. By 7 January

2011 all of the data had been submitted and all of the collectors had been paid for the completion of

their work.

During the time span of the fieldwork, the lead researcher continually was in contact with any

organization known to be supporting or working with young children to access a spokesperson

comfortable to discuss their work in a semi-structured interview. A total of sixteen formal

stakeholder interviews were completed including: UNICEF, Concern Worldwide, PLAN, World

Vision, ACF, Tipa Tipa, UJDHRD (Union des Jeunes pour le Developpement Human Reel et

Durable), GJARE (Youth Action for Reform), OCCAARH (Organization des Cityoyens

Conséquents pour l’Avancement d’Haiti), CAFT (Centre d’Apprentissage et de Formation Pour La

Transformation), Finn Church Aid/ACT, Digicel Fondation, Kindernothilife, Save the Children,

BUGEP (Bureau de Gestion du Préscolaire) and Compassion International. The interviews were

recorded when permission was granted and the transcriptions of the interviews are in the process of

being completed for further validation and analysis.

 

ECD Kits: Haiti study

 

For the purpose of this study, ECDiE is broadly defined as prioritizing the holistic needs of young

children ranging from age 0-6 years. UNICEF (2010) has drafted an Integrated Quality Framework

for ECDiE which provides a globally recognized detailed and comprehensive definition of ECD:

“Early childhood development refers to the processes through which a young child under

eight years develops his/her optimal physical health, mental alertness, emotional confidence,

social competence and readiness to learn. Through the process of development, the young

child gains the abilities to move from ‘less to more’ and acquires mastery of complex

skills...Holistic early childhood development programming integrates essential early

childhood interventions within the health, nutrition, water, sanitation and hygiene, education

and child protection sectors. This enables the young child to claim his/her rights to survival,

growth, development, protection and participation and ensures that the duty bearers, namely,

parents, caregivers, communities, sub-national and national authorities respect, protect,

promote and fulfil those rights.” (p. 6)

Early childhood development and education remains at the forefront of the education in emergency

agenda. During fragile situations whether by an emergency or disaster, young children are

considered the most vulnerable and must be provided with essential and adequate care and

education. Beyond the initial phase of instability, young children must also be provided with

appropriate care and education to address their holistic developmental needs. Early childhood

experts continually advocate for appropriate supports for young children in any fragile situation

well founded on evidence-based knowledge on cognitive child development. Regardless of the

situation or context, education is established as a fundamental human right; furthermore,

development is recognized universally to be a fundamental right for the young child. Quality early

childhood development, care and education are essentials in the field of education in emergencies as

early childhood is an effective entry point from which to address many social issues.

What remains a challenge is to identify best practices supporting the needs of young children, their

caregivers and their communities through rigorous monitoring and evaluation. Moreover, what

remains to be considered is how best to create a sense of agency and scale-up the involvement of

young children, families and caregivers and opportunities for social change to inform the

interventions and supports of the relief community. With ever increasing fragile situations in the

world, there is a push to identify effective tactics and responses to address the needs of the young in

a crisis. Amongst the international relief community, UNICEF was the first to launch an ECD kit in

July 2009 after piloting the kit in seven countries – Chad, Liberia, the Republic of Congo, Jamaica,

Guyana, Maldives and Iraq. Since then, the kit has been heavily used in the most recent

emergencies such as after the tragic January 2010 earthquake in Haiti.

At the official launch of the ECD Kit in Geneva in July 2009 UNICEF’s current Executive Director,

Ann M. Veneman, expounded on how “the Early Childhood Development Kit is a tool for young

children displaced or affected by war and natural disasters. It is the first of its kind within the

humanitarian community.”2 Furthermore, subsequently she has stated that, “it is essential to ensure

resumption of normal childhood activities without delay, even in an acute phase of an emergency.”3

 

In considering this surge to pre-position early childhood development material, “early childhood

development play and recreation material (e.g. UNICEF ECD kits) which is responsive to the needs

of young children should be pre-positioned for use in communities, baby clinics, nutrition centres,

child friendly spaces etc. When it is not possible to pre-position material, local procurement

arrangements should be made. This may be done either by identifying local suppliers of similar

products or by involving communities in preparing culturally relevant toys and games for young

children” (UNICEF, 2010).

 

Establishing the research context

 

On February 18-19, 2010 the EEWG Research Task Team4 met at UNICEF headquarters (New

York) and developed specific goals for the early childhood in emergencies agenda. At that time

UNICEF identified some key areas for investigation regarding the ECD response in Haiti, and

specifically the use of the ECD kits. This study was a result of such meetings to address such

concerns.

Thus, the catastrophic earthquake that struck Haiti on 12 January 2010 killing over 220,000 people,

injuring 300,000 and leaving over one million people homeless, provided an opportunity to

investigate the responses in consideration of a positive emergency.5 Apart from the loss of lives and

properties, infrastructures and complete governing bodies were also damaged or destroyed, with

recovery and repair hampered by the following hurricane season and the outbreak of cholera.

Presently, the cost of reconstruction is estimated at $11.5 billion USD. Prior to the earthquake, Haiti

was already the poorest country in the western hemisphere ranking 149 out of 182 countries on the

 

2 http://www.unicef.org/

 

3 http://www.unicef.org/

 

4 This proposal was developed by the Research Team at Macquarie University following a meeting of the EEWG

Research Task Team in February 2010, and with input from UNCEF personnel. The proposal incorporates research

questions contributed by the Consultative Group and Dr. Fabienne Doucet as indicated.

 

5 This term has been used in various personal communications amongst the humanitarian community in light of

emergencies providing an opportunity to essentially build back better.

 

Human Development Index 2009 with ineffective social services, a weak civil society and political

unrest (Report for the Disasters Emergency Committee, 2011). The disaster has impacted Haiti on

an unprecedented scale and its after-effects will impact Haiti’s development for decades to come.

In response to the earthquake in Haiti 2010, in the acute phase of the emergency, UNICEF had

distributed over 1,500 ECD Emergency Kits. As of April 2010, 1,546 kits have been recorded as

distributed, an additional 3,341 kits were to come from UNICEF Supply Division, based in

Copenhagen, from which 1,104 kits were to be sent to Haiti with the remaining 2,237 kits scheduled

to arrive during the end of May-June, 2010 (UNICEF Haiti Situation Report, April 2010).

Additionally, the UNICEF Latin American and Caribbean Regional Office has provided these

numbers: 200 ECD Emergency Kits have been locally procured in Panama as a first step to assess

regional procurement and the printing of 600 copies of the ECD guide in French for the UNICEF

Haiti office (personal correspondence). This extensive and unprecedented dissemination of the ECD

kit during the acute emergency response coupled with a long term commitment by UNICEF and

others for post-earthquake recovery provides an unparalleled opportunity to research the issues,

capacity, support, development and impact of services and programs which target young children

within and beyond emergency situations.

 

Purpose of study and methodology

 

This report is based on a compilation of information from secondary data sources, observations and

informal discussions during two field visits to Haiti. The process began with a desk review of

existing ECDiE literature and continued with a situational assessment of ECD in Haiti. Efforts were

made at all stages in the study to triangulate information through the varied methodology,

compiling both quantitative and qualitative data.

During the first visit, several stakeholder interviews were completed, site visits and observations

and a focus group. The following visit resulted in two additional focus groups. It is important to

note that all attempts were made to avoid assessment fatigue amongst the communities and adults

engaged in this study; a widespread occurrence due to the large scale international intervention

post-earthquake. Both field visits were completed in an effort to gain a better understanding of:

 

- the overall situation and the impact of the earthquake

 

- the response of the relief and humanitarian organizations’ efforts towards ECD

 

- the community perspective on children’s immediate needs, focusing on the very young child

(aged 0-6)

 

- gaps in provision

 

- document lessons learned and good practice

The agencies interviewed for this report were drawn from a compilation of logistics lists from

UNICEF HCO in addition to sectoral input from the involved sectors with the kit such as nutrition,

child protection and education predominantly. While organisations that comprise the evidence base

for this report by no means represent all those with ECDiE capacity at international level, they do

make up the majority within Haiti. Therefore, research for this report also incorporated other

stakeholders, internal to Haiti but not engaged with the ECD Kits and externally active advocates

and leaders within ECD, in order to develop a more comprehensive sense of the global ECDiE

capacity and direction.

Finally, the overall study has benefited from the expert advice of a panel of four peer reviewers.6

6 Professor Jacqueline Hayden, Dr. Emma Pearson and Dr. Kathy Cologon of Macquarie University provided oversight

and input to the design of the research study, the analysis and final report. In addition, Dr. Fabienne Doucet of New

York University also provided expertise and guidance.

 

Study constraints

 

The research process encountered two main constraints. First, in terms of information collection,

efforts were made to design survey questionnaires that could be used across different stakeholder

groups. Partly due to this, and partly due to the burden of information collection requirements,

survey responses differed considerably in terms of the questions answered and the amount of

information provided. As a result, parts of this report are based on quite limited responses. This led

to further difficulties in attempting to quantify gaps.

Secondly, the difficulty of conducting the study against the backdrop of a highly fragile and volatile

environment presented extreme weather conditions, a cholera outbreak and political and social

unrest that impeded the research to be conducted in a prompt and planned fashion.

There were a host of other challenges in the various stages of the data collection. Many of these

were in concern to the loss of institutional knowledge from the frequency of emergency staff

turnovers at both UNICEF HCO and their partner organizations. Therefore, simple mapping tasks

of who had received a certain number of ECD Kits and the specific sites for where they had been or

were being used were complicated due to the loss or movement of emergency personnel,

international and national, and their knowledge of the responses in the acute phase of the

emergency, making the initial task of securing and then maintaining updated records of who was

responsible for or working in and on ECD amongst the partners a challenge.

New ECD management and staff months post-earthquake when emergency personnel phased out

for new permanent staff posts, typically resulted with individuals new to Haiti and some to

emergency programming. This exemplified an issue with organizations’ staff continuity, which

potentially reflects in program discontinuity. It is duly noted that with emergency contexts, beyond

the staff movements, the responses in terms of distribution, coordination and monitoring are a

daunting task, particularly post-earthquake in Haiti, with a weakened government and destroyed

buildings and infrastructure, making the mapping and processing of reactive responses a continued

challenge.

When reflecting upon the ECD Kit response, it initially was a mass distribution to UNICEF HCO

partners through the five UNICEF sectors: nutrition, education, child protection, health and water

and sanitation (WASH). UNICEF HCO has some records, though not comprehensive, from their

supply section as to who received kits and how many kits were received. From there, efforts were

focused on where those kits were sent and used by the partners. For large partner organizations that

received over a hundred kits, keeping in mind emergency personnel shifts as well as movements

amongst internally displaced people and adaptive programming in shifting tent cities, there was not

always a specific address to be given or concrete knowledge to where the kit was being used or kept

in the field. Therefore, for the data collectors it was extremely difficult to first locate a tent city or

camp and then to arrive only to find that the programming for young children had never existed, had

closed or had moved despite the information provided by an organization’s central office in PaP.

 

Structure and contents

 

This report takes on an evaluative conceptual framework to understand how the ECD Kit measures

up in terms of efficiency, effectiveness, impact, relevance and appropriateness, sustainability and

quality and accountability.7 The following sections consider the relative strengths, weaknesses,

opportunities and constraints of the ECD Kit and its usage within the Haitian context.

 

7 The study strives to review the data in light of an adapted adherence to the Organisation for Economic Cooperation

and Development (OECD) Development Assistance Committee (DAC) Principles for Evaluation of Development

Assistance criteria (ALNAP, 2006).

 

The report ends by drawing together a series of recommendations with regard to the future potential

of using the ECD Kits and to the focus on ECD during the first phases of an emergency.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search