2018 EO: Evaluation of UNICEF's Response to the Rohingya Refugee Crisis in Bangladesh
Author: Ed Schenkenberg, Richard Luff, Anne Bush, Francesca Ballarin, Laura Olsen
BACKGROUNDCox’s Bazar District, one of the poorest areas of Bangladesh, has been the scene of several mass influxes of the Rohingya Muslim minority from Myanmar, including in the late 1970s and early 1990s. Since mid-August 2017, a record number of more than 700,000 refugees have arrived. Adding this number to those Rohingya who arrived before August 2017, it is estimated that there are approximately 900,000 refugees in camps and settlements. As the presence of these refugees have put the environment and local residents under huge strain, the United Nations has estimated that 1.3 million people are in need of urgent humanitarian assistance, including critical life-saving interventions. Some 703,000 of them are under 18 years old.
At the request of the Government of Bangladesh, several United Nations agencies, together with their government counterparts, international and national non-governmental organizations (NGOs), local civil society groups and others, began to respond to the urgent needs of the Rohingya and their host communities. UNICEF reacted to the influx immediately, announcing that it would scale up its response, and assumed responsibilities in nutrition, health, water, sanitation and hygiene (WASH), child protection and education. The organization formally activated a Level 3 emergency response on 20 September 2017. It also published a Humanitarian Action for Children (HAC) appeal and a revised response plan in October 2017 to address the immediate and urgent needs of affected Rohingya children, women and adolescents. The HAC appeal outlined how UNICEF would reach 716,000 people with interventions in nutrition, health, WASH, child protection, education and Communication for Development (C4D)/community engagement and accountability.
This evaluation, which was commissioned and managed by the UNICEF Evaluation Office in New York, was conducted between March and October 2018 by five independent consultants with expertise in all programme areas under assessment, with remote and onsite support from the Evaluation Office.
PURPOSE & OBJECTIVES
The primary purpose of this evaluation is to generate lessons to improve the ongoing response. The secondary purposes of this evaluation are to strengthen UNICEF’s accountability and to assist UNICEF and the broader international humanitarian community to better understand how to respond in situations of rapid mass and forced displacement and settlement.
In line with these overall objectives, the evaluation has worked towards three specific objectives:
- 1. To assess the adequacy of the UNICEF response in providing humanitarian assistance to vulnerable people who reside in camp settings and are integrated within Bangladeshi communities and in host communities;
- 2. To determine how well UNICEF is working with implementing partners, other agencies and the Government, for both the near- and medium/long-term; and
- 3. To identify actions to improve the response.
The team used Real-Time Evaluation Plus, a new approach for UNICEF that combines elements of a retrospective and formative evaluation with those of a real-time evaluation, with the intention of delivering findings and conclusions in a short timeframe. The basis of this approach, is informed by some of the Organization for Economic Co-operation and Development-Development Assistance Committee criteria (i.e., relevance, appropriateness, connectedness, coherence, coverage, effectiveness, efficiency and impact), and also borrows from other types of exercises and data collection tools. This new type of evaluation looks at the past to understand the course of events and the history of a response. At the same time, it involves direct observation and consideration of future scenarios and planning.
The analytical framework used for the evaluation combined qualitative and quantitative evidence organized around the evaluation matrix that was developed during the inception phase. The data collection phase consisted of a) documentary review; b) key informant interviews with current and former UNICEF staff, implementing partners, sister United Nations agencies, government stakeholders and donors; c) focus group discussions with affected populations and community volunteers; and d) two online surveys for UNICEF’s sector and implementing partners. It included a mission to Cox’s Bazar, Dhaka and Kathmandu in April 2018. The team leader returned to Dhaka in June 2018 for a workshop with staff from the regional, country and field offices to discuss the preliminary findings, conclusions and recommendations.
FINDINGS & CONCLUSIONS
Overall, the evaluation team found that, despite the enormous challenges posed by this crisis, UNICEF rose to the challenge. The organization’s services reached many Rohingya children and their families, which, undoubtedly, addressed their plight. The evaluation notes a range of impressive achievements in areas such as scaling up, advocacy, sector leadership, and, of course, service delivery. However, the evaluation also found critical challenges and identified several areas in need of improvement. The team understands that many of these improvements were set in motion in the weeks and months following the data collection phase, but these actions fall outside the timeframe under examination. In terms of the evaluation’s task to generate lessons, it has produced a number of specific findings.
Preparedness and scale-up
The entire international community, including UNICEF, was caught off guard by the size and speed of the refugee influx. Despite strong indications of massive human rights violations in Rakhine and the lack of humanitarian access, the lack of information from Myanmar about a potential massive outflow meant that UNICEF was under-prepared for such a crisis. Despite this, UNICEF’s scale-up efforts were impressive. Strong and experienced leadership, some appetite for risk, smooth contracting processes and strong human resource management contributed to the rapid scale-up. However, limited partner capacity hindered UNICEF’s ability to keep pace with the refugees’ needs.
From the start of the crisis, UNICEF has appropriately focused its advocacy on three key obstacles to this response: the weak protection environment; the extreme congestion in the camps; and the problematic inter-agency coordination model. The evaluation found that UNICEF’s advocacy for a better protection environment should have been stronger, given the organization’s normative role and its rights-based agenda. It rightly raised the issue of the extreme lack of space in appropriate fora, though it should consider documenting the consequences of congestion on its ability to respond to facilitate more evidenced-based advocacy.
The evaluation found that coordination structures that combined different coordination models caused confusion, delays and unnecessary agency tensions, and negatively impacted UNICEF’s ability to deliver for children. UNICEF was right to raise this issue and the evaluation found that the organization was a strong advocate for improved coordination, though it should have formally raised its concerns at the IASC level. Concerns about the response’s coordination extend beyond this crisis. Crucial to the effectiveness of UNICEF’s coordination responsibilities in nutrition, WASH, child protection and education is to clarify how the refugee coordination model, led by UNHCR, will be implemented in the future. UNICEF should actively engage in discussions about the future of the inter-agency coordination structure in refugee responses.
Although UNICEF did attempt to define its strategies, especially at the programme level, its overall strategy was found wanting. First, the rights lens was weak. All humanitarian organizations involved in a refugee response should structure their work around refugee rights, which should in turn underpin all aspects of the response. Second, the overarching strategy should have addressed the significant gender dimensions of this crisis. There was also insufficient reference to how UNICEF would address protection risks such as abuse, exploitation, trafficking and gender-based violence.
The overall strategy also neglected how the context, including congestion in the camps, would impact the organization’s ability to deliver and how UNICEF would adjust its programming accordingly. In addition, while the evaluation team recognizes that UNICEF has a wide range of responsibilities and commitments, the context meant that not all priorities could be implemented simultaneously, and as a result, some HAC targets were not reached. While there was tacit prioritization, UNICEF should have been more explicit about this and the associated sequencing. The organization would have been entirely correct to manage the expectations of its stakeholders by sequencing its response activities without infringing on its mandate. This would have helped other agencies define their own priorities, either in support of or complementing UNICEF’s approach.
The overall strategy should have better articulated inter-sectorality, a term that signifies strong links across sectors, which should work together in combining their services. In particular, the education-in-emergencies approach, which provides an opportunity to work across sectors, was weak.
Finally, although UNICEF’s strategy could be discerned from a collection of various materials, the evaluation team did not find a document that articulated UNICEF’s overall strategic vision, its main plans in service delivery, its advocacy and communications work, the technical support it might need from the regional office and headquarters, and the technical support that it would provide to partners. The HAC and the Response Plan appear to be more a collection of sector-specific work plans than a strategy, and were designed to communicate UNICEF’s funding requirements. UNICEF should reflect not only on the content of its strategy but on how to better articulate its overall strategy in future emergency responses.
In terms of programme strategies, the findings are more positive. Programme strategies were highly relevant and there is ample evidence that appropriate adjustments were made to address emerging and evolving risks. The concentrations on the prevention of disease outbreaks, improving WASH conditions and addressing the nutritional status of the Rohingya was appropriate. Gaps included the inadequate attention given to gender, UNICEF’s failure to integrate the critical life-saving aspects of an education-in-emergencies approach, and the inadequate attention given to adolescent education. The Fecal Sludge Management (FSM) strategy is somewhat vague and underestimates the scale and complexity of this area of work in this unique context.
Rights, protection, gender and gender-based violence
UNICEF did not sufficiently emphasize addressing protection, mainstreaming gender and addressing gender-based violence issues – a significant concern given the organization’s commitments to these priorities. An initially assistance-driven operation should have been quickly followed, if not accompanied, by a protection-orientated, rights-based response. Indeed, the quality aspects of the CCCs and Sphere and companion standards derive from the fact that they are rights-based. With some exceptions, gender mainstreaming aspects were insufficiently considered and implemented in the first several months. Programming to address gender-based violence was critically delayed. While plans were eventually made to address this, the evaluation found clear evidence of the lack of implementation of gender-based-violence-related services at least until February 2018. In UNICEF, gender-based violence falls under the responsibility of child protection, an arrangement that, in a crisis like this, doesn’t give it the attention it requires.
Effectiveness, coverage, timeliness and quality
The evaluation found that UNICEF has reached many of its programme targets against stated objectives in each of the five sectors it assessed, though some areas of work did fall behind. Due to the extreme speed and scale of the influx, the evaluation understands that quantity was prioritized over quality in the first weeks and months of the response. This approach was appropriate to reaching affected populations. But quality must follow quantity, and this did not happen across all areas of work. Some of the reasons for this were outside of UNICEF’s control, for example, the extreme congestion, the speed of the influx and the almost non-existent infrastructure. Other factors included gaps or delays in recruiting key staff positions, lack of implementing partner capacity and inter-agency tensions. In April 2018, the implementation of several priorities that should have been well underway had only just begun.
The response would have been more effective had inter-sectorality been better addressed. The push for greater integration needs to come from the top. The evaluation found that the CCCs, which guide how UNICEF responds in all emergencies, include little guidance on inter-sectorality.
The evaluation found that C4D, a service unique to UNICEF, has had an added value in the context of several UNICEF programmes. C4D also found its role and place by actively contributing to the CwC Working Group. While this may be seen as positive, it leaves the question open as to the value of C4D in relation to sector-wide initiatives to engage in a dialogue with affected populations.
There is a mixed picture of UNICEF’s performance in regard to its (sub-)sector (co-)leadership, which resembles the different levels of progress that the evaluation has seen within and among the programmes. The factors that could contribute to better sector leadership range from ensuring continued senior staffing (nutrition) and pushing for inter-sectorality (child protection and education), to ensuring a collective and genuine partnership approach in which UNICEF contributes to but does not dominate the sector (education).
While many of UNICEF’s partners were positive regarding UNICEF’s lead role in the sector, the evaluation also found that in some cases, the sector was dominated by UNICEF, while in other cases, UNICEF programmes were too distant from the sector. Finding the right balance is challenging and requires a collaborative spirit and open dialogue.
UNICEF was found to have strong relationships with its partners and many of these partners expressed positive views of UNICEF’s contributions to the partnership. The evaluation found evidence that (national) NGOs appreciated UNICEF’s sector leadership and their contractual relationships with UNICEF. That said, the NGO capacity in several programme areas was found wanting, something that not all sectors anticipated in time. The absence of UNICEF’s traditional partners in Cox’s Bazar contributed to this. The capacity-building effort, which should also be done in collaboration with other agencies and the ISCG, should pay attention to national/local NGOs’ understanding of the (international) legal frameworks, rights and standards and mechanisms that offer protection to Rohingya children.
The quality of information
The evaluation found that parallel data collection systems that are not necessarily compatible do not allow for easy comparison between the work of UNICEF programmes and the sectors. Much of this issue relates to the perennial problem of the division between individual agency responsibilities and collective arrangements in information gathering and coordination. The evaluation also found that data collection is too focused on coverage and the number of people reached and not enough on quality. Finally, advice from visiting Regional and HQ staff was not always adequately absorbed.
The human resources, supply and funding functions generally supported the response well. UNICEF deployed a significant number of staff from the Dhaka office, other Bangladesh field offices, and from its global surge capacity in a timely manner. However, the evaluation observed an over-burdened field office and a growing disconnect (less collaboration and communication) between the office in Dhaka and the office in Cox’s Bazar. Funding was only a challenge in the early days of the response, which UNICEF overcame, thanks in part to HQ provided Emergency loans. While the supply function has been stretched, and could have benefited from additional surge capacity, particularly in the early part of the response, it has generally worked well.
UNICEF took monsoon preparedness very seriously and did what it could to put plans in place. In several ways, monsoon preparedness has helped to mobilize actions and accelerate steps that have contributed to an improved response overall.
The recommendations presented below follow from the evaluation’s findings and conclusions. They outline the main priorities for improving UNICEF’s response to this crisis and, where relevant, UNICEF’s response to emergencies more generally.
1. Information and analysis for preparedness
The evaluation recommends that UNICEF invest in collecting better political, social and economic intelligence for forecasting to inform its preparedness actions. The intelligence should be cross-border (and, where necessary, cross-regional), include local context and, where possible, be shared with other agencies. It is also recommended that the intelligence be translated into risk analysis and preparedness plans. [For action by: UNICEF Regional Offices under the leadership of UNICEF Headquarters]
a) Raise the findings from this evaluation with the Senior Executive Group and the ISCG. Linked to other initiatives to strengthen coordination, UNICEF should work with the resident coordinator and the head of the ISCG to clarify lines of accountability and relationships, including the roles of sector leads with their home agencies and with the inter-agency coordination structures. [For action by: UNICEF Bangladesh and the Cox’s Bazar field office]
b) Share the relevant findings from this evaluation about coordination with the IASC and promote the inclusion of the future of the refugee coordination model on the IASC agenda. In this process, review accountability issues in this model and make use of the cluster approach experiences. [For action by: Office of Emergency Programmes (EMOPS)]
3. Context analysis for planning and advocacy
The evaluation recommends that UNICEF document the specific ways in which the congestion has impacted its ability to deliver and has ultimately denied Rohingya children and their families their rights. This work should inform UNICEF’s future strategies to respond to this crisis, be used to support UNICEF’s continued advocacy in this area, by providing a stronger position grounded in evidence. [For action by: UNICEF Bangladesh with support from the Regional Office for South Asia (ROSA)]
a) Review UNICEF’s strategy for 2019 and beyond. Ensure it includes an analysis of the context (in line with recommendation 3), identifies existing and potential issues and obstacles and explains how the strategy will address these. Be explicit about prioritizing and sequencing activities. [For action by: EMOPS, the Programme Division, ROSA and UNICEF Bangladesh]
b) Review how strategies for Level 3 emergencies are informed, developed and adjusted throughout a response. [For action by: EMOPS]
5. Rights, protection, gender and gender-based violence
a) Review UNICEF’s guidance on advocacy in emergencies. The review should consider UNICEF’s comparative advantage as an advocate for children in crisis contexts, how to maximize the relationship between operational response and advocacy, and UNICEF’s advocacy position in relation to other actors. [For action by: EMOPS]
b) Strengthen efforts to address protection risks, including gender-based violence. In so doing, it is recommended that the relevant offices strengthen and deepen inter-sectoral work among all programme sections and ensure attention to psychosocial support, children with disabilities and similar other risks and vulnerabilities. [For action by: UNICEF Bangladesh with the support of ROSA and the Programme Division]
c) Strengthen efforts to mainstream gender in all aspects of UNICEF’s response. Ensure gender is integrated across all sectors (and that all of the actions from the ISCG Gender Matrix have been implemented). [For action by: UNICEF Bangladesh with the support of ROSA and the Programme Division]
d) Develop a position on the relocation of Rohingya children to Bhasan Char island and their return to Myanmar from a normative perspective and ensure that this position is framed in a set of advocacy messages in coordination with other United Nations agencies. [For action by: UNICEF Bangladesh with the support of ROSA and EMOPS]
6. Positioning of C4D
Review the extent to which C4D fits (better) within the humanitarian community’s work on engaging with communities in emergency situations and assess the nature of investments needed. [For action by: EMOPS and the Programme Division]
7. Innovation, out-of-the-box thinking and next steps
a) Experiment with innovative ways of building the capacities of its partners, for example, by seconding staff members for financial management, peering and mentoring rather than training. This should include capacity building on protection and rights issues. [For action by: UNICEF Bangladesh with the support of ROSA and the relevant HQ Divisions].
b) Due to the layout of and congestion in the camps, the densely-populated space in the highly rural environment, and the initial lack of design for pit emptying, the issue of FSM is extremely complex. Working with the sector, UNICEF and other key stakeholders should experiment with new ways of addressing this issue by engaging the private sector and universities. [For action by: the Programme Division, the Supply Division, ROSA and UNICEF Bangladesh]
8. Integrated programming and working arrangements
a) The evaluation recommends that UNICEF undertake a light management review that would consider the reallocation of roles and responsibilities between the Dhaka and Cox’s Bazar offices and promote staff work across programmes. This review should also examine how an education-in-emergencies approach can forge closer programmes linkages through the education programme. [For action by: UNICEF Bangladesh]
b) Ensure that the revision of the CCCs looks at strengthening inter-sectorality and builds links between UNICEF programme areas. [For action by: EMOPS]
9. Knowledge management and data
a) Further invest in knowledge management. This could include developing a standard format for reports made by visiting advisers and setting up a system for monitoring the implementation of their recommendations or adapting the Emergency Management Team’s Action Tracker system. [For action by: the Emergency Management Team and UNICEF Bangladesh with the support of ROSA]
b) Review the commonalities and differences of the information and data needed at the programme level and the sector level and ensure that these datasets are compatible from the onset of data collection. [For action by: UNICEF Bangladesh with support from ROSA]
Read the blog on Rohingya refugee crisis — how effective is UNICEF’s response?
You will find the following labelled as attached:
- Evaluation report - Report
- Annexes - Part 2
- Evaluation Summary - Part 3
- Evaluation Review - Part 4
- Executive Summary - Part 5
Full report in PDF
PDF files require Acrobat Reader.