Evaluation database

Evaluation report

2010 Pakistan: Inter-Agency Real Time Evaluation of the Humanitarian Response to Pakistan's 2010 Flood Crisis



Author: Riccardo Polastro (team leader), Aatika Nagrah, Nicolai Steen, Farwa Zafar

Executive summary

Background:

This is the report of the Real Time Evaluation of the International Humanitarian Community’s response to the 2010 Floods in Pakistan. The evaluation was commissioned by the Inter‐Agency Standing Committee (IASC), funded by OCHA and undertaken by a team of four evaluators between January and March 2011. The team visited Pakistan two times.

The first mission took place in January to undertake an extended field visit in three of the worst affected Provinces to interview aid providers and aid recipients as well as to at federal level, observe patterns of the response and collect evidence (a teleconference was organised with representatives from Balochistan). An initial debriefing was held with the HCT to present preliminary findings. Within two weeks a draft report was shared.

In mid‐February, during the second visit, three provincial and a national workshop were held with key stakeholders involved in the humanitarian response to the floods. Findings, conclusions and recommendations were initially presented by the team leader during the workshops. Then, stakeholders jointly validated and prioritized recommendations and defined the organization(s) responsible to implement them (by whom) and timelines (by when). The main changes in the formulations resulted from group discussions. This process contributed to boost the ownership of the evaluation recommendations and fostered real time learning among stakeholders engaged. Once workshops ended the Resident Coordinator/Humanitarian Coordinator agreed that the HCT would draw an implementation plan of the recommendations outlined below.

Following the second visit to Pakistan, headquarter debriefings were held in with IASC representatives Geneva and New York.

As this participatory and utilization focused approach is new to Inter Agency Real Time Evaluations a lessons learned exercise on the process will be held in Geneva in mid April so that in can be integrated in future IA RTEs.

Objectives and Methodology:

1. This evaluation is the ninth Inter Agency‐Real Time Evaluation (IA‐RTE) conducted for the Inter Agency Standing Committee (IASC)12 of the current series and the fourth IA‐RTE to be rolled out in Pakistan. An IA‐RTE is an evaluation that provides immediate feedback in a participatory way to those executing and managing the response. IA‐RTEs seek to unlock operational challenges and provide real time feed‐back for both immediate corrective action and more system‐wide institutional learning.
2. The UN’s Office for the Coordination of Humanitarian Affairs (OCHA) managed and funded the evaluation on behalf of the IASC. The Terms of Reference (ToR), which define the objectives and tasks of the evaluation team, are presented as Annex I. Between January and February 2011, a team of four consultants working for DARA13, an independent organization specialized in humanitarian evaluation, conducted this RTE. It is the fourth IA‐RTE and the second in Pakistan that DARA carried out.
3. The evaluation commenced with a home based in‐depth documentation review. Upon arrival, the team travelled directly into the field, following the river Indus from Karachi to Peshawar visiting Sindh, Punjab and KPK Provinces to grasp operational response realities on the ground. Balochistan could not be visited due primarily to security constraints as well as time restrictions. Field visits were complemented by meetings with a range of stakeholders in Islamabad.
4. An initial feedback session was organized in both Multan (Punjab) and Peshawar (KPK) at the end of each of field visit, and an overall debriefing session was held at the Humanitarian Country Team HCT at the end of the field mission to present initial findings and conclusions. Workshops will be carried out in Pakistan in mid February both at provincial and federal level to further validate the findings of the report and draw Specific, Measurable, Accountable, Reasonable and Time bound (SMART) recommendations and draw a process action plan.
5. This report is structured according to the different dimensions of the IA‐RTE Framework as outlined in the ToR, namely context, needs assessments, funding, response, coordination, connectedness and cross‐cutting issues. Each dimension is subdivided into a number of key issues addressing the questions outlined in the ToR.
6. The evaluation followed a deductive analysis based on a mixed methods approach for data collection. Data collected has been entered into an evidence table organized according to key issues outlined in the ToR14. The findings of the evaluation are based on:
 A desk review of key documents, web‐pages and other relevant publications – references are outlined in the bibliography in Annex 8
 Semi‐structured individual interviews (interview guide for semi structured interviews – Annex 5) and group interviews with some 1,107 key stakeholders of which 421 were carried out with representatives from the UN agencies and programmes, the Red Cross Movement, international and national non‐governmental organizations (INGOs), government at the central and local level, the military and donors. Annex 4 outlines the full list of people met, by organization, affiliation and the type of interview method used.
 Group interviews held with over 686 people from the affected population including people still displaced in spontaneous camps and people that had returned to their land annex 3.1 outlines the list of camps and locations visited and the number of affectees interviewed at each.
 Direct observation of coordination process both at district, provincial and federal level as well as the outcomes of relief and recovery responses.
 Field visits to 20 different locations in three different provinces and the Federal Capital.
The locations visited include two provincial capitals and 11 districts. An overview of localities visited can be found in annex 4
7. The evidence collected was used as basis to draw conclusions and recommendations. To the degree possible, the evaluators triangulated data and drew on multiple sources to ensure that findings could be generalised and were not the result of bias or views of a single agency or single type of actor involved in the response.
8. Adopting utilization focused approach in order to boost the ownership of the process, conclusions and recommendations were validated, prioritized and organizations responsible to implement them and timelines were defined through three Provincial workshops and one national workshop (see annex 4). Following the workshops in Pakistan, headquarters debriefings were held with IASC representatives in Geneva and New York.
9. Evaluation Constraints
 Timing of the evaluation; although this IA RTE was fielded at an earlier stage of the humanitarian response to the epic floods disaster, as compared to the previous IA‐RTE done in Pakistan, the main relief efforts had been carried out between August and December (2010), prior to the visit of the RTE mission in January 2011. When the evaluation team visited the affected areas, small pockets of relief operations were still ongoing, but the majority of organisations were preparing, and some undertaking, early recovery activities;
 Staff turnover – especially among surge capacity, with rare exceptions this meant that people that dealt in the initial phase of the response could not be interviewed;
 Limited time for fieldwork; the team spent only three weeks in Pakistan. While the team divided itself at several locations to maximize coverage, the numbers of locations visited were still relatively limited compared to the geographical spread of the disaster;
 Security: In Pakistan, security represents a major concern were humanitarian actors operate. The team was required to travel under police escort.

Conclusions:

1. Funding for other non life‐saving and early recovery activities was slower and funding commitments were lower.
2. Currently in Pakistan the UN has two stand alone appeals; the PHRP and PFRERRP ‐ with limited funding. PFRERRP took PHRP out of the limelight.
3. Effectiveness, efficiency and accountability of some UN agencies & INGOs have been questioned by donors, GoP and implementing partners. Broadly there is insufficient commitment to the aid effectiveness agenda.

Recommendations:

1. Funding for other non life‐saving and early recovery activities was slower and funding commitments were lower.
2. Currently in Pakistan the UN has two stand alone appeals; the PHRP and PFRERRP ‐ with limited funding. PFRERRP took PHRP out of the limelight.
3. HCT members will ensure that response is monitored and results shared.
4. The UN must reduce transaction costs.
5. UN agencies and IPs produce results commensurate to the level of funding received (i.e. through unit cost analysis).



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search