Evaluation database

Evaluation report

2010 Global: Inter Agency real time evaluation of the humanitarian response to Pakistan's 2009 displacement crisis



Author: John Cosgrave (Team Leader), Riccardo Polastro, Farwa Zafar

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:

This is the report of the Real Time Evaluation of the International Humanitarian Community‟s response to the 2009 and 2010 Internally Displaced Person (IDP) crisis in Pakistan. The Inter-Agency Standing Committee (IASC) commissioned the evaluation and it was undertaken by a team of three evaluators in May and June of 2010.

This was a humanitarian response set against the backdrop of a very complex environment, in which the Government of Pakistan (referred to as the Government from here on) reluctantly mobilised the military to reassert control of the national territory. Given that the primary driving force of the military operation was the re-establishment of sovereignty, and that underlying causes of the security problems included international factors, it was not surprising that the GoP constrained the response of international humanitarian organisations.

The overall response was a success in which the international community played a key role. There was no large-scale death or disease outbreak even though millions of people fled from their homes in a very short time frame. There were a number of innovative features in the response, including relatively rapid registration and verification and the use of smart cards for distributing cash assistance. The population of the IDP inflow areas, and of Pakistan as a whole, played a major role in the response as did the Government.

Donors gave generous support during the relief phase in 2009, but have been far less willing to fund work in 2010, either to meet recovery needs of returnees or of those still displaced.

The response to Pakistan again illustrates problems with the current UN Humanitarian Reform process. While the Emergency Response Coordinator (ERC) in New York appointed a Humanitarian Coordinator (HC) with solid humanitarian experience, the HC was still not a full-time HC. The Humanitarian Country Team (HCT) has not been as effective a forum for leading the international humanitarian response as it should have been.

Cluster performance in Pakistan has been very variable. Many clusters lacked trained full-time coordinators. Some of the same problems with cluster identified in the response to the 2005 earthquake and the 2007 floods in Pakistan were again evident in this crisis. The clusters have also suffered from unrealistic expectations regarding their ability to manage pooled funding. Field coordination rolled out very slowly, leaving coordination in the hands of the military for several months.

Pakistan is a dangerous environment for humanitarian workers, and insecurity has been a major problem for the humanitarian response. Security concerns have limited assessments and monitoring. Security rules have had large operational costs and have had a negative effect on programme quality.

The Government was simultaneously a party to the conflict and the gatekeeper for humanitarian assistance. The Government constrained access to the affected populations by humanitarian actors, partly for security and partly for other reasons. Access constraints in the form of government controls or agency security rules have completely shaped and constrained the humanitarian response in much of the affected area. The No-Objections Certificate (NOC), through with the authorities control access, system has been a major constraint in the delivery of humanitarian assistance.

Pragmatism and the need for access to the affected community, rather than the humanitarian principles of independence, neutrality, and impartiality drove the actions of the international humanitarian community. Different humanitarian actors compromised these humanitarian principles to different extents so as to get access to the affected population. Pakistan is a member state of the UN and it is a One UN Pilot Country. This means that the UN‟s programme in Pakistan is closely aligned with the Government priorities. UN agencies did not work in an impartial, independent or neutral manner, but instead strongly supported one party to the conflict, the Government.

The military‟s desire to create free-fire zones in the areas controlled by militants led to large-scale displacement as populations fled the areas when notified to do so by the military. The Government sought to repeat the success of the 2005 earthquake response, in which the military led the response while working in close cooperation with the international humanitarian community. The dual role of the Government, both as a party to the conflict and as a gatekeeper for humanitarian assistance, led to a significant encroachment on humanitarian space, with decisions about who got assistance sometimes being influenced by political and military considerations. This encroachment has gone largely unchallenged in Pakistan.

The failure to challenge these limitations in this crisis has set a bad precedent in a context where further complex emergencies can be expected. The Government is not monolithic, and different components of the Government have different interests. While there was criticism of the slow pace of the work of NGOs by some parts of the Government, other parts of the Government were interested in what the international community could provide. There was some space for humanitarian diplomacy to push for a less restrictive humanitarian space. However, there was relatively little effort by the humanitarian community to push issues of humanitarian space up the agenda.

Relatively few needs assessments were conducted, and joint assessments were rare, despite the development of useful tools like the Multi-Cluster Rapid Assessment Mechanism (McRAM). The needs assessments that were conducted were more for individual agency use than for use by the broader humanitarian community. The affected population was rarely consulted in any effective way. Consultation mainly took place with male leaders and not with the broader population or with women. The lack of consultation led to some inappropriate assistance, such as concentrating on shelter kits rather than on rent assistance or on the costs of house repair.

Agencies often did not specifically address the particular needs of women, girls, boys, or men, and took little account of differing needs. This improved somewhat in 2010. However, interviewees commented that this lack of attention to differing needs was driven in part by a lack of willingness to confront cultural norms of patriarchy and gender inequality.

The affected population that the team met with generally preferred cash to in-kind assistance, with good reason. However, agencies were still reluctant to move away from their giving in kind comfort-zone even where such in kind assistance represented poor value for money. The high attractiveness of cash has led to significant political pressure for the inclusion of ineligible beneficiaries, and has provided an incentive for fraudulent registration.

While the registration exercise was a logistic triumph, the registration criteria were fundamentally flawed and led to inclusion and exclusion errors both on IDP status and on the need for assistance. In any emergency, there is a trade-off between speed and accuracy in registration. The registration in this case was a good example of fast and relatively effective registration. However, there was no transition to a more accurate targeting of the bulk of the affected population in the first six months of the response. Thus, the response was category- based (on the registration criteria) rather than status- based (IDP) or needs-based. The major humanitarian actors accepted the flawed registration and used it as a basis for their own distributions.

Assistance was not proportionate to need, but varied with many extraneous factors and over time. Assistance was much harder to access for female- headed families. Relief assistance was much more readily available than recovery assistance. Those in camps were far better assisted by the international community than those hosted by families, families in rented accommodation, returnees, host families or stayees.
The camp population was the best assisted but was also the poorest group of IDPs. However, some of those who stayed in the notified areas were even poorer, as they had stayed because they could not afford the costs of displacement. Paradoxically they received less assistance than anyone else.

Some of the most difficult gaps in assistance were gaps which predated the crisis and which reflected chronic underlying problems in the provision of basic services in Pakistan. This, together with the complexities of varying patterns of need, and varying levels of assistance for different groups, provides a very complex pattern of need that has made communicating the needs to donors difficult.

Most of the response focus has been on relief rather than on recovery, rehabilitation or development. Recovery needs are not being met, largely because of a lack of funding. Direct damage to infrastructure and livelihood assets is a major problem in some areas of return. There has been relatively little livelihood support.

While the response was far from perfect, it was effective in preventing large-scale death and suffering. This is due in part to the support provided to the IDPs by the local community, but the international community has also played a key role. The international community in Pakistan is already working to resolve many of the issues identified in this report.

Purpose/ Objective:

The HCT has requested an inter-agency real-time evaluation as a means of obtaining real-time analysis and feedback to help better focus and adjust on-going implementation strategies. The evaluation will as such provide an analytical snap shot of the current situation. It will also offer an independent perspective on issues such as main successes and shortcomings within the current response. It will also address the issue of whether current humanitarian action in Pakistan is adequately grounded in humanitarian principles including gender equality. The evaluation will also focus on the crucial issue of whether the current response is addressing and covering actual needs on the ground particularly vulnerable individuals and families and whether partnership arrangements with key national actors are wholly adequate. The evaluation seeks to complement the work already done in the frame of the Inter-Cluster diagnostic missions. Hence, it will be targeted at the main results and outcomes of the aid provided by the whole system and will not address coordination issues already analyzed by the Clusters.

In early December 2009 a preparation mission allowed to further elicit specific expectations by the members of the Humanitarian Country Team on desired focus and outcome of an RTE:

Most appropriate approach:
An evaluation to look at results (in terms of what has been delivered) / adherence and compliance to standards and coverage of assessed needs

Thematic focuses of RTE:
Special attention needs to be given to issues of humanitarian space and access (to all parts of the population irrespective of sex, age, ethnicity, religion etc.) as well as principles, Civil-Military Coordination, Common strategies and Monitoring and Evaluation systems to track performance.

Geographical focus:
The Swat (Malakand Division) and the Bajaur (FATA) districts have been identified as the most appropriate and feasible geographical areas in which to conduct the RTE (security permitting by the time of the consultant team deployment). The rationales for selecting these districts are as follows:
In Swat, about 162,674 families (approximately 1,138,718 people) were displaced from Swat to Peshawar Valleys, Nowshera, Mardan and other safer locations. During the humanitarian response, there was a close collaboration with the Government. Cluster Focal Points had been identified and district coordination systems have been instituted jointly with the civil administration authorities. By July 2009, many areas in Swat were declared safe by the Government and by July the Government started implementing its return plan. Despite significant number of returnees, thousands of families remain to this day unable or unwilling to return. There are valuable lessons to be learnt on return practice and early recovery.
In Bajaur, internal displacement is still ongoing and no return plan has been implemented by the Government up to date. However, the HCT assumes that the response might soon start to plan for returning IDP‟s and durable solutions. The HCT is therefore in need to learn in „real time‟ how to best respond in the near future to the Bajaur displacement. It will offer valuable lessons to be learnt on questions such as IDP return policy and practice, camp establishment and closure, the interweaving of counter-insurgency, humanitarian goals and humanitarian space.
In comparing the humanitarian response in the Swat with the situation in Bajaur, the IA RTE team might come up with findings and recommendations which will be of immediate and „real time‟ use for the response planning of the HCT.
If the security situation should not allow for access to Swat or Bajaur, the HCT identified the following geographical areas as possible locations for the IA RTE: S.I. Khan (South Waziristan) and Hangu/ Kohat (FATA).

Potential users and uses of evaluation results in real-time:
The whole of the humanitarian country team and partners including government and military actors to adapt and improve strategies of engagement for ongoing and future operations to assist and protect IDPs.
The results are also expected to feed into the ongoing Post Crisis Needs Assessment (PCNA).

Methodology:

The evaluation will be carried out through analyses of various sources of information including desk reviews; field visits to Islamabad, Peshawar and Mingora / Swat; interviews with key stakeholders (such UN, I/NGOs, donors, beneficiary communities and government including the Core Group on PCNA) and through cross-validation of data. Briefing workshops in Islamabad will serve as a mechanism to both feed back findings on a real-time basis, and further validate information.

The approach will be essentially deductive, trying to identify critical success factors from the assessment of timeliness, adequacy and effectiveness of the response. It will also be highly participatory, facilitating „space for reflection‟ by key international actors involved in the response on how well the response was conducted and how it could be strengthened.

There will not be sufficient time to do a random sampling which ideally should be proposed. Instead, since the aim is to identify only critical successes and failures, the identification of affected populations to meet should be made by targeting recipients (female and male) where aid has been provided and also those who have not received any or sufficient / adequate aid. Focus groups as well as individual interviews can be foreseen to that end. These will be constituted so as to reflect views of different groups of recipients so that „elite capture‟ of data doesn‟t skew the evaluation findings. In a second step, interviews with decision-makers in Peshawar and Islamabad as well as joint briefings and debriefings have to be foreseen.

While maintaining independence, the evaluation will seek the views of all parties. Compliance with United Nations Evaluation Group standards and ALNAP quality pro forma is expected and the evaluation report will be judged in this regard. The two documents are available from the website of the OCHA Evaluation and Studies Unit (http://ochaonline.un.org/ess). All external evaluation reports will also be submitted to ALNAP for inclusion in the regular meta evaluation process that rates the quality of evaluation reports.

Conclusions:

Effectiveness
The overall response, in which the international humanitarian community played a vital role, meant that there was no large-scale death or disease outbreak in a context where millions of people were displaced from their homes in a very short time frame. There were a number of innovative features in the response, including relatively rapid registration and verification based on flawed criteria, and the use of smart cards for distributing cash assistance.
While donors gave generous support for the relief phase when the crisis was in the news in 2009, support in 2010 has been far less than needed, either to meet recovery needs in the area of return, or the needs of those who are still displaced. Paradoxically support for the early recovery, in terms of funding against the requested amount, increased from 2009 to 2010, but NGOs complained that they had great difficulty in getting funding for recovery.
The response of Pakistan again illustrates the lack of UN commitment to the humanitarian reform process through the failure to appoint a full-time HC free of role conflict. The HCT is not currently an effective forum for leading the international humanitarian response.
Cluster performance has been very variable. With some exceptions, the clusters were plagued by well- known issues such as the lack of trained full-time coordinators. Some of the problems seen in the clusters repeated those of the response to the 2005 earthquake and the 2007 floods. The clusters have also suffered from unrealistic expectations regarding their ability to manage pooled funding. The slow pace of the roll out of field coordination left humanitarian coordination at the district level in the hands of the military for some considerable time.
Insecurity has been a major problem for the humanitarian response. Well-founded security concerns have limited assessments and monitoring and the level of programmes staffing. Security has imposed large operational costs and has had a negative effect on programme quality.
The Government constrained access by humanitarian actors, partly in response to security concerns and partly as a control measure. Access constraints in the form of government controls or agency security rules have completely shaped and constrained the humanitarian response. The manner in which the NOC system operates is a major constraint in the delivery of humanitarian assistance.

Consistency
Overall, the response to the IDP crisis was driven by pragmatism and the need for access to the affected community rather than by the humanitarian principles of independence, neutrality, and impartiality. The extent of the trade- off between principles and access varied by actor. The limitation of humanitarian space have gone largely unchallenged in Pakistan.
Pakistan is a member state of the UN and it is hardly surprising that UN agencies did not work in an impartial, independent or neutral manner, but instead strongly supported the Government. Clearly it is very difficult for the UN to work in a principled manner in a UN member state when that UN member state is one of the parties to the conflict and insists that the actions of the UN are aligned with those of the state.
The Government sought to repeat the success of the earthquake response, where the military-led the response while working in close cooperation with the international humanitarian community. The GoP had little understanding of the differences in the relationship between humanitarian agencies and the Government in natural and complex emergencies.
The local community played a key role in the response, with 85% to 90% of the displaced being housed by them. The response of the provincial government, of the Pakistani public, of the national government, and of other provinces was also critical in ensuring that needs were met. The international community also played a key role in the response, not only with resources, but also with specialist skills.
While the level of attention to gender issues improved in 2010, women's needs were often not specifically addressed. Interviewees commented that this lack of attention was partly driven by a lack of willingness compounding the practical problems of attending to gender issues in a society where women are so marginalised.
The joint problems of access and security meant that monitoring was not strong. This has been compounded by relatively little review or evaluation. While there was some evidence of learning from previous operations and from the early stages of the IDP response, the overall process of learning has been haphazard and ad-hoc.
The internal displacement was spurred by the desire of the military to create free-fire zones in the areas controlled by militants. However, the military also plays a key role in responding to humanitarian emergencies in Pakistan. This role conflict led to a significant encroachment on humanitarian space, with decisions about who got assistance being sometimes influenced by political and military considerations.
The largely unchallenged limitation of humanitarian space has set a bad precedent in a context where further complex emergencies can be expected. The Government is not monolithic, and different components of the Government have different interests. While there was criticism of the slow pace of the work of NGOs by some parts of the Government, other parts of the Government were interested in what the international community could provide. There was some space for humanitarian diplomacy to push for a less restrictive humanitarian space. However, there was relatively little effort by the humanitarian community to push issues of humanitarian space up the agenda.

Relevance
Relatively few needs assessments were conducted, and joint assessments were rare, despite the development of useful tools like the McRAM. The needs assessments that were conducted were more for individual agency use than for use by the broader humanitarian community. Assessments generally looked at the household as a monolithic unit, without considering the different needs of women, boys, girls, and men in the household.
The affected population was rarely consulted in any effective way. Consultation mainly took place with male leaders and not with the broader population or with women. The lack of consultation led to some inappropriate assistance, such as concentrating on shelter kits rather that assistance with rent or with the costs of house repair.
The affected population that the team met with generally preferred cash to in-kind assistance, with good reason. However, agencies were still reluctant to move away from their giving-in-kind comfort-zone even where such in kind assistance represented poor value for money. The high attractiveness of cash has led to significant political pressure for inclusion, and has provided an incentive for fraudulent registration.
While the registration exercise was a logistic triumph, the registration criteria were fundamentally flawed and led to inclusion and exclusion errors both on IDP status and on the need for assistance. In any emergency, there is a trade-off between speed and accuracy in registration, the registration in this case was a good example of fast and relatively effective registration. However, there was no transition to more accurate targeting for the bulk of the affected population in 2009. Thus, the response was category- based (having a CNIC with a particular address) rather than status- based (IDP) or needs-based. However, the major humanitarian actors accepted the flawed registration and used it as a basis for their own distributions without raising any major issues.
The in-camp population was the best assisted. However, these were also those who were the poorest among the displaced. Some of those who stayed did so because they could not afford the costs of displacement, and paradoxically received less assistance than anyone else.
Some of the most difficult gaps in assistance were gaps which predated the crisis and which reflected chronic underlying problems in the provision of basic services in Pakistan73. This, together with the complexities of varying patterns of need, and varying levels of assistance for different groups, provides a very complex pattern of need that cannot be described with a single indicator. This has led to difficulties in communicating the level of needs to donors.
Assistance was not proportionate to need, but varied with many extraneous factors and over time. Assistance was much harder to access for female- headed families. Relief assistance was much more readily available than recovery assistance. Those in camps were far better assisted than hosted families, families in rented accommodation, returnees, host families or stayees.

Timeliness
The response to the crisis by the local community was fast and timely. The response meant that delays in the initial relief response by the international community were non-critical. However, funding problems, including the delay of launching the PHRP, has resulted in large delays in responding to recovery needs.
The NOC system is a serious bureaucratic constraint that prevents timely action by NGOs. It is one of the bureaucratic measures used to control access of NGOs, but the normal constraints within the Government bureaucracy greatly amplify its impact.

Connectedness
Early recovery is strongly constrained by funding with recovery needs not being met. Most of the response focus has been on relief rather than on recovery, rehabilitation or development.
Direct damage to infrastructure and livelihood assets is a major problem in some areas of return. There has been relatively little livelihood support.
The biggest changes since 2009 are the scale of movements, donor interest and level of funding, the shift from assisting the displaced to recovery, and the geographical focus of the crisis and response.

Recommendations:

1. The ERC should immediately introduce a requirement that, except in exceptional circumstances, the heads of the three main UN humanitarian agencies (UNICEF, WFP, and UNHCR- where present) should attend the regular meeting of the HCT. The ERC should also require HCs to provide a summary of HCT attendance by heads of agencies each quarter.
2. The ERC should consult with the IASC about changing the HCT guidelines to limit membership on the UN side to the key UN humanitarian operational agencies (typically UNHCR, WFP, and UNICEF) with a single representative for other UN agencies and the possibility of inviting specific UN agencies as dictated by the agenda.
3. The chair of the HCT meeting should, with immediate effect for all decisions, establish who exactly is responsible for implementing the decision, who should report back on it, and when progress reports should be made. Progress against implementation should be reviewed at subsequent meetings.
4. The IASC should develop a common reporting format that can be promoted as a single reporting format in all new emergencies.
5. The HCT should develop an active strategy of humanitarian diplomacy to work toward a more principled approach and a less constrained humanitarian space in Pakistan, including putting the issue on the agenda for donors.
6. The HCT should encourage members to implement the IASC policy on gender in humanitarian action and should actively monitor progress towards the achievement of this policy.
7. The humanitarian community in Pakistan should reassess all existing programmes, both in terms of the appropriateness of the assistance that they are providing and whom they are targeting. The priority for reassessment should be programmes delivering goods in kind. In kind assistance should only continue where the affected population prefer it to cash or vouchers on an equivalent cost basis.
8. The HCT should roll out the vulnerability assessment model for the whole affected area and base assistance on vulnerability rather than just on registration.
9. The HCT should begin a process of consistently mapping needs across the affected area disaggregated by region, age, and sex. Such a mapping may require that the HCT commission a detailed survey of the affected population to discover what types of assistance has been most useful, and what types of assistance are still needed.
10. The Government should introduce a „presumed to comply‟ rule for NOCs where, unless NOCs are formally refused, they are deemed to come into effect within seven days of the submission of the application.
11. The HCT should continue to advocate with donors to adequately fund recovery in the short to intermediate term while waiting for the medium to long term mechanisms to come into play.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search