Author: Jon Bennett (Team Leader) and Jenny Reid Austin (co-author)
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
This report is a synthesis of the evaluation of UNICEF's response to the 2004 Indian Ocean tsunami in Indonesia that was undertaken in August 2008 to July 2009. The evaluation assessed UNICEF's response in four sectors where it had major involvement: child protection; basic education; water, sanitation and hygiene; and child and maternal health and nutrition. This report seeks to provide a larger picture of UNICEF’s response from 2005-2008, with a main focus on the relief and early development phases. It does so by drawing on the findings and lessons obtained from each of the independent sector evaluations that constitute the evaluation in Indonesia. The report also examines cross-cutting issues related to recovery and transition, and asks whether appropriate strategic choices were made during UNICEF's effort to help Indonesia "build back better", and how these were likely to impact the wellbeing and rights of children and women.
The purpose of the evaluation in Aceh, Indonesia, is to determine outcomes and impacts of UNICEF’s response to the tsunami and draw lessons and recommendations – both for the UNICEF and the sectors as a whole - that will be useful for strengthening ongoing programmes or policies to improve the wellbeing and rights of children. In addition, the evaluation draws lessons for recovery/transition programming that will be useful for future response and emergencies.
A thorough literature review, including data not always in the public realm (e.g., country-level NGO reports and academic studies), comparing and contrasting approaches undertaken in the recovery phases was carried out. Extensive interviews with senior and technical government ministry staff was done to determine overall progress within each sector and to assess the relative contribution UNICEF has made to developments in the country over a 4-5 year period. Additional methods included: interviews with previous and current UNICEF programme staff to nuance existing documented lessons; field surveys (per sector, though in some cases combining sectoral questionnaires): teams were responsible for conducting primary, field-based data gathering that included focus group discussions, questionnaires and transect walk methods. Field survey teams were also responsible for collating the data.
Each sector evaluation also looked at a number of important cross-sectoral issues. In each sector report reference is made to UNICEF Core Commitments for Children in Emergencies, the extent to which UNICEF took part in inter-agency needs assessments and/or other surveys, and how it reported on the general situation of children and women. Likewise, for the recovery and early development phases, each evaluation refers to UNICEF’s MTSP and issues related to human rights-based approach to programming, gender mainstreaming and national/local capacity development. Finally, each sector evaluation examines the extent to which UNICEF has contributed to disaster preparedness and risk mitigation efforts. In Aceh, an issue of particular importance is how UNICEF managed to deal with the differences and challenges between conflict-affected and tsunami-affected areas, and what the experience was in working with various parties, including non-state entities.
Findings and Conclusions:
In returning to a regular—and much reduced— programme after four years of unusually high financial inputs, a key question is the extent to which UNICEF’s strategic approach complements, enhances and influences efforts of both the GoI and of development agencies which, like UNICEF, will remain in Aceh in the foreseeable future. One clear finding from the evaluation is that UNICEF’s construction programme (health and education facilities) entailed high transaction costs in terms of staff and was ultimately a distraction from wider developmental concerns of capacity development and addressing chronic concerns such as malnutrition.
At this stage in the recovery process, UNICEF programme outcomes in the education sector are the most positive in terms of impact data. Unusually large funding in the wake of the tsunami allowed previously dormant policies in education to be reinvigorated while also drawing international attention to issues of quality. Optimistically, one can assume some of these changes to be permanent; for example, the extension of free education up to age 12, in line with national targets. The revitalisation of School-Based Management and use of School Committees as a means of realising good school governance—each underpinned by policy under NADESP—would be another example. Outcomes on learning are more difficult to ascertain, not least because measurement methods are still poor. However, UNICEF’s support to EMIS school-based data has shown some early promise.
In the health sector, a successful emergency phase was not matched by strategic multi-year planning in the recovery phase. More attention should have been paid to the development of district- and provincial-level capacity for planning and administration. Structural weaknesses in government, apparent before the tsunami, persist; UNICEF’s preoccupation with bricks and mortar has prevented appropriate attention to this. Nevertheless, UNICEF has helped to build institutional capacity through training of health care workers, social workers and volunteers, especially in IDP communities. UNICEF’s support to the 2007 Demographic and Health Survey was critical to improving health service data collection systems in Aceh.
In child protection, there are promising trends towards recognising and institutionalising approaches that are still very new in Indonesia but for which government and communities have an increasing awareness and concern. UNICEF’s support has now appropriately shifted almost exclusively to technical and coordination advice as government funding becomes more readily available. Two other positive developments have come from the tsunami response: first, a dramatic increase in human resource development at provincial and district-levels, coupled with policy and coordinating mechanisms; and secondly, an increasing recognition of the role to be played by NGOs in child protection services.
Evidence from the WASH sector suggests there is still some way to go before attitudes and practices change, not least because the convergence of hardware (infrastructure) and software (health practices) have yet to achieve greater coherence in the approaches adopted by a disparate number of actors. Nevertheless, despite being stretched quite thinly across the sector, UNICEF has contributed to increasing access to improved water as well as increasing the capacity of communities to manage their own water systems as well as that of PDAMs and local government agencies.
Attributing change to any one agency would be dishonest, but UNICEF’s sizeable interventions over four years, and the encouraging manner in which it has assimilated lessons from these interventions, have given it a unique opportunity to guide and influence a provincial government whose resources are equal to that of a small country and which, in a short period of time, has demonstrated a willingness to accommodate and adapt international standards after many years of isolation.
1) Capacity needs assessment should be undertaken very early in the recovery phase. In particular, this should include analysis of capacities at provincial and district levels, cross-referenced with an understanding of work being undertaken by other agencies in relevant sectors. This is important not only for intervention strategy, but also for sequencing of priorities in UNICEF’s capacity work.
2) The management of large-scale construction programmes in all sectors should not be undertaken by UNICEF – it should be outsourced. Capital-intensive projects, such as construction and the purchase of significant assets entails high transaction costs for materials as well as staff, and can distract from wider development concerns. However, new facilities present unique opportunities to develop and promote quality issues, for example, in the education sector.
3) Continuity of senior staff over the transition period should be assured. Effective exit strategies for emergency programmes are unlikely to occur unless more development-oriented staff are in place at an early stage of the recovery.
4) UNICEF should ensure that technical assistance offered to government ministries is complemented by a public communications strategy that increases demand and community ownership of any new services being offered.
5) UNICEF should promote the collection and analysis of disaggregated data on vulnerability—both qualitative and quantitative. Although domestic and local safety nets will always be of paramount importance, adherence to human rights principles and international standards will only be assured through institutionalising participatory methods of recognising and measuring inclusion and exclusion.
6) UNICEF should positively discriminate in favour of women’s grassroots and advocacy organisations as implementers, and provide appropriate capacity assistance to enable this. This would counter inherent cultural and gender bias in the selection of partners. Resource commitments should, however, recognise that capacity provision as well as capacity development is appropriate in some cases, in order to get programmes underway.
7) Protecting the many investments made in the relief, recovery and development phases includes greater attention given to maintenance issues. Plan for sustainable programming, support, maintenance, continuity and human resources development in order to support the interventions, systems and infrastructure implemented post-tsunami. This would include anticipating needs after agencies and donors exit.
8) Support evidence-based systems to inform planning, implementation and monitoring and evaluation. It is critical to ensure information management and data collection systems. This involves supporting existing data collective processes and establishing new approaches to ensure consistency, reliability and data access across agencies. Relevant staff should be trained in the usage, search and analysis of such data, which should be used to strengthen monitoring and evaluation, and inform decision making and strategic planning.
9) Support and develop guidelines for the provision of needed items, equipment and materials during emergencies. Such guidance should include criteria for implementation capacity and decision making of expensive items in an environment with high staff turnover and competing priorities. Given the context and stress of disaster environments and complex emergencies, accounting for the need to decentralise fast decision making, supportive guidelines should address materials and equipment and materials sourcing. These could be included in preparedness plans and incorporated into existing long-term agreements.
10) Ensure that programme efforts, including emergency response and assessment tools, are inclusive and include the most vulnerable children or population groups. Pre-determined plans that identify the most vulnerable children, include strategies for reaching them and incorporate them into programming will help ensure that the needs of these most vulnerable populations are not excluded.
Recommendations per sector are included in the report.
Lessons Learned (Optional):
The following are the overarching lessons for ongoing efforts and for future responses to humanitarian emergencies:
1) Planning for emergency response through recovery and early development should balance both longer-term capacity development with investments in assets and infrastructure, all of which should be reflective of local need.
2) Leverage the knowledge and encourage the commitment and meaningful participation of communities, CBOs and national NGOs to sustain and strengthen outcomes of tsunami interventions.
3) As part of recovery planning, the focus on vulnerable, disadvantaged and less reached children or population groups needs to be made more explicit for each programme area. Despite the appropriate strategic shift to cover non-tsunami affected conflict areas in Aceh, the need to improve inclusion was not evident in the education sector in Aceh.
4) Protect the investments made in the relief, recovery and development phases by planning for sustainability, such as maintenance support, and continuity of human resources capacity development.
5) There should be support for and maintenance of evidence-based systems of data collection and management to better inform policy, strategic planning and allow for strengthened monitoring and evaluation.
6) Disaster preparedness planning should include needs development assessment tools for each sector and trainings that have been pre-tested in various contexts.
PDF files require Acrobat Reader.