Evaluation database

Evaluation report

2011 Pacific Islands: Final Evaluation of the Recovery Action and Rehabilitation Project (RARP)



Author: B. Lockton Morrissey

Executive summary

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:

The Recovery Action and Rehabilitation Project as a Solomon Islands Government (SIG) response to the tsunami/earthquake in the western part of the country in April 2007, which affected specific areas of Western and Choisuel provinces. Preliminary assessments and planning for the education sector were conducted immediately after the disaster by the Technical Advisory Teams installed by the Ministry of Education and Human Resources Development (MEHRD) in May-July 2007. Fifty-two people had died. More than 40,000 people were affected, many of them were traumatized. 4,000 homes were damaged or destroyed. In education sector 89% of the enrolled students (18,258) had their education disrupted and 11% of all available school infrastructure was completed destroyed and 36% of the school infrastructure experienced major damage.
During the first period of the recovery, activities in the education sector were being guided by a Taskforce of Education in Emergency Situation (TEES) chaired by MEHRD and with representation from the Development Partners (New Zealand, EU, UNICEF, Save the Children). Based on an assessment report, the TEES drafted a 6-month Recovery Action Plan (RAP) which was endorsed by the TEES and the Permanent Secretary in July 2007.

Purpose/Objective:

1.To enable pupils, students, teachers and community members to physically, materially and mentally recover from the earthquake and tsunami
2.To give relief and support pupils, students, teachers, provincial education staff;
3.To re-establish within the shortest time possible good quality early childhood, primary, secondary and vocational and technical education
4.To construct, rehabilitation and repair school infrastructure including classrooms, offices, dormitories, water supply and waste water/septic facilities and other identified infrastructure need with a view to ‘building back better’
5.To strengthen the capacity with Solomon Islands, particularly of MEHRD, teachers and the Western and Choiseul Education Authorities in disaster response and recovery

Methodology:

The evaluation team has taken the following broad approach in carrying out the tasks detailed in the RARP TOR
1.Document review of relevant project documentation and reports
2.Prepare an evaluation plan
3.Develop and outline set of questions to be used in focus/key informant meetings
4.Attend initial and final briefings with the TEES in Honiara
5.Conduct key informant meetings and reviews with stakeholders in Honiara 17 – 19 January 2011. To include MEHRD, the donor group NZ, EU and UNICEF, NDMO and other education/response partners within Government and NGO
6.Undertake field work in Western and Choiseul Provinces to includes:
a. Key informant meeting with stakeholders and counterparts in the education and tsunami response sectors, largely in Gizo
b. Field trip to examine work across selected sites in Western and Choiseul Provinces to include Gizo, Simo Islands, Vella la Vella Island, Vonavona Lagoon, New Georgia Island, Kolombangara Island and South Choiseul. Visits included inspections of physical progress, meetings with education partners, school committees, community groups, local Governments and other actors in that area.
c. A workshop/s in Gizo with partners/stakeholders to elicit further information and test preliminary findings
7. Further consultation in Honiara with a range of key informants and potential detailed analysis of reporting with MEHRD staff to test any initial hypotheses
8. A key stakeholder workshop in Honiara to outline broad draft findings, discuss detail and their implications
9. Follow up with former project staff now departed and further document review
10. Submission of draft evaluation report for consideration by stakeholders
11.Completion of final evaluation report by team leader

Findings and Conclusions:

Relevance : As elements of education and its infrastructure are included in the MDGs, and also the Hyogo Framework for resilience, the project addressed global priorities for assistance after a rapid onset event. Hence Relevance can be considered a strength of the design and implementation of RARP.
RARP Relevance Rating: 6 (Very Good – needs for relevant beneficiaries appropriately identified and design right to meet them)
Effectiveness : the effectiveness of RARP would be considered acceptable compared to other MEHRD projects which did not generally have the limitation of being rehabilitation after natural disaster.
RARP Effectiveness Rating: 4 (Adequate Quality – outputs and outcomes are being largely achieved
Efficiency: In terms of an overall assessment of value for money, without the full financial picture, this remains an opinion not a calculation. To weigh the totality of the project up in terms of the time taken, the outputs, the impact, the inputs in the staff and their cost, the cost of implementation overall and the quality of the infrastructure outputs, it is the opinion of theis author that the project did not offer optional value for money. The project could have been completed more quickly, with less staff and costs, but that given the lack of a thorough project design, the inappropriate implementation model in 2008 and the staff allocated at the time, what was achieved was credible, if not impressive. However, with this model and design, when held against the required outcomes the RARP is not a high value for money project.
RARP Efficiency Rating: 3 (Less than Adequate – scope for improvement in value for money)
Impact: The quality of work done in the tree planting and water/sanitation components of the track two activities was a generally high quality and has provided positive impact (NRDF interviews). In spite of the number of these insufficiences in the program which reduce impact, it should be restated that 2,444 classrooms for 7,540 students, 395 latrines, 204 water supply projects and 3,462 desks, chairs and tables were constructed. Given the challenges of procurement, sourcing timber, the logistics of an island based program and continual restrictions oon cash flow, this is a significant output which indicates acceptable level of impact for the Development Partners given their objective to faciliate a return to school for children of Western and Choiseul Provinces.
RARP Impact Rating: 5 (Good, net positive impact equivalent to planned impact)
Sustanability: Activities to raise the capacity of MEHRD at national and provincial levels were not done. RARP did not retain a presence in Honiara so building capacity in MEHRD Honiara was not possible. Four Education Authorities managed schools being assisted by RARP. Their willingness and ability to be involved in RARP and still complete their own workload was minimal. Their involvement was little more than being appraised of the plans and progress. Hence, no emergency education capacity was built in MEHRD in either Gizo or Choiseul.
Nevertheless, four years to come children in Western and Choiseul Provinces will be able to attend classes in schools which are now far better than the ones which existed in 2007 and this is a consequence of RARP. If teachers are more motivated to teach in these locations because they have staff houses with water, latrines and good classrooms then one would conclude that the standard of education may improve. So whilst capacity has been built, a better education environment has been fostered and this would satisfy the main goal of the RARP to return children to school and improve their likely education outcomes.
Sustainability Rating 5 (Good – positive benefits are expected to be sustained)

Recommendations:

The following key recommendations concentrate on the future of recovery in education not on the detail of RARP as that project has finished and longer exists to benefit from modification or lessons learned.
Recommendation1: Future infrastructure based recovery programs will as a first phase, be designed by engineers to include scope of works, detailed staffing requirements, approved designs specifications, bills of quantities, costing estimates, risk analysis, logistic plan, financial management policy and outline of associated non-construction activities. Additionally a Performance Assessment Framework to assess all aspects of the project against milestones to enable both monitoring of outputs and evaluation of impact be produced in parallel to the engineering design work by development evaluation experts. Based on this detail, a programme implementation team can be deployed for the implementation phase
Recommendation 2: In Future, programs of the level of work and complexity of RARP will need a single management unit under a single entity which will be the only pathway for cash inputs and disbursement. This entity which might be an engineering firm, an aid consulting firm, a major NGO/international agency or a private firm which will be contracted to the SIG Ministry and hold the head contract for all donor inputs and funds and remain responsible for all staff, activity and reporting. It is recommended that programme of such a level of activity and funding are not implemented directly by MEHRD, but have an external implementing contractor on MEHRD and any Development Partners agreeing to participate
Recommendation 3: Infrastructure projects of the RARP type to utilize the community participation and engagement model, but only when the capacity of communities to contribute after a rapid onset event has been thoroughly assessed. The role and involvement of women should be better articulated through a simple set of guidelines to be used in awareness raising activities.
Recommendation 4: Development Partners consider directly developing the capacity of the TEES and MEHRD along with other key SIG elements to plan, manage and evaluate the educational aspects of disaster response and recovery. This would be done in association with the NDMO and under the guidance of the NDC, but using a coordination of local and regional expertise in disaster management and particularly recovery in education. The training would focus on capability to make more detailed assessments including the development of suitable tools for MEHRD, and the development of early and medium term recovery strategies to assist the scoping of future projects of this type. Any work in this area would need to complement existing Frontline Responder training being developed and in harmony with the draft School Disaster Preparedness and Response Manual.
Recommendation 5: The projects deferred from RARP which will be supported from the programme budgets over the course of 2011-12 should be largely sub-contracted and not run in detail by MEHRD officers or management of these projects will consume all available management time and restrict the ability of the infrastructure office to develop its capacity for oversight of infrastructure projects. Some former RARP staff remain in Gizo so it is a consideration that the deferred projects be subcontracted to a group of those persons familiar with the operational era and mechanism of completing tasks by ‘piece work’ with local communities, in association with the use of artisans such as plumbers in the region.
Recommendation 6: Development Partners consider ways to jointly support the continuation of the TEES as a management mechanism of education issues in disaster management, perhaps through support to a small secretariat function and the conduct of simulation work to reinforce skills and lessons learned; and support for the development of an education recovery strategy to guide future implementation of the type used in RARP. The strategy should be harmonised with standard World Bank and UND recovery mechanisms
Recommendation 7: Development Partners consider support for the enhancement of stocks of education emergency stores, such as UNICEF tents and kits, for deployment to regions of Solomon Islands in addition to holdings currently with NDMO; and establish a small fund within MEHRD for deployment and maintenance of supplies

Lessons Learned (Optional):

1) To manage and implement a project of the complexity and scale a professional management team is needed close to the site of work, not in Honiara.
2) A greater emphasis on engineering expertise is need in assessment and management, rather than relying on education expertise.
3) A thorough design process must precede implementation work and this must be accompanied by a Performance Assessment Framework, both documents compiled by sector experts.
4) International procurement standards need to be applied to a project with as much expenditure in procurement and subcontracting.
5) A community-build infrastructure program can work, but the level of technical advice via field officers is critical to the quality of the outputs. The role of women, and the extent to which payments are made to community members, needs to be considered
6) A single pathway for funds, paid in advance, is needed to allow for even and continous project activity against work plans.
7) Any recovery program needs to comprehensively address resilience through DRR.
8) An environmental recovery projects needs community awareness and attitundiual change to be sustainable and can not rely on inputs and activities.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search