Author: Egbert Sondorp for UNICEF and the European Commission. Institution: UNICEF and the Europea Commission
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
This document reports on the final evaluation of the project “Support to Health Sector Development in Somalia – Capacity Building, Strengthening System Coordination and Performance”, better known as the Lot 3 project. The Lot3 project was commissioned by the EC, implemented by UNICEF from January 2007 – June 2009. From the budget of 2.15 Million Euro in the end 1.31 million Euros was disbursed. EC procedures prevented further no cost extension.
Prior to 2006, both a report commissioned by the EC and a comprehensive Joint Needs Assessment recommended to steer away from the long term costs of a perpetual short term, project focused approach to health system intervention in Somalia and pay more programmatic attention to broader health system analysis and capacity building. Some of these notions echo in a Call for Proposal the EC launched by March 2006. This CfP was issued, when the EC had already decided to withdraw from the health sector in Somalia in 2 years time.
The CfP called for three proposals, labelled Lot 1, 2 and 3, to support basic quality health services within the framework of a structured and harmonized health system while addressing gender issues. Lot 3 was seen as an umbrella project to the more traditional NGO run projects (Lot 1 and 2) that were to support a number of hospitals and some other health facilities in various parts of Somalia. Lot 3 was then to provide support across the sector and work on institutional development of the three Ministries of Health in Somalia.
Since no one submitted a bid for the Lot 3 CfP, the EC started negotiations with UNICEF to take it on. UNICEF agreed, provided it could change the ToR. It so did through a ‘concept note’. The concept note took on a broad health systems approach and proposed implementation primarily by setting up Health Capacity Building Units in the three MoHs. Lot 1 and 2 could then be supported from these HCBUs.
This concept note was then transformed by UNICEF into a fully fledged technical proposal by November 2006 and approved by the EC. In retrospect, this technical proposal comes across as highly over‐ambitious, with a set of objectives that touches on too many facets of a very poorly developed health sector in an environment with very poor state functions. The proposal was not realistic and probably could never be achieved, and definitely not in the given time frame of 2 years and with a limited budget. In addition, unlike the optimism that had prevailed in early 2006, the situation in Somalia had started to deteriorate again and continued to do so until today.
To provide UNICEF and the EC with sufficient information as to the:
• appropriateness of the proposal,
• feasibility of the action,
• suitability of the programme implementation strategy and
• impact of the programme
– with lessons learned for similar programmatic approaches both in Somalia and elsewhere in comparable contexts.
A particular focus of the study was to be an evaluation of the application and suitability of the approach to institutional development of the MoH, which was applied through the programme, and recommendations on how this could be strengthened and followed through in the future.
The ToR furthermore acknowledged that while the programme has been implemented Somalia wide, the current security context in Somalia will prevent the evaluator to visit all project sites and all implementation zones. In this regard, the evaluator will be expected to provide an overview of the implementation of the entire programme, based on a thorough in depth desk study, interviews/meeting with partners and stakeholders in Nairobi and accessible zones, and support this with more in‐depth insight into/assessment of implementation in zones and sites to which field visits can be undertaken (Somaliland).
The evaluation was conducted by Dr Egbert Sondorp, with a visit to Nairobi and Somaliland between 20th October and 11th November 2009. For the purpose of the evaluation a reference group was set up i) to ensure that the evaluator has access to and has consulted all relevant information sources and documents related to the project/programme; ii) to validate the evaluation approach; iii) to discuss and comment on reports delivered by the evaluator; and iv) to assist with feedback on findings, conclusions, lessons and recommendations. This reference group consisted of representatives of the European Commission, UNOPS, and UNICEF as well as the NGOs COSV and COOPI. The reference group met three times with the evaluator, an initial briefing, a second meeting to primarily discuss the proposed field visit and a final session with a debrief from the evaluator presenting the preliminary results.
The evaluator conducted a review of an extensive list of project documents and other documents around Somalia and its health sector. Throughout the visit the evaluator conducted 31 interviews with a range of individuals, 4 focus group discussions with a total of about 20 people, and several meetings with hospital and clinic staff during field trips in Somaliland. See annex 7 for a list of people met.
Findings and Conclusions:
Project design phase
• The notion to pay more attention to broader health system issues was excellent and very much valid up to today. The willingness of both EC and UNICEF to take this on, should be commended. However, the final stage of the formulation phase was flawed.
• Due to time pressure and lack of technical expertise in both EC and UNICEF, a project was set up with over‐ambitious, unrealistic objectives, to be achieved in a short, unrealistic timeframe, with insufficient budget, against a background of deteriorating political and security circumstances in Somalia.
• The EC should not have approved this project because of its technical flaws as well as the substantial deviation from what the EC initially had intended with the Lot3 programme; in particular since the EC knew it was withdrawing from the health sector and the project could not be extended
• UNICEF should not have submitted this proposal, as was developed by its staff, to the EC for approval because of its technical flaws and the lacking prospect of continued funding beyond the life of Lot3, a necessary condition to embark on a project of this nature.
• Late arrival of the project manager, continous changes in the political and security situation, and the poor project design led to the need for repeated changes in the project, adapting to circumstances, but also with an eye to grab opportunities
• Changes were formally laid down in amendments and new logframes. However, these documents retained the unrealistic tone and objectives of the initial proposal, only partly reflected actual activities and were not consistently reported against.
• A core element of the proposal to set up Health Capacty Building Units in the three zonal MoHs did not materialize, primarily due to deteriorating security in all parts of Somalia.
• Main emphasis therefore turned to what became the major achievements of the project:
o The production of a series of analytical reports and tools
o Significant contribution to strategic leadership for the health sector
o Capacity building activities in MoH, mainly the Somaliland.
• Expectations, from the EC and Lot 1 and 2 projects, that Lots would support Lot 1 and 2, were not met.
• The project has shown that even under the extremely difficult circumstances as Somalia poses it is possibly to develop broader, health system oriented activities
• There is no blueprint for these activities and hardly any lessons are available from other fragile states, in part to the uniqueness of the Somali context, in part due to virtually total lack of research in this field
• These health system strengthening activities should ideally not be ‘projectised’, but be placed in an ongoing programme, well linked to the health coordination platforms, and be instrumental in strengthening the coordination and strategic leadership
• Such HSS programme will need access to best available data, or should produce its own data; documentation and analysis of relevant health system issues should be ongoing
• While being informed by available data and analysis and aware of available resource capacities (technical and financial) with the invariably manifold stakeholders, the strategic leadership should provide sound guidance as to:
o Desirability and feasibility of longer term health system investment
o Strengthen health governane through capacity building of not only the health authorities, but all stakeholders in health like the communities, civil society and health providers, including private providers.
o Strengthen service delivery, with a view to maximise health gains as would be possible in the given circumstances
o Desirability of small scale piloting of novel approaches, ensuring learning
o The ‘early recovery’ and HSS potential of humanitarian project activities
PDF files require Acrobat Reader.