Evaluation database

Evaluation report

2012 Somalia: Final Evaluation of UNICEF’s Programme “Making PPP Work for Rural Water Supply in Somalia”



Author: Hydrocounseil

Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."

Background:

This evaluation report concerns Unicef’s project “making Public-Private Partnerships work for rural water supply in Somalia”, a project financed under the European Union Water Facility and implemented from August 2007 until October 2012 in north-eastern and north-western zones of Somalia.

Since 1997, Unicef has been supporting, along with the national government, several successful experiences of public-private partnerships for water supply in the major towns of the same zones. Meanwhile, the rural water supply operations repeatedly request rehabilitations and further periodic support for even minor repair or replacement of facilities. It was identified by Unicef as a failure of community management to maintain water facilities, caused by “a lack of availability of spare parts and insufficient training, worsened by the increasing rural-urban divide where many of the trained people leave the villages for better economic prospects in the larger towns”. [... more in report]

Purpose/Objective:

The present document is the final evaluation report of this project, with following objectives assigned:

“The final evaluation study will provide UNICEF and the EC with sufficient information as to the final impact of the programme, and make recommendations for future needs and support that may be required for interventions undertaken within the programme and to the rural water sector. A particular focus of the study will be an evaluation of the application and suitability of the public private partnership approach for the management of rural water services, which was applied through the programme, and recommendations on how this could be strengthened, supported in the future.”

“The scope of the assignment covers all stages and aspects of the project including Project preparation and design, Relevance to the problems of the project area and beneficiaries, efficiency and effectiveness of implementation, actual and potential impacts with special emphasis to women and children, Factors affecting sustainability and replicability, economic and financial analyses.”

Methodology:

The assignment has been carried out in September and October 2012, mostly through desk review of existing documents, interviews with stakeholders and field visits to selected accessible communities.

Owing to logistic and security considerations, field visits could only be made to very few accessible locations, which limited the number of field-based evidences to support the findings. However, the stakeholder interviews gave the opportunity to build and support the consultant’s assessment through discussions.

Findings and Conclusions:

B.3.1. Relevance
The initial concept of the project gave interesting prospects for addressing the issue of the community management shortcomings, with the evidence of success in urban utilities. Moreover, the use of PPP arrangements was in line with the sector policy in both Puntland and Somaliland (and even with the Water Act for the latter).
However:
the initial design for the PPP option, set as a result of the logframe (urban utility expansion) proved to be inapplicable,
- the shortcomings of the community management is not the only reason for the poor functioning of rural facilities.
Hence, the overall objective set in the logframe (improving livelihood through better access to water) is only partially addressed by the PPP approach (rehabilitations works were needed as well, and were added during implementation), and the PPP approach had to be reoriented to be relevant for the Somali context.
The relevance of the project design is that of a pilot project, to test and adjust the PPP approach, but the initial concept of extending urban PPPs should not have been set as a result in the logframe, as it was the way to be tested and not the goal itself.

B.3.2. Effectiveness
The expansion of urban PPPs to rural water systems is not effective, because of the significant change in the project implementation (local private companies instead of extension of urban utilities). In spite of this, the number of beneficiaries reaches 64% of the initial target, and some of the expected benefits from urban PPP extensions are at least partially retained:
Trainings have developed some level of technical skills available for rural water supply,
The commercial operation yields first signs of improved management in some localities,
The contractual relationships between all the communities and the private companies have been soundly defined in elaborated documents,
The project developed a wider acceptance of the PPP concept by communities and central government,
The involvement of national authorities in the process developed a nascent capacity for replication at national level.
An effective improvement of service delivery quality can be observed in the field, but is mostly a result of rehabilitation works, which do not appear as planned activities in the logframe (but proved to be relevant)
Although not much documented yet, the experiment of PPP settings in the rural areas of Somalia yields some interesting lessons learnt.

B.3.3. Efficiency
The start-up of the project has been very slow before any concrete activities were implemented in the field, most of the progress towards operational PPPs in the selected locations was achieved after 2010 (year 4 and 5 of the project).
The initial partners (WEDC and Horn Relief) were less involved than expected and Unicef did not, at that time, manage to make up for that.
Unicef was not being comprehensive enough and relied on the inputs of the partners to start the project, while the main capacity to drive the project should have been within Unicef and partners could have acted as contractors.

B.3.4. Impact
Considering the difficulties of access in the field and the small number of field visits, it is difficult to draw an objective and evidence-based assessment of the project impact. However, some facts are worth mentioning as they indicate the prospects for positive impacts:
The first impacts in target villages in terms of autonomy are encouraging: Unicef and national authorities responsible for water no longer receive support requests from the target communities, which deal with their company on their own.
More and more stakeholders have a clearer understanding of PPPs and the benefit they can bring for public service delivery.
The project enabled some communities to build up confidence in undertaking other projects and considering private management/investment as a relevant option.
Children and women directly benefit from the improved service delivery, thanks to the works component of the project. 

Recommendations:

B.4.1. Short term

a) Knowledge capitalisation
The project achieved some results in the field, but a lot of the activities carried out for that purpose were implemented by partners of Unicef and their methodology is not documented yet.
The project has planned to prepare a “lessons learnt” document, but beyond the lessons learnt, some significant know-how is retained neither in Unicef nor at central government level. Complementary to the “lessons learnt” document, Unicef should involve the project partners into the simple drafting of the implementation methodology used (mostly community mobilisation and negotiations with the company, company creation and capacity building), before project memory vanishes.

b) Test open bidding process
A particular situation in Lowya’addo, the second location in Somaliland, keeps the project from completing the process. One solution that could both be tested for this specific context and for future replication would be to tender out the operation of the system, instead of having the local community trying to create an ad-hoc company (to no avail so far).

B.4.2. Mid-term

a)Mentoring
A significant mentoring effort is still needed to ensure the PPP are viable. Particularly, the private companies do not have the experience that was initially expected from the urban utilities. This has been partially compensated by some training, but the need for training and mentoring is still strong.
There is also a need for mentoring towards the central government, which is already taken into account by Unicef. But a simple support to provide would be to draft a short document on “how to establish a PPP for rural water supply”, listing all the key ingredients, steps to be taken and “no to do” list.

b) Preparing the central government to monitor the PPPs The second most important support to the central government should focus on reinforcing their ability to monitor the rural PPPs to keep learning from them, improving next batches and anticipate possible failures.

B.4.3. Longer term

a) Technical support centre
Given the distances of the locations and the size of the companies, it is very unlikely for them to face any technical breakdown on their own. Other countries such as Mauritania and Mali have developed experience in building up a national or regional technical support centre for rural water supplies. A similar concept could be tested (with a significant external support at first) to provide services like remote diagnosis, spare part procurement or even risk sharing on equipment, through a subscription of each company.

b) Careful use of external funding
It would be very risky to continue funding investment in the systems run by PPP companies on the same basis as when they were community managed. The financial autonomy is one of the key results expected from PPPs and private companies have to develop a commercial strategy regarding investment, in order to maintain the assets rather than “consuming” them (trade-off between immediate profitability and sustainability of operation).
If companies continue to expect rehabilitation funding from external agencies or donors, they will not include sufficient maintenance budget in their provisional operating accounts.

c) Fostering local government into rural water supply
Local governments are supposed to play an important role in rural water supply (according to the law of decentralisation). The current status of Local Governments is far from this ideal situation and requires a lot of progress of the decentralisation reform. But the first step will be to progressively involve the local governments, as owners of the facilities, thus responsible for the procurement of a company to operate them.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search