We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 Sri Lanka: Evaluation of the EU-SEM project: Widening Horizons and Creating Opportunities for Sustainable Livelihoods in North and East Sri Lanka



Author: International Institute of Development Training

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

This report describes the findings and recommendations for the final evaluation of the EU-funded project “Widening Horizons and Creating Opportunities for Sustainable Livelihoods in North and East Sri Lanka” which was implemented by UNICEF in partnership with the Government of Sri Lanka, local authorities, community-based organisations and private sector in the Northern and Eastern Provinces in 2012-2014. 
The end of the armed conflict in the Northern and Eastern provinces of Sri Lanka in May 2009, required the provision of basic services such as housing, health, water and sanitation, education etc., social protection/support schemes and livelihood opportunities, among many other necessities, to those populations who were returning to their original habitats after a long period of time. The United Nations Children’s Fund (UNICEF) with financial assistance from the European Union initiated an integrated project “Widening Horizons and Creating Opportunities for Sustainable Livelihoods in the North and East”, with the overall objective of improving the livelihood potential and living conditions of vulnerable communities and strengthening of government service provision capacity.

The project consisted of four interventional components: Health and Nutrition, Child Protection including Youth, Education, and Water, Sanitation and Hygiene, designed in response to the demands as outlined above, and was implemented in two Divisional Secretariat (DS) divisions in the North and in eleven DS divisions in the East, selection of which was based on agreed criteria to include the most vulnerable categories, namely, children and youth, women, and persons with disabilities, as project beneficiaries.

Purpose/Objective:

Objectives of the evaluation were to:

  • Assess the degree to which planned results, at various levels of the result chain have been achieved – specially assess the extent to which equity issues have been addressed via project results;
  • Assess the relevance of the project with regard to national and institutional priorities and analyse the appropriateness of the strategies, including partnerships and the management arrangements that were put in place;
  • Draw lessons which can inform UNICEF on the implementation of similar programmes;
  • Provide recommendations on areas for scale up and upstream work related to supporting policy and advocacy in line with the new country program strategy.

In the fulfillment of the conditions of the contribution agreement between the EU and UNICEF, UNICEF contracted the International Institute of Development Training, Sri Lanka, to carry out the end of project evaluation, having followed its procedures in the procurement of services.
Four standard OECD/DAC evaluation criteria: relevance, efficiency, effectiveness and sustainability of the key strategies, activities and implementation process, as per the TOR, were used to evaluate each of the project components of WASH, Health and Nutrition, Education and Child Protection. Due to the nature of the project and as agreed with the donor, the impact was not assessed taking into account a very short time that passed after the completion of the project activities. However, as the project was implemented in multiple districts across multiple sectors, analysis was supposed to be done to assess the efficiency and effectiveness of coordination systems. The scope of the evaluation included an assessment and analysis of reported results at the output and outcome level according to the project design.  Evaluation of each sector components has also been, wherever possible, looked through the equity and gender perspective.

Methodology:

As the project was implemented in multi-districts across multiple sectors, analysis should be done to assess the efficiency & effectiveness of coordination systems
The evaluation was conducted in July-August 2015, on a purposively selected sample of seven DS divisions, two divisions from the North and five from the East. The DS divisions in which the highest number of project interventions was implemented were purposively selected for the evaluation to ensure that the evaluation would cover as many project interventions as possible. 
The evaluation employed both qualitative and quantitative research methods: desk research, key informant interviews (KIIs), focus group discussions (FGDs), community meetings, and site observations. A total of 38 FGDs (14 in the North and 24 in the East) and 33 KIIs (16 in the North and 16 in the East and 1 in Colombo) were conducted during the evaluation using separate questionnaires for each project component and each category of respondents. Due to the time constraints coupled with the vast geographic scope of the project, fieldwork took more of a qualitative turn.  The quantitative data was obtained by examining the record keeping data maintained by the four sectors in the respective DS divisions, existing reports, and quantitative information obtained through KIIs.

Findings and Conclusions:

The EU-SEM Project has fully met its output targets of improving the social status of beneficiaries, especially women, through coordinated and collaborative efforts between sectors; project achievements, which varied across sectors, could be attributed to the use of a combination of effective strategies; coordination among multiple public sector service providers; and regular monitoring and use of corrective measures by UNICEF;  strong commitment and ownership of the Government of Sri Lanka (GOSL) at all levels.
Whereas overall sustainability depends very much on the ownership of the project interventions and results, to a greater degree when they are Government  or beneficiary-owned  with a questionable level of sustainability at the community level, funding mechanisms may significantly affect the sustainability and institutionalization of project interventions.
With regard to improving the nutrition status, in addition to the use of applicable behavior change strategies to improve knowledge, attitudes and skills of mothers on disease prevention, child care, breastfeeding and complementary nutritious foods, etc., and the delivery of growth monitoring services and distribution of food supplements, the status of household food security and the income earning capacity of the families, are also known to exert a critical influence.
In child protection, overall output target in vocation training was successfully achieved delivering vocational skills training to 1,283 out of a targeted 1,300, which is an achievement of 98%. Within 06 months of completion of training 42% of those who received vocational training were either in employment or were established in successful self-employment activities.

Recommendations:

  1. Strengthen interagency collaboration. UNICEF, in future should partner with relevant sister UN agencies in projects where interventions outside of UNICEF’s mandate are included.
  2. Ensure robust monitoring and evaluation mechanisms in place for all future projects. This includes, inter alia, development of a set of effective indicators at different levels, regular review of the project objectives, targets and achievements as well as effectiveness of monitoring methods. 
  3. Increase cross-sectoral integration and convergence.
  4. Adopt flexible approach towards development programmes. When conceptualizing and planning development programs for vulnerable populations and/or for areas with potentially adverse physical, climatological or geological features a flexible planning/ implementation model that allows for appropriate deviations from centrally planned technological interventions should be adopted.
  5. Develop selection guidelines for integrated projects.  In planning future integrated projects for vulnerable communities, it would serve the interest of vulnerable families more effectively, if GOSL, EU and UNICEF could develop a guideline for selecting areas for integrated projects based on two basic principles (a) co-location of services and (b) convergence of an optimum number of services on selected households/families through the planned project.
  6. Integrate behaviour change strategies at an early stage. BCC initiatives should be planned as early as possible, and respective communication strategies should be developed for each programme component.
  7. Align hardware and software activities. In an integrated program where both physical infrastructure (hardware), and community organization and behaviour change interventions (software) are key designed strategies. 

Lessons Learned:

An integrated project for vulnerable populations must be designed and strategically implemented to move beneficiaries beyond outputs to achieve outcomes such as helping mothers to progressively increase the weight of their babies who are under-weight.
Development projects that are designed to achieve behaviour change outcomes especially in vulnerable communities should formulate and use strong and effective community mobilization and organization activities approaches relevant to local contexts and culture. It appears that with regard to the health and nutrition and child protection components that the degree of success in establishing active community support groups was not very high, though.
Strong learning was also with regard to monitoring system: even if there was a strong coherence between most of the project outcomes for the identified beneficiary groups and the indicators and methods of measurement, there were a few important outcomes with regard to which coherent and logical linkages had not been established. This led to the mismatch between outputs and outcomes and their indicators, thus disabling the true picture of achievements to emerge.
The project designers had introduced innovative activities which go beyond the mandate of UNICEF, the main UN partner in this project, though after all it was considered that UNICEF should not overburden itself by undertaking technical responsibilities for programming income generation, micro credit, self-employment grants and vocational training even in support of vulnerable communities.
Finally, though there were constraints going beyond control (floods and droughts), it was felt that more attention should be paid to the assessment of the capacities of government partners and service providers before the commencement of the project.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2015

Country:
Sri Lanka

Region:
ROSA

Theme:
Advocacy and Communication

Type:
Evaluation

Partners:
Government of Sri Lanka, NGO partners

Language:
English

Sequence #:
2015/005

New enhanced search