Evaluation database

Evaluation report

2012 Lebanon: Evaluation of Child Friendly Community Initiative



Author: Consultation and Research Institute

Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."

Background:

Despite the recent growth rates in the Lebanese economy, social policies did not succeed in bridging the gap between economic growth and social development and reducing poverty, vulnerability and exclusion1. In 2004-2005, 8% of the Lebanese population was living in extreme poverty (less than USD 2.4 per person per day) and 28.5% living under the upper poverty line (USD 4 per person per day)2.

According to the MDG progress report in 2010, closing poverty gaps is one of the three targets that Lebanon might not be able to achieve by 2015.Several studies in the nineties and the years following 2000 identified chronic poverty pockets in Baalbeck-Hermel, South of Lebanon, and particularly in North Lebanon3.Only 6% of Beirut population live under the upper poverty line while it rises in North Lebanon (Akkar and Tripoli) to 53% of which 18% living in extreme poverty4. In South Lebanon and the Bekaa, respectively 42% and 29% of the population is poor. This constant delimitation of poverty in Lebanon pinpoints the invariable problems of accessibility and quality of social services and gauges the limited capacity of the GOL to provide and maintain service delivery.

These regional disparities are not only reflected in skewed distribution of income and wealth but also in socio-economic issues, services and activities that continue to be concentrated in Beirut and its surrounding area of Mount Lebanon. For instance, in Akkar where population below the age of 20 constitutes 52% of the total population, the insufficient availability of basic services leads to unsatisfactory mother and child health outcomes, low educational achievement (illiteracy rates up to 30%), youth unemployment reaching 40% and outward migration rates as high as 25%5

In this context, the Children Friendly Communities Initiative (CFCI) is a joint initiative developed by UNICEF Lebanon Country Office and funded by the Italian Government through its Development Cooperation Office in Lebanon.

Purpose/Objective:

The overall objective of this report is to evaluate the pilot project of the CFC Initiative implemented in three villages in Akkar (DeirDalloum/Zouk el Mukashareen, Fnaydeq and Wadi el Jamous) and “to articulate what worked, what did not work and why, and capturing feedbacks and lessons learned in a participatory approach including beneficiaries and donor, for future interventions and scaling up. The evaluation will evaluate successful and not successful interventions, as well as collecting in systematic manner valuable information from the 3 communities, planning and implementing partners and donor through which the project is conducted”6.

The evaluation has been conducted in line with UNICEF evaluation report standards and aligned with OECD DAC criteria7. In other words, the evaluation aims at verifying the CFCI in terms of relevance, efficiency, effectiveness, impact and sustainability.

All through the evaluation process, an equity-focused approach is adopted in view of increasing the understanding of the causes and drivers of inequities and exclusion in the deprived areas as well as examining the outcomes for the most disadvantaged beneficiaries of the Initiative.

The outcomes of the evaluation will be used for a better understanding of the actual status of the Initiative and for the improvement of the future interventions and activities within the Initiative and other similar projects.

Methodology:

The data collection and analysis for the evaluation followed a qualitative approach and participative methods. The evaluation has been conducted through three phases: (1) Desk review, (2) Field work, and (3) Desk Analysis. Each of these phases is presented in more details in the following sections.
- Desk review: The first phase of the evaluation has the purpose of verifying the CFCI outputs through the analysis of available documents and data pertaining to the Initiatives.
- Field work: In order to gather a wide range of opinions on the implementation and achievements of the CFCI, 22 semi-structured interviews have been conducted with stakeholders in Beirut, Wadi El Jamous, Fnaydeq and DeirDalloum: The donor representative, Government representatives and members of the CFCI advisory board, UNICEF staff and consultants, members of the Municipal councils of the targeted villages and members of the working committees designed by the CFCI8.
- Desk analysis: The analysis of the information obtained from the desk review allowed the working team to obtain a deeper understanding of the objectives and outputs of the CFC Initiative and their actual status. In addition, qualitative information was collected from the fieldwork following a standardized checklist. The collected information was then processed and classified following a set of indicators (e.g. “relevance to community needs”, “time efficiency, etc.) directly linked to the standard evaluation criteria (relevance, efficiency, effectiveness and sustainability). Based on this classification, the collected data was scored on a scale ranging between 0 and 2 (0 being an unmet criterion and 2 being a met criterion) 

Findings and Conclusions:

Relevance:
The CFCI is consistent with the UNDAF 2010-2014 expected outcomes especially Outcome 4 pertaining to weak governance and the challenges of regional disparities and socio-economic exclusion.
The CFCI is consistent with the Italian Cooperation Development Office in Lebanon notably the ARTGOLD project which is its most important intervention in Lebanon.
The CFCI is consistent with the Social Policy component of the UNICEF Country Programme 2010-2014. It is in line with the results pertaining to the capacity building of local communities’ actors as well as the transferring of experience of integrated packages of UNICEF supported interventions into planning, budgeting and operational guidelines for decentralized development.
Although CFCI is consistent with the expressed needs of targeted communities stakeholders (participatory approach), no thorough needs assessment was carried out in order to determine the actual priorities in terms of needs prior to the design and implementation of the CFCI.

Efficiency:
The resources have been managed in a transparent way and have respected in general the allocation of funds as per the budget.
Based on the actual expenses of year 2010 of the CFCI, 22% of total budget covered the costs of human resources, including UNICEF staff and the consultants implementing the project. The cost of UNICEF staff comprises 19% of the total budget and only 3% is allocated to external consultants. This distribution of funds shows the involvement of UNICEF staff in all the phases of implementation without an extensive resort to consultants or NGOs.
The remaining 76% of 2010 actual expenditures covered all the costs of logistics and the implementation of all the CFCI interventions in the three pilot villages.
Although the evaluation does not comprise auditing purposes, it is important to note that all the implementing partners were selected based on a transparent and competitive procurement process.
The logical framework suggested has not been followed due to the modification of the initial project proposal and therefore no systematic performance monitoring and reporting was carried out.
The involvement of the UNICEF team in the implementation of all the activities has implied a timely implementation of the CFCI.
The involvement of local committees in the management of the CFCI increased the ownership of the local communities vis-a-vis the CFCI and was based on the concept of “learning by doing” which consisted of a coaching and mentoring methodology adopted throughout the CFCI project.

Effectiveness and results:
The overall objective of the project has not been met at this stage of implementation and it is not possible to measure its level of achievement. So far elements of the proposed model of decentralized planning and resource management have been developed. 
The central government represented by relevant ministries was not consistently involved in the implementation process which is necessary for the validation of such a model and its up streaming at a later stage.
All the activities planned to achieve the first expected outcome “To achieve better local governance” have been accomplished.
Most of the interventions of the CFCI pertain to the rehabilitation of basic infrastructure and the provision of service in support to education, culture and healthcare.

Sustainability:
According to different sources, most of the interventions pertaining to rehabilitation of infrastructure do not have a specific planned maintenance system.
Despite their limited funds, the municipalities are providing for the time being the necessary funds for the sustainability of interventions pertaining to the provision of services. However, there are no maintenance plans or systems in place.
At present, none of the interventions have clear and set financing plans necessary to secure the sustainability of the interventions following the end of CFCI pilot project.
The CFCI exit strategy consists of a set of studies undertaken in 2011 which aim to support the targeted municipalities for their future interventions and fundraising activities.
The CFCI is currently aiming at strengthening the different committees that have played a role in the CFCI in order to be able ensure the sustainability of the implemented interventions.
On the basis of the evaluation findings UNICEF and local partners donor will determine the relevant implementation approach of the remaining period.

Recommendations:

Recommendations for development assistance:
In order to maximize their impact and effectiveness, development interventions ought to be integrated within established national strategies such as the National Physical Master Plan or strategic plans pertaining to relevant government agencies such as MEHE (Ministry of Education and Higher Education). UNICEF would gain from selecting the parts of these national strategies that are relevant to UNDAF and the UNICEF country program and designing its development interventions on their intersection.

The chances of success of UNICEF interventions are maximized when they are founded in existing local or regional strategic plans. Such plans have been developed in a number of Lebanese cities/regions and UNICEF could tackle aspects or issues addressed within these plans.

This development approach which takes national and regional strategies as a platform for individual development interventions has many advantages including avoiding overlap in the work of various agencies, coordinating various development initiatives, and ensuring that the projects developed respond to actual community needs and contribute to the achievement of fundamental development objectives.

Recommendations specific to the CFCI
The CFCI is a pilot project whose purpose is to test the initial design and extract the lessons learned in order to perfect the program prior to its deployment to a wider geographic area. Following is a number of guidelines that if followed will increase the chances of success of the wider initiative.
Conduct an objective needs assessment in each of the targeted areas. The needs assessment should comprise an overview of the various aspects that together constitute the livelihood and social conditions of the targeted community. The surveys which were implemented in the three pilot villages may be used as a foundation for a needs assessment that reorients the interventions in the proper direction
Carry out a mapping of development interventions in the wider region in order to avoid overlap of initiatives and in order to determine unaddressed development issues that may constitute appropriate targets for UNICEF
Have a clear and focused overall objective to be achieved through a set of interlinked interventions that targets specific dimensions in order to maximize impact. Objectives may be focused on the intersection of UNICEF’s areas of interest and the results of the needs assessment so that they may target either a specific profile of beneficiary e.g. school children or a specific need such as maternal health
The effectiveness and efficiency of the Initiative will benefit from increased cooperation on a number of different levels:
Cooperation with complementary initiatives implemented by other agencies;
Cooperation with neighboring municipalities;
Effective cooperation within the local community in the form of an institutionalized and empowered committee that encompasses the various stakeholders whose buy in is essential for the success of the project. This committee needs to have a specific mandate whereby each member may be in charge of a specific sub-project. This is a very important guarantee of the sustainability of the project.
Continue the successful practice of “learning by doing” which consists of coaching and mentoring municipalities throughout the process of implementation.
Design baseline indicators for each project in order to allow the continuous follow up of its impact on the targeted aspects of living conditions of the communities.
Strengthen the capacity of healthcare providers to provide communities with health related behavioral change messages in line with decentralized led interventions.

Lessons Learned:

The purpose of the evaluation is to verify whether a model of decentralized planning and resource management is in place and to identify the successful and non successful interventions within the CFCI pilot project. The evaluation aims at providing a series of lessons learned and a set of recommendations based on the identified factors of success as well as weaknesses and gaps.

The CFCI has been so far able to set some elements of “a model of decentralized planning and resource management able to address socio-economic exclusion in the most deprived areas of Northern Lebanon”.

The initiative comprises panoply of interventions that initially aim to enhance local governance, to provide necessary services and to mobilize the local community for development purposes. The evaluation has identified in previous sections of the report the extent of success of interventions based on the criteria of relevance, efficiency, effectiveness and sustainability. As specified in the evaluation request for proposal, the impact of interventions cannot be assessed at this stage as the results are still premature.

The following section presents a number of lessons learned from the results of the evaluation followed by two sets of recommendations, one of which pertains to development initiatives in general and the other tackles the CFCI in a specific manner.
- The source of most development problems and therefore the solutions for these problems are often found at the national or regional level (e.g. basic infrastructure issues such as power, water, public education policy, healthcare system, etc.). Therefore, any program that exclusively focuses its efforts on the local level is bound to have relatively limited impact that is restricted to the alleviation of the direct symptoms of national/regional development malfunctions.
The interventions were selected based on a participatory approach on the expressed needs of local stakeholders. However, because of the lack of a systematic needs assessment/strategic plan, these stakeholders themselves do not have clear idea of what is needed.
The initiative was successful in terms of work execution, respect of timeline and the transparent use of funds. Thus, the main challenge of the CFCI resides in its initial design. In addition, the initial project logical framework was not referred to during the implementation process. A ongoing monitoring was carried out which prevented from a continuous and systematic identification of the divergence of the project from its initial track during the implementation phase.
Rather than going through NGOs to implement the CFCI at the local level, UNICEF directly went to the field and the initiative was implemented through local stakeholders. This approach maximizes efficiency by eliminating unnecessary overhead and ensures ownership of the project by the targeted local communities. However, it is important to note that in certain instances, NGOs have the capacity to play a positive role within a proper context.
Municipalities may not be the best implementing partner for all types of interventions for instance activities related to education such as the “Revolving textbooks” project and the sewing atelier.
Sustainability issues were identified in most interventions and are exacerbated with municipalities being caught with expenses to be maintained on a long term knowing their already limited budget. These sustainability problems are magnified by the fact that the implemented activities are not considered as a priority by targeted municipalities (such as, municipal library, cultural buses, sewing atelier).
The way that CFCI tackled the complex cultural barriers of communities through its participatory approach with local stakeholders, in bringing together different actors in the community to work together, set committees and working groups (which sometimes worked and sometimes did not)can be a crucial lesson for future endeavors of such projects in similar contexts of marginalized and remote communities.
As the interventions planned have mainly addressed bottlenecks on the supply side, health awareness stands out as one of the main barriers on the demand side improvements on the supply side should be matched by improvements on the demand to increase access for quality health services.

1 MOSA and UNDP, 2011, National Social Development Strategy
2 According to MoSA and UNDP’s “Poverty, Growth and Income distribution in Lebanon” (2008). “The upper poverty line is defined as the value of the basket of goods and services actually consumed by households whose food and every intake is equal to the minimum requirement of 2200 calories per person per day. At the official exchange rate this poverty line translates into about USD 4 per capita per day.
Mapping of Living Conditions - MOSA/UNDP, National Physical Master Plan of the Lebanese Territory - CDR and Poverty, Growth and Inequality in Lebanon - UNDP
4  UNDP, Millennium Development Goals, Lebanon, Progress Report, September 2011
5 UNICEF Annual Report for Lebanon, 2010
6 Request for Proposals, UNICEF – Evaluation of Child Friendly Community Initiative, RFP/LEBA/2011/001, Lebanon Country Office, 25 July 2011.
7 OECD DAC Evaluating Development Co-operation, UNICEF Evaluation report standards, UNICEF – Adapted UNEG Evaluation Reports Standards, UNICEF Guidance On Equity – Focused Evaluations.
8 The detailed agenda of the mission is reported in Annex 2



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search