We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2018 Liberia: Formative Evaluation of Be A Change Agent Project



Author: Kara Apland, Elizabeth Yarrow and Jorun Arndt

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

 

Background:

Adolescents in Liberia face a combination of complex challenges impacting on their well-being, development and life trajectories. These challenges, which include lack of access to (quality) education, income poverty and underemployment, early and unsafe sex, lack of family support, gender inequalities and violence, are particularly pronounced for girls. In 2010 the World Bank undertook an ‘Assessment for the Economic Empowerment of Adolescent Girls and Young Women’, which considered the multiple and interlinked challenges that Liberian girls face, and recommended the development of integrated programmes to address these. The assessment concluded that in order to effectively empower adolescent girls, programmes should combine vocational, business, life skills and literacy, with parenting skills, sexual and reproductive health (SRH) interventions, and initiatives aimed at incorporating psychosocial counselling and peer support. Against this background, and in recognition of the Liberian Governments’ increasing commitment to prioritising the needs of adolescent girls, UNICEF developed the ‘Be a Change Agent Project’ (B-CAP) in collaboration with key line ministries and two partner organisations, SHALOM and Samaritan’s Purse International Relief in 2016. In 2018 Coram International conducted a formative evaluation of B-CAP in Liberia.  B-CAP, which operates in two of Liberia’s most deprived urban slums. The project aimed to reach 1500 ‘at-risk’ adolescent girls between the ages of 10-19. B-CAP’s holistic design was informed by a growing consensus that in order to effectively empower adolescent girls, programmes should deliver an integrated package of support services, including vocational, business, life skills and literacy training, as well as parenting skills, sexual and reproductive health (SRH) interventions, psychosocial counselling and peer support.

Purpose/Objective:

The formative evaluation took a learning-oriented approach: its primary purpose was to generate evidence to inform and improve B-CAP’s implementation in its second phase. To this end, the evaluation sought to determine the extent to which B-CAP has achieved its planned (and any unintended) results, to identify factors that affected its implementation, and to develop recommendations which can strengthen the project going forward. In addition, the evaluation sought to identify good practices and lessons learned to guide the development of future interventions for adolescents. The evaluation’s primary audience includes UNICEF Liberia, the UNICEF West Central Africa Regional Office and the project’s donors. In addition, the evaluation may be used by relevant Liberian government ministries, NGOs and development partners, to strengthen interventions and initiatives aimed at adolescent empowerment.
The evaluation covers the period from the start of the project’s implementation in February 2016 until the end of October 2018, with the geographical focus on the implementation sites in West Point and New Kru Town communities in Monrovia, Montserrado County.

Methodology:

The evaluation was designed to assess the relevance, effectiveness, efficiency and sustainability of B-CAP in line with the Organisation for Economic Co-operation and Development/ Development Assistance Committee (‘OECD/DAC’) Principles for Evaluation of Development Assistance.  It employed a mixed-methods approach, in order to draw on the strengths of both qualitative and quantitative methods: to gather data that was rich, accurate and measurable, and ensure the validity of results through triangulation. In addition to conducting a thorough review of relevant project documents to understand the project’s design, costs and other detail related to its implementation, the evaluation team conducted primary data collection in both West Point and New Kru Town. Data collection methods included Key informant interviews (N=28), In-depth interviews(N=34), Focus group discussions (N=3), Direct observation of life skills training and alternative basic education sessions (N=3), and B-CAP beneficiaries were surveyed (N=126). Qualitative and quantitative data were reviewed and analysed systematically to identify key themes, patterns, relationships and explanations relevant to the evaluation questions. A validation workshop was held in Monrovia on December 11, 2018 in order to gather feedback from key stakeholders on the validity of evaluation findings and analysis, and the actionability of recommendations.

Findings and conclusions:

Relevance: Findings on B-CAP’s relevance are largely positive, however the evaluation did identify a potential limitation in this area. Whilst B-CAP’s selection criteria are well designed to identify the most vulnerable and disadvantaged adolescents (in line with the UNICEF key principle of equity), its broad scope has arguably undermined the relevance of project activities to meeting beneficiaries’ needs.
Effectiveness: B-CAP has effectively contributed to the empowerment of vulnerable adolescent girls. The evaluation found that the combination of mutually reinforcing services and support provided by B-CAP has led to improved outcomes for beneficiaries.
Efficiency: The evaluation found that B-CAP is using available resources economically to reach its objectives.  The project has made efficient use of staff time, in order to maximize outcomes for a large number of beneficiaries.
Sustainability and Scalability: Given that B-CAP is in its early stages, evidence on the sustainability of outcomes for individual beneficiaries is limited. The projects’ design was found to be conducive to sustainable outcomes, however; by addressing multiple aspects of adolescent girls’ vulnerability and building their capacities in a number of complementary areas, B-CAP’s approach reduces the risk that challenges in one aspect of a project graduate’s life will undermine achievements in another.

Recommendations:

A number of recommendations emerged from the evaluation. These were validated, discussed and developed during a participatory workshop held with key stakeholders in Monrovia on 11 December, 2018 in order to ensure that they are practical and actionable. Key recommendations include:

  1. Reconsider selection criteria, to ensure that they reflect B-CAP’s aims. After refining targeting, consider strategies for ensuring that equity considerations are taken into account during recruitment.
  2. Establish a comprehensive database to gather standardised data on all beneficiaries across project sites.
  3. Consider introducing elements of case management into B-CAP to improve the service and ensure that the complex needs and vulnerabilities of each individual are addressed. Whilst B-CAP is not intended to provide a comprehensive child protection service, elements of a case management approach would strengthen the project without adding significant additional costs.
  4. Clearly establish that it is B-CAP’s policy to include beneficiaries who become pregnant. All project messaging should be reviewed to ensure that it is non-stigmatising to pregnant girls.
  5. Consider the development and provision of additional services including: child care support for beneficiaries with young children; non-gendered skills training opportunities informed by a market assessment; a scholarship fund to support beneficiaries’ access to formal education; a discretionary fund to address any urgent and unmet needs.
  6. Draft a concrete and actionable sustainability strategy. This should include clear targets for increased government responsibility for funding and delivering B-CAP. UNICEF should consider engaging local government / service providers in the delivery of B-CAP, and forming partnerships with established youth services to deliver B-CAP at a reduced cost.

    Lessons Learned:

A number of ‘lessons learned’ emerged from the evaluation, which can be used to inform future adolescent empowerment programming:

  1. A multi-disciplinary approach is key to effective empowerment programming: B-CAP’s holistic and multi-disciplinary project design has been key to its success. Particularly in a context where beneficiaries face multiple and interrelated types of disadvantages, adolescent programming should address different dimensions of vulnerability in order to achieve empowerment outcomes.
  2. Building self-esteem reinforces other effects: A sense of self-esteem and self-worth is integral to achieving results in other areas. Self-esteem creates a foundation which improves the likelihood of success in relation to other outcomes. These outcomes reinforce self-esteem, resulting in a positive feedback loop.
  3. A standardised monitoring database should be established and implemented from the beginning of a project: Maintaining a basic database is neither costly not time-consuming, but is necessary for basic monitoring, evaluation and planning, as well as for demonstrating project outcomes.
  4. At the project design stage it is important to balance concerns about the project’s purpose and aims with equity concerns: Prioritising the inclusion of all vulnerable groups or marginalised persons may have the unintended effect of compromising the coherence of the project, or the relevance of its activities to the target group.
  5. It is important to be sensitive and responsive to unintended consequences, particularly in gender empowerment programming: Given the pervasive nature of discriminatory gender norms and expectations, it is likely that project activities may inadvertently replicate the gendered norms and inequities that exist in society. It is important that programming remains self-aware in order to identify these unintended consequences when they occur and correct them.

 

Please find the attached labelled as follows:

  • Evaluation Report - Report
  • GEROS Evaluation Review - Part 2
  • GEROS Feedback Summary - Part 3
  • Evaluation Management Response (EMR) - Part 4


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year: 2018

Office/Country: Liberia

Region: WCARO

Type: Evaluation

Theme: Child Protection; Multi-thematic CP

Language: English

Sequence #: 2018/001 

New enhanced search