We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2018 Malawi: Evaluation of Malawi Child Protection Strategy 2012-2018



Author: Mei Zegers (Team Leader) Silvia Cifarelli, Bright Sibale, Sadia Mahmud

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

 

Background:

 

To address the great challenges to child well-being, the Government of Malawi with the support of UNICEF and other development partners, developed a five-year National Child Protection Strategy (NCPS) 2012-2016 extended to 2018. The overall outcome level result of the NCPS is to “protect children from violence abuse, exploitation and neglect while mitigating the impact of HIV on them”.
In line with the child protection (CP) systems approach, the main strategic actions are:
1) Coordination: Institutionalizing coordination for results
2) Legal & Regulatory: Fast tracking implementation of legal & regulatory frameworks
3) Capacities: Fortifying the child protection sector (human resources, infrastructure, financing)
4) Continuum of care: Improving quality, expanding scope & coverage of protection services
5) Process of care: Taking to scale case management, early identification & referral frameworks
6) Accountability: Building accountability mechanisms & Information Management Systems
The Malawi NCPS is accountable for the achievement of 4 main results:
1) Regulatory frameworks, institutional coordination, standards and enforcement frameworks in place by 2016 (Strategic actions 1 and 2)
2) Ten most disadvantaged communities in 250 Traditional Authorities adopt protective practices and have access to expanded range of protection services – early identification, case management and referral by 2016 (Strategic actions 4 & 5)
3) A comprehensive Child Protection model (including prevention and response services, human resources, infrastructure and financing mechanisms) tested in seven districts by 2014 (Strategic Actions 3,4,5)
4) Functional national Child protection Information Management System in Place and the Capacity of the Department of Social Welfare developed to operate it by 2014. (Strategic action 6)

 

Purpose/Objective:

 

The purpose of this evaluation was to provide the Government of Malawi, UNICEF and other key stakeholders with an independent assessment of the overall performance and results at the outcome level of the NCPS. The evaluation thus struck a balance between retrospective aspects focused on accountability and legitimacy and forward-looking aspects. Specifically, the evaluation feeds into refining and strengthening the MGDS III, new UNDAF and new UNICEF Country Programme. It also provides useful insights for the renewed political commitment to address violence against girls and boys with the other UN Member States in view of the agreement to support the 2030 Sustainable Development Goals (SDG) Agenda.

Specifically, the evaluation aims to assess the following standard OECD/DAC evaluation criteria:
• Relevance -  the extent to which the expected results are in line with the needs and rights of children, the national policies and priorities, UNICEF Country Programme and UNDAF • Efficiency of the Strategy, i.e. if and how the resources available  have been efficiently converted into concrete results for children and women; • Effectiveness of the Strategy, i.e. extent to which the outcomes were achieved;  extent to which the Strategy has contributed to the achievement of the strategic objectives;

• Impact of the Strategy in terms of contribution to identified and measurable longer-term changes occurred in the concerned districts.
• Sustainability of the Strategy, i.e. i) the possibility of continued benefits in the long-term due to the strategy and ii) the resilience to risk of the net benefit flows over time;
The evaluation covers the following additional dimensions:
• Scalability, i.e. extent to which the strategy components can be replicated in other target districts and similar countries;
• Gender and equity responsiveness of the strategy in terms of designing, implementation and results

Methodology:

The evaluation follows a theory based and systems approach to evaluate the degree to which the strategy contributed to positive changes on child protection in Malawi. The Evaluation Team needed to reconstruct a diagrammatic ToC for this purpose. The ToC is based on the NCPS results framework and underlying logic in the CP programming documentation. The Evaluation Team additionally developed an Ex Post ToC that fine-tuned the original ToC from the Inception Report after the field work and initial data analysis had been completed.

An Inception Phase was followed by a field work phase. The field phase included visits to districts of Mzimba, Dedza, Blantyre, Machinga, Ntcheu an, Chikwawa districts.  Field work included 105 interviews with key informants and other stakeholders and 42 Focus Group Discussions (FGDs) of which 12 consisted mostly of children. A Contribution Analysis Workshop was conducted with representatives of key stakeholders.

Qualitative analysis was conducted through the development of an evaluation question-based coding system in the software Atlas.ti. S ubsequently over 1880 quotations from interviews and FGDs were assigned to the codes to facilitate the identification of major trends by evaluation question. Quantitative data from available national databases were analysed with the support of STATA and Excel software. It should be noted that no baseline was conducted at the inception of the NCPS and a Child Protection Information Management System was not yet implemented and harmonised. This meant that only the nationally available data could be used for quantitative analysis.

A draft evaluation report was prepared, reviewed and discussed at a validation workshop in Malawi. This was followed up by the preparation of a next draft that incorporated the feedback obtained. The entire evaluation was conducted tobe as participatory and transparent as possible. An Evaluation Reference Group provided inputs throughout the process.

Findings and conclusions:

• The NCPS is highly relevant to the national child protection context in Malawi. The NCPS has set out a robust agenda for strengthening the national child protection system.
• The NCPS formed the framework for a child protection system incorporating appropriate and relevant prevention and response services in a relatively brief time. Contribution analysis indicates that the basic child protection systems structure exists in the focus areas. Nevertheless, there is a need to use a feedback loop across and between levels to improve the system at regular intervals.
• Six of the 13 targets were achieved. Three targets were partially achieved. Two targets on violence remain inconclusive because the NCPS does not define which indicators of violence are a priority and should be tracked. Two targets were identified as not yet achieved: child labor and on number of communities adopting protective child protection practices.
• The NCPS was implemented efficiently with the available resources. They were well allocated and implementing staff and volunteers maximized the way that they could use the resources. The evaluation found that the collection of data and analysis of data sets into an overall Child Protection Information Management System (CPIMS) need to be more harmonized. • The expected impact of the NCPS can be considered as sufficiently achieved. A comparative analysis with available data was only able to show impact in two areas: reduction in violence against women and early marriages in UNICEF supported districts. Nevertheless, the numerous interviews and focus group discussions supported this conclusion.
• Sustainability of the NCPS is most notably visible in the development of the overall child protection system with all its components. The development of the case management, capacity strengthening, district and community support and other components are key in this regard

Recommendations:

• Design a child protection strategy and results-based NCPS framework with indicators that can be objectively measured through a robust monitoring and evaluation framework. The new NCPS should be developed in a participatory and user-centred manner:
• Ensure the realistic costing, detailed description of roles and responsibilities and clearly designed coordination mechanisms in the next NCPS.
• Consider in the development of the new NCPS special attention to vulnerable children as related to existing strategies, policies and frameworks such as the existing National Plan of Action for Vulnerable Children, the Adolescent Girls and Young Women’s Strategy and National Strategy for Ending Child Marriage and other relevant strategies.
• Ensure clearly designed and implemented coordination mechanisms for the implementation of the new NCPS. with a special focus on guidance and monitoring of community and district level implementers.
• Place high focus on scaling up child protection system to new localities and sharing good practices and lessons learned.
• Promote awareness raising with strong focus on behaviour change.
• Continue and strengthen current district and community-based activities particularly Children’s Corners and Safe Schools initiatives.
• Further develop the case management system and as a national cause to be championed for the sake of the protection of children.

Lessons Learned:

• A well-defined Theory of Change with an evaluability assessment and adequate data collection planning prior to launching a future strategy are needed to ensure evaluability of results.
• Having indicators that are difficult to link and compare to actual results over time is challenging. 
• The lack of costing and appropriate allocation of resources to the NCPS resulted in an ambitious plan that was not as realistic as could have been the case.
• Coordination of planning and implementation needs to be more integrated and include focus on creating joint synergistic actions across agencies and thematic areas to be fully effective to address child protection.
• All child protection systems need to have a feedback loop to use incoming information in a well-planned manner.
• Within a systems approach, focus on the most vulnerable children still needs attention to ensure that their particular needs are addressed in the future while also maintaining a systems child protection based approach.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year: 2018

Office/Country: Malawi

Region: ESARO

Type: Evaluation

Theme: Child protection

Language: English

Sequence #: 2018/001 

New enhanced search