2017 Sierra Leone: External Evaluation of Effectiveness of UNICEF Nutrition Accelerated Reduction of Child and Maternal Under-Nutrition in Seven Districts of Sierra Leone
Author: Christina Blanchard-Horan
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as ‘Part 4’
The Government of Sierra Leone is committed to improve the nutrition status of children during their first 1,000 days of life from conception to two years of age. With the efforts undertaken by the national government, a decreasing prevalence trends are noted for both stunting and wasting. However the national figures do not accurately capture the status of severe acute malnutrition prevalence countrywide, where almost 70,000 children still suffer from severe acute malnutrition and more than 400,000 children are stunted.
Analysis on the underlying causes of under nutrition in Sierra Leone indicates that poor infant and young child feeding practices, frequent childhood illnesses, insufficient coverage of primary health care interventions, poor quality, low demand for health services (exacerbated by the recent Ebola outbreak), and poor sanitation – are the predominant factors.
UNICEF supports the Ministry of Health and Sanitation to prevent and treat child undernutrition. The governments’ Integrated Management of Acute Malnutrition (IMAM) programme provides treatment to the severely malnourished children in Outpatient Treatment Program (OTP) facilities and inpatient care in health facilities (IPF). In order to prevent child malnutrition, UNICEF supported the establishment of mother-to-mother support groups in communities for screening, referral and community IYCF counselling (as a part of stunting reduction). UNICEF with its implementing partners also supports the implementation of biannual Mother and Child Health Weeks (MCHWs) for vitamin A supplementation and deworming; and in 3 districts, supports the micronutrient powder (MNP) supplementation pilot programme.
The Programme objectives are:
• Improve the quality of the treatment of severe acute malnutrition by increasing adherence to national Integrated Management of Acute Malnutrition (IMAM) protocols at Outpatient Therapeutic program (OTP) and In Patient Facilities (IPF); strengthening the supply chain management of Ready to Use Therapeutic Foods (RUTFs); and improve communities’ adhesion and participation in IMAM activities;
• Improve Infant and Young Child feeding practices at the community level through community-based structures and Primary Health Care Units (PHUs).
• Maintain adequate coordination of nutrition partners and activities from decentralized to central level.
In April 2016, Global Health Liaisons LLC was contracted by UNICEF/Sierra Leone to evaluate the Irish Aid Funded Nutrition Programme. The purpose of this evaluation was to understand how the nutrition programme in the seven Irish Aid-supported districts worked and determine what worked well, where, why, and under what circumstance(s). It also sought to generate knowledge, as well as identify best practices and lessons learned.
The objectives of the evaluation include:
• Assess the relevance, appropriateness and coherence of the nutrition programme in relation to global, regional and country programme strategies, and in relation to Sierra Leone.
• Assess the effectiveness and efficiency of the nutrition programme in the 7 Irish Aid supported districts in terms of delivering services to infants and young children and in promoting appropriate IYCF behaviours/ practices.
• Assess UNICEF’s leadership, guidance and technical support at all levels and the adequacy of staffing/institutional capacity to respond to the lead role UNICEF is expected to play at the field level for contributing to sustainable and equitable reduction of child undernutrition.
• Assess to what extent is the programme is sustainable in addressing undernutrition in young children with particular attention to the less reached, disadvantaged and vulnerable groups.
• Provide forward looking learning, conclusions and recommendations for strengthening UNICEF’s leadership and advocacy, country programme response and partnerships for reducing child undernutrition in Sierra Leone.
This evaluation was based on the 17 research questions based on the OECD criteria namely (i) Relevance, Appropriateness, and Coherence, ii) effectiveness and efficiency, and iii) scale-up and sustainability. Six data collection methods were used – these include: a literature review, nine key informant interviews (KIIs), in-depth and 1,920 semi-structured interviews, seven focus group discussions (FGDs), and secondary data from surveys and reports – including donor reports, previous programme evaluations decentralised monitoring reports etc . In addition programme monitoring data was collected for the period 2013 -2015. Analysis of data from KIIs, semi-structured interviews, and secondary data sources contributed to the understanding of programme impact on health over time and validation of reported results and findings from the literature review. Data collection and reporting were grouped into the primary themes outlined by the Terms of Reference.
Findings and Conclusions
Relevance, Appropriateness, and Coherence of the Programme
Assess the relevance, appropriateness and coherence of the nutrition programme in the 7 Irish Aid supported districts in relation to global, regional and country programme strategies in relation to the specific context of Sierra Leone.
UNICEF has been a major advocate for child health and contributor to the development of health policy that supports nutrition improvement around the globe. UNICEF Sierra Leone collaborated with other UN partners to establish strategies, while appropriately leveraging existing relationships with the Sierra Leone MoHS and DFN. This contributed to nutrition policy change and identification of barriers and bottlenecks to improve IYCF practices. They built capacity at government facilities to address these by expanding the availability of IYCF counseling through health facilities, community health workers (CHWs), and Mother Support Groups (MSGs) in the community.
UNICEF has made numerous contributions to the development of knowledge around the problem of malnutrition in Sierra Leone. At a national level, the FNS Policy Implementation Plan of 2012-2016 was developed by the MoHS, with support from UNICEF. Together with WHO and the World Food Programme (WFP), UNICEF has been a key partner in the treatment of acute malnutrition in Sierra Leone.
UNICEF supported knowledge building with the implementation of the Multiple Indicator Cluster Survey (MICs) survey; they conducted a situation analysis in 2013, and supported a feasibility study for micronutrients in 2013, which was followed up with a pilot in 2015. Targets for micronutrients were not achieved in 2015. The target was 50% of the target facilities were reporting postpartum vitamin A supplementation, while UNICEF achieved 45%. Of the 50% targeted for reporting on vitamin A and deworming for children under five years of age, 31.3% of the target was achieved for micronutrient powders for children. Also in 2015, UNICEF supported the Post Ebola Comprehensive Food Security and Vulnerability Analysis (CFSVA). Following Ebola, they supported a facility assessment. UNICEF supported the World Health Organization (WHO) to design the guides for the integrated management of acute malnutrition (IMAM). UNICEF partnered with the Worked Food Programme (WFP) to roll out the IMAM guidelines and associated training for the use of RUTF, and for Ebola, ready to use infant formula (RUIF) to treat severely malnourished infants.
The IYCF counseling guide, the National Food and Nutrition Security Implementation Plan of 2013, as well as decentralized monitoring results indicators were designed and aligned with global standards. Most recently (2016), the M&E handbook for infant and young child feeding (IYCF) and CMAM was developed in a partnership that included UNICEF, WHO and WFP to strengthen IYCF practices in Sierra Leone. Infant and young child feeding promotion would emphasize exclusive breast feeding up to six months, and complementary feeding with continued breast feeding until two-years-of-age.
The nutrition programme has been widely appreciated at multiple levels (e.g. national, district, PHU, community) for its role in nutrition advocacy, planning, and support. UNICEF used its unique position to provide guidance and advocacy to the government. UNICEF practiced relevant advocacy to develop national guidelines and programmes in partnership with MoHS, local and international NGOs, and with other UN agencies. The Country Programme Action Plan (CPAP) was an agreement between the government of Sierra Leone and UNICEF that contains a logical framework for their collaboration. The framework has been the basis for UNICEF work in Sierra Leone for the past 5 years. The UNICEF and DFN-MoHS collaboration has been critical to the successful integration of nutrition services into the existing health system. Following was a summary of findings related to relevance, appropriateness, and coherence:
• UNICEF appropriately engaged local non-government organizations (NGOs) as implementing partners, building local capacity to support programme monitoring for quality assurance, training, and supply management at the district and facility levels, as well as providing support for the community.
• The IMAM strategies to scale up facilities and train healthcare staff have been appropriate to the region. Though some of these efforts were delayed during Ebola, UNICEF and MoHS have made strides toward increasing the number of facilities and training and retraining staff to provide quality nutrition services.
• Efforts toward collaborative multi-sectoral initiatives that provide treatment and addresses food security were not evident. Although this was discussed during national level interviews, it was not evident at the facility level.
• Community training through mother support groups (MSGs) was an appropriate and efficient method of reaching hard-to-reach communities with IYCF counseling, and was consistent with UNICEF global strategies to address malnutrition. However, there was a need to ensure for powerful messages and champions that can diffuse appropriate IYCF knowledge to communities.
• The reporting framework for the programme was captured in Log Frames, housed in proposals and progress reports; these were lacking details and indicators on supply management, Child Health Days, nutrition screening and referrals, and training.
• Coherence of programme strategies and planned results were evidenced not only by the connection at national levels, but also by the connection between community and facilities. Facilities valued community counseling and referrals and coordination with district-level implementing partners (IPs). Mother support group (MSG) members, MSG Leaders, CHWs, and Maternal and Child Health aids (MCH Aids) indicated that they had a strong connection with the facility, some going daily, and/or weekly to the facility. However, coherence was not evidenced in supply provision.
• The connection between district level implementers and facilities was also demonstrated during interviews with staff at outpatient therapeutic programme (OTP) sites, when they stressed the importance of training and interaction with IPs.
Effectiveness and Efficiency
Assess the effectiveness and efficiency of the nutrition programme in the 7 Irish Aid supported districts in terms of delivery services to infants and young children and in promoting appropriate IYCF behaviors/practices.
Evidence of the effectiveness and efficiency of severe acute malnutrition (SAM) management programme services was determined by the improvement of cure rates, default rates and M&E reporting over the evaluation period. It was noted that UNICEF and the Director of Food & Nutrition (DFN) had recognized challenges in the area of supply management, responding with the construct of an accountability matrix. The matrix was not provided to evaluators for investigation. Donor respondents underscored ongoing concerns around supply chain management accountability.
Based on focus group discussions (FGDs), the roll-out of IYCF in facilities has done well in changing people’s perspectives. They understand their ability to make a difference in their children’s health. Perceptions such as these in the end tend to lead to more successful and sustaining behavioral changes in health practices. While the effort to improve IYCF practices has improved the knowledge and beliefs of Sierra Leoneans, focus group data show that self-regulation skills and abilities, such as tracking child growth, are poor in the seven districts. Child growth monitoring needs to be emphasized as an element of the nutritional campaign.
Based on the review of the secondary data available (e.g. 2008 and 2013 Demographic and Health Surveys, 2010 Multiple Indicators Cluster Survey, 2014 Sierra Leone Nutrition Survey) severe stunting and wasting were declining in some districts, prior to the 2014 Ebola Virus Disease (EVD) epidemic, which may be attributed to a combination of effects that include the Nutrition Programme. The lack of data from 2015 onwards constitutes a gap in the evaluation of programme impact during and after the Ebola outbreak.
For the promotion of appropriate IYCF Behaviour systems to improve infant and young child feeding practices have been established. CMAM has been integrated into the overall national nutrition system, from policy improvements that support efforts to identify and treat SAM cases, to the training of health workers to implement the protocol. These resulted in increased screening and reduction in cure rates demonstrates the efficacy of these efforts.
Discussions to discern behaviour change during FGDs revealed a need to ramp up IYCF counseling and dispel misunderstandings about EBF; food density, such as the relationship between thick porridge and constipation; food variety, and access to vitamin rich. There was a need to continue efforts to increase EBF for children < 6 months, to improve complementary feeding, and to work with other agencies in a multi-sectoral approach toward improving access to an improved variety of foods.
Following is a summary of findings related to efficiency and effectiveness:
• Although the framework did not detail nutrition screening, per data collected from IPs, close to 2 million (1,935,266) screenings were reported by IPs in six districts (covering December 2013-August 2016).
• GAM and MAM were reduced in the seven districts – Secondary data and the project End Report demonstrate that GAM and MAM rates were declining. However, these data need to be assessed cautiously and further data are needed to prove causality, as other factors are also likely influencing these outcomes.
• Cure rates, non-recovery and default rates improved over the three years between January 2013 and December 2015, suggesting IMAM training was impactful.
• M&E reporting contributes to the efficiency and improvement of nutrition programming and was line with core Sphere Standards, Humanitarian Charter and Minimum Standards in Humanitarian Response. Decentralized monitoring for reporting resulted in improved tracking systems for performance. However, as stated above, data are unreliable, and there was a lack of secondary data from 2015 onward. Although UNICEF and the MoHS have made strides in the development of monitoring systems, there was need to strengthen these further: for instance, the evaluator faced months of missing data from some districts (which were most likely related to EVD epidemic). This could also speak to the capability of decentralized entities to keep track of performance measures in a timely/constant manner during emergencies. The evaluator took these factors into account when analyzed existing data sets combined with primary data collected during the evaluation.
• The desk review, and national and district level interviews also revealed concerns about the ability of the programme to effectively meet the supply demands. UNICEF was one of the key players supporting the Government in supply management, but the sole responsibility should not lie in UNICEF alone. Reports conveyed systemic and structural challenges related to supply chain management. UNICEF worked with the DFN to design a supply chain accountability matrix to address concerns about supply management. It was rolled out at the national level in 2015 and to the districts in 2016. Whether the matrix was used in a timely or regular way was not investigated, and results from this investigation suggest there remain issues around supply chain management at district and facility levels.
• UNICEF Sierra Leone remains flexible within the environment they are working, adapting as needed. UNICEF diverted funds to conduct a facility assessment after the Ebola crisis, which was needed to understand Ebola’s impact on health services. UNICEF experienced challenges with the roll out of the IMAM protocol changes. Training of MSGs and health facility staff was also delayed and re-established post-Ebola. Roll out of the IMAM guidelines was completed in late 2015 and in the first quarter of 2016.
Sustainability and Scale Up
Assess to what extent the programme is sustainable in addressing undernutrition in young children with attention to the less reached, disadvantaged and vulnerable groups
In summary, numerous programme activities demonstrated integration into the national system and support for sustainable policy and interventions. UNICEF, in collaboration with government of Sierra Leone (GoSL), supported the National Food Nutrition Security Plan (NFNSP) implementation policy, and National Technical Committee for Nutrition Surveys. In collaboration with WHO, UNICEF supported the MoHS to develop and distribute IMAM guidelines and train staff in its use. The technical assistance and financial support provided by UNICEF to the MoHS was essential for the development and production of the June 2014 guidelines.
Assess leadership, guidance and technical support at all levels and the adequacy/institutional capacity to respond to the lead role UNICEF is expected to play at the field level for contributing to sustainable and equitable reduction of child undernutrition.
Since the start of the evaluation period in December 2013, UNICEF has had several changes in the Nutrition Programme staff, including changes in leadership and virtually all team members. These shifts in leadership and team members occurred during the Ebola outbreak and post epidemic. Changes involved senior leadership at UNICEF, and all but one staff member that was at UNICEF during interviews had been involved in activities before 2016. Therefore, leadership, guidance, and technical support at all levels have shifted over time. Pre-Ebola, when the programme clearly had some momentum in changing policy and working with government, they had worked with other UN agencies to design appropriate guidelines and strategic plans, roll out the integration of the nutrition programme into the health system, and started to build a multi-sectoral effort to address food security. UNICEF advocacy with the GoSL led to MoHS joining the international Scaling Up Nutrition (SUN) movement. They established the Directorate of Food and Nutrition (DFN) at the MoHS and housed her in an office adjacent to the UNICEF Nutrition Programme.
The current leadership appears to be experienced in addressing hard-to-reach communities, and is building the UNICEF team to further the mission of the UNICEF office in Sierra Leone.
• There is a level of disorganization experienced by the evaluation team when it came to requests for information, especially regarding raw data. Files were disorganized and had half of the districts under evaluation missing data from the main report that were captured in other reports from January 2015, which had not been incorporated into the combined files.
• Given that UNICEF’s greatest financial commitment is the procurement and provision of nutrition supplies, the future should see UNICEF taking a leading role, in partnership with the DFN, to ensure supply management accountability.
• Building local capacity by strengthening local IPs will contribute to sustainability. UNICEF’s support for local NGOs as IPs should ensure they are properly trained to aid in the efforts to improve programme data reporting, supply distribution and provide additional support for community outreach. At the district level, it could improve programme performance data collection and training of healthcare staff, and community educators and screeners.
• Two years into the project, UNICEF expressed concern about the capacity of the SUN secretariat to develop a multi-sectoral plan and coordinate related activities. UNICEF should take a leading role to encourage multi-sectoral approaches to food security, and provide coordination support to move this forward. UNICEF could strengthen efforts with other UN agencies like the WFP and the Food and Agriculture Organization (FAO) to connect communities with opportunities to address food shortages and increase food diversity. This would be in line with UNICEF’s Multi-Sectoral Approach to Nutrition Services (2016).
(i) Need to strengthen Programme Design: Organize the different elements of the nutrition programme in the Log Frame and conceptual framework; consider alignment of key measures with the decentralized M&E framework to enable the tracking of programme elements over time. There is a need for a more structured and nimble reporting system. This could be achieved using a simple database that generates reports on progress and various grant-related activities.
(ii) Use knowledge sharing platforms open to programme partners in nutrition activities to diffuse information to GoSL, district implementers, NGOs, and other stakeholders.
(iii) Explore staff rotation in Sierra Leone as an opportunity to expand OTP, in addition to current efforts. Shifts in staffing cause challenges to UNICEF and result in sites not having OTP trained staff on site. Offer periodic training to address shifts in staff at facilities; improve coordination with MoHS to determine when staff is moving to new facilities and build knowledge capacity at those sites to improve IYCF practices.
(iv) Continue health facility staff training in treatment protocols and proper use and tracking of RUTF and nutrition emergency response; Continue IYCF counseling training and conduct more training on proper IYCF with a focus on hard-to-reach PHU-affiliated CHWs and MSGs
(v) Strengthen Supply Chain Management. Continue to support MoHS toward more engagement and tracking of accountability for the distribution of and accountability for nutrition supplies.
(vi) Strengthen programme monitoring and nutrition information systems to ensure quality data are recorded, and provide support and troubleshooting expertise where needed. Conduct an analysis to understand the reason for reporting gaps and facilities needing support.
(vii) Strengthen ownership and capacity of the SUN Secretariat to establish more robust multi-sectoral strategic meetings that identify and address underlying causes of malnutrition.
(viii) Identify and support champions publicly visible in communities to promote IYCF practices, e.g. nurses and traditional birth attendants. These might also be well-known government officials or local community leaders.
(ix) Build accountability measures that are achievable and feasible to track, while maximizing the existing decentralized monitoring system. Integrate HIV testing and tracking and teen mothers into decentralized monitoring. The indicators appear to be in the decentralized database. However, as of June 2016, there were no reports on HIV+ persons who receive treatment for malnutrition; identify the status and move it forward. Track progress over time to determine how well UNICEF is reaching vulnerable and hard-to-reach communities.
(x) Coordinate with multi-sector partners, ensure establishment of linkages between OTP sites and livelihood programmes to build food security among those most vulnerable, including caregivers who have had malnourished children.
(xi) Enhance MSG strategies to engage hard-to-reach populations. Work with WHO and local agencies to build appropriate messages about EBF and complementary feeding strategies for children affected/infected by HIV/AIDS. Incorporate these strategies into health staff training.
(xii) Conduct in-depth study to determine behaviour change strategies that work in Sierra Leone.
(xiii) Display educational materials at local markets; utilize public media used to increase overall knowledge of risk and severity of malnutrition if feeding recommendations are not followed.
(xiv) Direct resources to further educate and support the MSGs to enhance IYCF counseling; provide appropriate tools for MSGs to meet nutritional programme goals in communities, such as transportation reimbursement.
(xv) Leverage UNICEF strength in advocacy to encourage multi-sectoral activities, engaging the SUN Secretariat to better coordinate the effort to address food security and emergency preparedness. A multi-sectoral approach to emergency response should involve government, other UN agencies such as FAO, WHO, and WFP, international NGOs and local NGOs working in these districts.
• The programmes were found to be relevant to the Sierra Leone context, in that UNICEF identified challenges in the target community and designed systems to address them. However, work remains to be done. There was no evidence of growth monitoring and tracking after the Ebola epidemic. Other than IPs reports of almost two million screenings of children, there was no community level screening documentation outside of UNICEF reports to verify these numbers.
• UNICEF improved the quality of the treatment of SAM through geographical expansion of IMAM services, increased access to SAM treatment, improved cure rates, reduced defaulter and mortality rates. To reach the full burden of SAM children further expansion in geographical coverage is needed.
• UNICEF and the MoHS recognized the issues around poor supply chain management and designed a plan to address it. Together they developed an accountability matrix.
• UNICEF reached tens of thousands of communities through MSGs, CHWs, and facilities. This led to large scale SAM screening for participation in and adhesion to IMAM activities, as was demonstrated by the high number of screenings and a reduction in default rates over the years.
• Staff and health community training was effective, in that implementers understood the strategies that were being used to change behaviour and applied this knowledge to curing children with SAM. Facility staff were trained in the IMAM guidelines and data collection; yet more work is needed to improve the data collection aspect of their work and to continue to improve the quality of IYCF counseling intended to change behaviour.
• Behaviour change communication was strengthened by being mindful of cultural practices, using local language, and using illustrations of foods grown in-country. However, access to foods, drugs, and the ability to identify with MSGs and medical personnel still proves to be challenging.
• The role of UNICEF as nutrition sector lead partner with government and its technical leadership in pushing forward the programme policies and guidelines, demonstrates a strong relationship with MoHS. This has positioned UNICEF to guide the integration of nutrition services into the Package of Basic Health Essential Services, critical to transition from emergency response to country-led and owned services. With the support of UNICEF, the MoHS has led the initiative that integrated nutrition into public health systems in hundreds of communities.
Full report in PDF
PDF files require Acrobat Reader.