We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2017 ESARO- Evaluation of ESAR Institutional Strengthening Support Initiative on Decentralized Programme Monitoring and Response



Author: Stephen Ladek, International Solutions Group

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

 

Background

 

In 2014, UNICEF headquarters, in collaboration with its East and Southern Africa Regional Office
(ESARO), took the decision to address certain issues related to the health of women and children by
implementing a regional institutional strengthening support initiative to improve the Programme
Monitoring and Response (PMR) capacity of the centralized government decision-making bodies and
communities in four countries (Kenya, Swaziland, Uganda and Zimbabwe).

 

The ESAR PMR was designed as a learning initiative with the following goals:

  • Improve service provision and program performance during implementation through improved
    monitoring of intermediate results, establishment of citizen feedback loops, and the use of
    monitoring and feedback data to adjust strategies as needed in a timely manner; and
  • Identify and promote those promising practices of programme monitoring in response, tested and
    proven by early adopter districts, through the development of national scale-up plans within
    participating countries and the establishment of knowledge exchange platforms.
    Four countries were selected for participation in the PMR Initiative (Malawi, Swaziland, Uganda and
    Zimbabwe). Each implementing country customized and localized the actual implementation of the
    Initiative.

The hypothesis underlying the PMR initiative is that, if decentralized management and/or front line
health worker decisions and responses are timelier and evidence-based, and if citizen feedback loops and social accountability structures are established, then over time improvements will be seen in:

• Sector service provision/program performance (i.e. coverage and quality of services);
• Communities’ commitment to invest in their own health and well-being; and
• The health and well-being of vulnerable and previously underserved children and women,
thereby improving equity of health outcomes.


Purpose/Objective

  • The evaluation was carried out between 27 MAR 17 and 31 MAY 17. Field work was completed between
    15 April 17 and 10 May 17.
    The purpose of the evaluation is to:
    • Support the further national scale-up of documented, effective PMR institutional strengthening
    interventions in the four participating countries;
    • Guide anticipated replication in other countries in which UNICEF operates;
    • Add to the regional and global knowledge base by generating evidence on processes and outcomes
    related to strengthening decentralized PMR systems in the health and nutrition sectors; and
    • Provide information on the extent to which the work undertaken in the four countries is
    applicable to other sectors beyond health and nutrition.
  • The objectives of the evaluation are to:
    • Assess achievement towards the programme’s expected results  as per the initiative’s Results Framework;
    • Assess the extent to which the programme activities have changed how (more) real time data is
    collected, analysed, packaged, disseminated, and used;
    • Assess whether, and how, mutual and ongoing learning on PMR is taking place across target
    decentralized government units and countries, as well as within target UNICEF offices;
    • Understand and document the range of contextual/environmental factors that served as
    enablers/disablers to the results achieved, including programme design, implementation,
    management and coordination processes;
    • Assess the programme’s relevance, effectiveness, sustainability, scalability, and replicability to
    determine if the package of support interventions and tools employed by UNICEF’s four
    participating COs can serve as applicable models for other UNICEF countries and stakeholders in
    similar context and sectors, and other social sectors;
    • Determine if the four UNICEF COs can serve as ‘centres of excellence’ within UNICEF globally on
    decentralized PMR or specific aspects of decentralized PMR;
    • Document best practices and lessons learned related

Methodology

Evaluation criteria
The evaluation was carried out using a limited selection of standard OECD/DAC Criteria that included:
relevance, effectiveness, and sustainability. Efficiency and Impact were not included, as agreed in
consultation with UNICEF, because of the intended foci of the evaluation and overall timing of the
Initiative.
 The following were evaluated: RELEVANCE;  EFFECTIVENESS / ACHIEVEMENT OF INTENDED RESULTS;and SUSTAINABILITY AND SCALABILITY; ;

Findings/Conclusion

1: The PMR Initiative has its genesis in an understanding of the continued evolution of ICT4D
innovations in ESAR and how these innovations have contributed to building a culture of management
using data in the health sector. Because of this, the PMR Initiative has been a response that can be
localized and contextualized to specific country needs, priorities, and capacity.
2: The PMR Initiative has established or enhanced platforms that produce reliable, quality and
more real time data in all four countries. However, the emphasis of the initiative has been on data use.
3: The evaluation cannot determine if UNICEF has a comparative advantage in the production
of technology-based solutions. However, UNICEF appears to currently enjoy a significant competitive
advantage, but one that needs to be carefully articulated and defended.
4: The PMR Initiative has not resulted in a process for incorporating citizen feedback that rivals
or eliminates traditional, or established, community and government structures.
5: The evaluation cannot determine if UNICEF has a comparative advantage in community
engagement. However, UNICEF’s citizen engagement tools offer an opportunity for competitive
advantage, but one that needs to be established and validated.
6: A significant achievement of the PMR Initiative is its catalytic contribution to a “management
using data” culture in the four implementation countries.
7: By emphasizing data use, even with incomplete data sets, the PMR Initiative has contributed
to a virtuous cycle of increased demand for more data and a desire for improved data quality.
8: The PMR Initiative has highlighted the need to ensure resource mobilization for identified
health sector issues as paramount.
9: UNICEF actively pursued the exchange of knowledge generated by the PMR Initiative
through a variety of platforms and opportunities, with mixed results

Recommendations

1: Continue to invest in the “Management Using Data” culture in ESAR by
leveraging UNICEF’s unique relationship with government partners for long-term investment
plans and replicating gains within healthcare and in other sectors.
2: Bolster UNICEF’s competitive advantage as a manager of enabling ICT
solutions by investing internally in skilled information managers.
3: Invest in establishing UNICEF’s comparative advantage to make the case
that ICT investments are the best use of donor funds for achieving better health outcomes for
children.
4: Continue to invest in collecting and responding to citizen voice by iterating
current solutions and seeking new solutions from a larger community of practice.
5: Continue to evolve knowledge exchange at UNICEF through changes in
organizational structure, professional culture and leadership buy in
6: Seek to shift UNICEF staff and partner mindsets about transferability of
ICT enabled programming, and knowledge exchange, from “plug and play” to “customization.”
7: Use ICT enabled programming as a bridge for multisectoral programming
and scaling vertically.
8: Capitalize on opportunities exposed by the success of the PMR Initiative
in closing the feedback loop to address the “no available resource” challenge and ensuring data
collection is sustainable.

Lessons Learned


1. Leveraging existing national programmes, processes and systems (i.e. HIMS) improves buy in, accelerates development and adoption and enhances sustainability
2. Software and technology are living processes.  As such technology should be adopted with the understanding that it is a long-term investment that will require constant care and feeding.
3. culture of communication that is technology focused remains a significant hurdle to overcome and programming support for target populations should reflect this.
4. PMR demonstrated that better data and better management analysis of that data leads
to identification of problems at the local level. However, this process must also include mechanisms for
closing the feedback loop with articulated solutions from decision makers.
5. Future initiatives that adopt this type of support should ensure the management review/analysis process of data always results in clear, reasonable action items that are ultimately “owned” by one individual..
6. The bigger hurdle for ICT4D initiatives is user adoption and a professional culture that expect technology to be at the centre of individual skills, roles and practices.

 

.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2017

Country:
ESARO

Region:
ESAR

Type:
Evaluation

Theme:
Program Excellence (Cross-Cutting)

Language:
English

Sequence Number:
2017/010

New enhanced search