2013 Fiji (Pacific Islands): Evaluation of the Response Fund Supported Activities of the UNICEF Pacific HIV & AIDS Programme
Author: Steve Powell
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report."
Many indicators point to the likelihood of increasing HIV prevalence in the Pacific Islands in the future. Incidence has been growing slowly in the PICs taken together in the last decade. Against this background, UNICEF in the Pacific through its HIV & AIDS Programme “aims to reduce the vulnerability to and impact of HIV and AIDS … with a special focus on children and women, through strengthened integration of maternal and child health services and community-based initiatives".
One of the main regional responses to HIV is The Pacific Island HIV and STI Response Fund 2009-2013, a multi-donor pooled funding mechanism to achieve the overall goal of the Pacific Regional Strategic Implementation Plan for HIV and other Sexually Transmitted Infections Implementation Plan 2009-13 (PRSIP II): to reduce the spread and impact of HIV and other STIs while supporting people infected and affected by HIV in Pacific communities.
In 2009, UNICEF Pacific HIV & AIDS Programme applied for funding from Stream V of the Pacific Response Fund (PRF). UNICEF Pacific received AUD 450,000. The funding was approved for the implementation period of 1st July 2009-30 June 2012. In February 2012, UNICEF requested and was approved for a grant extension to 31st December 2012. The goal of the evaluated PRF-funded Activities within UNICEF's broader Programme was: to support national and regional efforts in the Pacific Island Countries to prevent the spread and minimize the impact of HIV and other STIs on individuals, families and communities.
The purpose of this evaluation is to assess progress made by UNICEF in relation to the key response fund PRF-supported results expected by the end of 2012.
The key objectives of the evaluation are to determine the relevance, effectiveness, efficiency and sustainability (the Evaluation Criteria ) of the Pacific Response Fund (PRF) Stream V- supported activities (“the Activities”) implemented by the UNICEF Pacific HIV & AIDS Programme (“the Programme”) in line with the OECD DAC evaluation criteria.
As PRF-supported activities were closely linked to the rest of the UNICEF Pacific HIV & AIDS Programme, they are assessed here in that broader context.
The countries covered in the evaluation were Kiribati, Vanuatu, Solomon Islands and Fiji. Altogether 42 separate group and individual interviews were conducted by the consultant during May and June 2013.
The countries covered in the evaluation were Kiribati, Vanuatu, Solomon Islands and Fiji. It was not possible to visit all countries because of time restrictions, so Vanuatu was not visited in person, but some telephone interviews with respondents in the country were carried out.
The stakeholders interviewed are listed in the appendix. Altogether 42 separate group and individual interviews were conducted. In many cases triangulation could be conducted directly in the interviews using “circular questioning” techniques.
Fieldwork also allowed the evaluator to make both targeted observations (to gather data to evaluate the current evaluation questions and sub-questions) and ad-hoc observations (to help develop and refine the evaluation sub-questions). Fieldwork observations included, but were not limited to:
• Availability and quality of services and products (planned and additional) at field sites
• Social and economic circumstances of target and at-risk populations
• Nature and quality of cooperation and communication between the UNICEF Pacific HIV & AIDS Programme Team and their partners, and between the latter and other stakeholders
A literature review and additional analysis used the sources listed in the 5.2.Appendix, some of which were made available at the start of the Evaluation, some of which were later requested by the consultant. Triangulation and data processing:
The triangulation and data processing procedure is described in more detail in Section 5.4 . Briefly, data from interview, observation and documentation was combined in order to reach answers to the evaluation questions; more detailed evaluation sub questions were developed continuously as new hypotheses arose during the evaluation and added to the main list of evaluation questions.
Findings and Conclusions:
Due to word limitation, only 2 of the 4 findings and 4 of the 13 conclusions are stated below. See chapter 2 for Evaluation Findings and chapter 3 for Conclusions, in the attached.
Finding 1: UNICEF has a distinct comparative advantage of expertise in relation to young people and mothers and young children especially when working with governments on standards, data and medical issues. However UNICEF can sometimes be cumbersome when working or supporting work directly with beneficiaries,
Finding 2: Overall, the activities, outputs and programme model are relevant to reaching the planned outcomes. UNICEF support and activities depend to an extent on strong MoHs and/or a competent personal UNICEF presence in-country. HIV activities are most relevant where they are integrated into other services at health service level and where they are well integrated into other programmes and with other agencies at programme level.
Conclusion 1: The stronger focus on data, policies and guidelines, alongside PPTCT, has worked well as a more appropriate and sustainable role for UNICEF. More country-specific work needs to be done to ensure basic record keeping, reporting and analysis becomes a habit before HIV data initiatives are really sustainable.
Conclusion 2: UNICEF contribution has been most effective in the form of strategic support rather than in directly or indirectly funding services to beneficiaries.
Conclusion 7: While the programme and its priorities have the broad support of the national institutions, HIV has been loosing rather than gaining prioritization over the last few years. Motivation for HIV work is highest when it overlaps with other higher priority areas of work.
Conclusion 9: UNICEF often succeeded in adapting it's approach to specific countries and partners but even more country and partner specific approach would better suit stakeholder needs.
Due to the limitation on words, only key recommendations of the 5 themes are presented below:
a) Strategic information and planning
UNICEF should continue to provide capacity building and systems for PPTCT and strategic data collection, increasing efforts to ensure a step-by-step, bottom-up approach is taken which is adapted to the different needs and motivation of individual partners. The stronger focus on data, policies and guidelines, alongside PPTCT should be further expanded upon.
b) Prevention services to youth and pregnant women
Particularly on youth services, UNICEF should focus more on providing expertise for guidelines, training and communication materials and peer education rather than for direct implementation such as supporting “model” or “pilot” services to youth.
In working with Ministries and other partners, UNICEF should find ways to keep up the urgency of HIV prevention messages while avoiding fear-based messages, perhaps by directly addressing the perception that international agencies have been “crying wolf”.
d) Programme model
UNICEF programming should continue to further adapt its approach to different countries and query one-size-fits-all plans for activities, outputs and outcomes.
e) Capacity development of national partners
UNICEF could consider conducting Organizational Development with Ministries or even other partners like provincial-level institutions, just as IFRC does with its national partners.
f) Planning and reporting
Keep reporting frameworks, formats and targets within GF as few and as simple as possible while ensuring a minimum of effective performance monitoring: a very few key indicators which are so important and easy to measure that they are frequently assessed, reported, and responded to.
The below lessons learnt have been extracted from the end-of-the project report submitted to the donor as the evaluation report doesn`t provide specific information on lessons learnt.
1) As a strong regional partner, for the past several years UNICEF has advocated at the regional level for countries to have sound strategic information to inform their national response to HIV and STIs. Only recently have both governments and regional partners recognized the need for updated behavioural and serological data on HIV and STIs. However, strategic data is critical for influencing resource mobilization and, noting that two critical funding mechanisms are ending in June 2013, it would have been important to have conducted these studies earlier.
2) UNICEF Pacific relies on systems within each country to implement planned activities, but capacities within HIV units are limited. This needs to be recognized and addressed. Therefore, one of the lessons learnt during the course of implementing activities was that it is important to focus on capacity-development activities for partners.
3) Capacities and systems to manage funds vary between countries, regional agencies, civil society organizations and government partners. Grant management should be designed as per situation and not based on a one-size-fits-all approach.
Full report in PDF
PDF files require Acrobat Reader.