2002 PIC: Fast Track Evaluation of the Pacific Programme of Cooperation in the Pacific Island Countries 1997 - 2001
Author: UNICEF NYHQ
In January 2002, the Executive Board recommended to undertake a fast track evaluation of the 1997-2001 Pacific Programme covering 13 Pacific Island Countries, which would inform the Country Programme Recommendation to be presented during the Session in September 2002. The Programme was designed to contribute to the realization to Goals of the World Summit for Children of 1990 and Pacific Goals for Children of 1993.
The Programme was implemented through four regional programmes (child and youth advocacy and planning, health and nutrition, early child and primary education, and monitoring and evaluation) as well as through eight Integrated Area Based Programmes at country level (Federated States of Micronesia, Fiji, Kiribati, Marshall Islands, Samoa, Solomon Islands, Tuvalu, Vanuatu) and one multi-country project (Cook Islands, Niue, Tonga, Tokelau, Palau). The total approved budget amounted to US$ 7,000,000 in General (Regular) Resources and US$ 14,000,000 in Supplementary (Other) Resources.
Purpose / Objective
It is believed that an evaluation would strengthen the rationale of the new Programme. The information that such an evaluation would provide would be useful to make explicit conditions that affect UNICEF's efficiency and effectiveness, including on issues related to comparative advantage and complementarities with the programmes of partner governments and other agencies.
The objectives of the evaluation of the Pacific Islands Country Programme were six fold:
- to assess the role and relevance of the Programme of Cooperation as to the situation of children and women in the Pacific Islands;
- to assess the realisation of the Country Programme objectives as spelled out in the Master Plan of Operations against the background of World Summit for Children (WSC) and National Plans for Action (NPA) goals;
- to assess effectiveness, efficiency and impact of supported projects and programmes and analyse to what extent activities and results are sustainable and /or replicable;
- to assess the comparative advantage and complementarities with the programmes of other partners, governments and agencies;
- to assess the rationale for the new programme process; and
- to assess the risks and level of support for the current and proposed programmes.
An extensive desk review of all relevant documents related to programming, reporting, monitoring and evaluation was conducted. Meetings were held with representatives from Pacific Island Countries (PIC), Australia and New Zealand accredited to the United Nations in New York to discuss scope and methodology as well as preliminary findings of the evaluation. A series of interviews with governmental and non-governmental partners in Fiji and Vanuatu as well as with the Secretariat of the Pacific Community, The Forum Secretariat, UN organisations (UNDP, UNFPA, WHO, UNIFEM) and representatives of donor countries and organisations (Australia, New Zealand, Japan, USA, France, Asian Development Bank, European Commission) were held. A survey was conducted among programme staff of the UNICEF Office in Suva and in the field. Field visits were made to UNICEF supported projects in Vanuatu.
Key Findings and Conclusions
In spite of limited resources and major problems related to transport and communication, attempts were made to serve all PIC. The choice of priority countries (Kiribati, Solomon Islands, Vanuatu as well as Federated States of Micronesia and Marshall Islands) was justified with reference to a number of very general criteria: their recognised LDC status (Kiribati, Solomon Islands, Vanuatu) and their poor social indicators (Federated States of Micronesia and Marshall Islands). The very small population of some of the countries justified a more limited presence in spite of their having met other criteria. However, the Programme was still spread rather thin limiting chances to achieve significant results.
The goals and objectives provided a broad framework and rationale for the choice of individual projects and programmes. The choice of programmes could also be related to specific issues of non-fulfilment of children's rights as detected in the country-specific Situation Analyses. The Programme was thus relevant in the sense that it addressed major issues identified in the Situation Analyses.
The design of the Pacific Programme, as reflected in the Country Programme Recommendation (CPR) and Master Plan of Operations (MPO), shows some conceptual weaknesses, which are admittedly not uncommon in UNICEF's programming process and therefore not unique to the Pacific Programme. The programme objectives are not SMART (specific, measurable, achievable, relevant, time-bound) and constitute an insufficient basis for performance monitoring and evaluation. There is also a general lack of indicators. There is neither a clear indication of results to be achieved nor a distinction between different levels of outputs and attributable outcomes or even impact. The confusion related to different levels of results also entails a rather undifferentiated approach to partnerships and alliances.
Above-mentioned weaknesses in the design of projects and programmes make it difficult to adequately assess programme effectiveness, i.e. determine to what extent and in what ways objectives were achieved and results obtained. Annual Reports and new programming documents make mention of the following major achievements of the Pacific Programme 1997-2001. The listing is illustrative rather than exhaustive. Examples can be grouped along specific objectives mentioned in the CPR and the MPO:
- Identify and advocate actions on priority issues affecting children's well-being ...: National Children's Coordinating Committees were strengthened in their role to advocate children's issues to be emphasised in national development plans. As a result, legislation has been reviewed in some cases and reviews have been conducted leading to new initiatives for child justice and assistance for children with disabilities.
- Strengthen ... the monitoring of survival, development and protection of children ...: Situation Analyses have been completed in five countries and four more are in final stages of completion. A regional Situational Analysis will be completed shortly. Studies and evaluations in the areas of nutrition, youth, and early childhood development have contributed to a better understanding of children and to an assessment of progress notably in disabled children, hepatitis B immunisation and Vitamin A supplementation.
- Establish and strengthen decentralisation of programmes at island level and community-based participation ...: Immunisation rates have improved and hepatitis B vaccines have been incorporated in all vaccination programmes. UNICEF has supplied vaccines through the Vaccine Independence Initiative (VII), assessed cold chains, procured supplies and equipment and trained health staff. UNICEF has also contributed to the Micro-Nutrient Initiative by providing Vitamin A capsules and by support to training, monitoring and promotional activities.
- Continue to support development of innovative approaches to child and youth survival, development, protection and participation: The Baby-Friendly Hospital Initiative has been introduced in all PIC, resulting in documented improvements in breastfeeding rates. Supporting studies were conducted and communication and training materials were developed. In coordination with WHO, the Integrated Management of Childhood Illness (IMCI) was introduced in Vanuatu, Solomon Islands, Kiribati and Fiji to address the five major causes of childhood mortality and morbidity. Provincial programmes are starting in three countries.
- Sustain and increase collaboration and strengthen partnerships between UN organisations, regional organisations and institutions, and non-governmental organisations working on high-priority children's issues': Apart from cooperation on initiatives already mentioned, partnerships with other UN agencies have been strengthened in preparation of CCA / UNDAF. There has also been increased cooperation with local and regional media involving training of journalists and support to productions (e.g. with the Pacific International Broadcasting Association). Participation of children was enhanced, as they were encouraged to express themselves in public fora, in the printed press and on radio and television. In Fiji, the first-ever Youth Parliament was held.
Unfortunately, no explicit analysis of the key constraints and challenges that affected successful implementation was conducted at the outset at either the Pacific Programme as a whole or at the individual project level. There was no overt strategy for managing these key risks that could be monitored by management. However, constraints and challenges to the successful implementation of individual programmes were to some extent implicitly dealt with in the programming practice through a combination of the following processes: a) the analysis of lessons learned from earlier programme interventions; b) the evolution of programmes through long term commitments and harnessing of broader international experience of well considered and tested strategies, e.g., the fortification of flour with micro-nutrients to address anaemia; c) the emphasis on development and use of simple, low cost and hence low risk technologies to address problems, e.g., the emphasis on enhancing existing good nutrition practices such as encouraging exclusive breastfeeding as a means of improving nutritional intake for babies; and d) the process of in-depth consultation with stakeholders in developing the programme helps to focus UNICEF on PIC priority issues while fostering local ownership and commitment.
The system constitutes a tremendous improvement in terms of results based management. The outputs seem appropriate products or steps towards achieving the objectives and generally the indicators being monitored seem measurable and appropriate. However, at present, not all UNICEF programme managers consistently complete the IMEP report. The M&E system is hence not comprehensive.
From discussions held with a range of donors and national partners there would appear to be a consensus that UNICEF had a number of actual and potential comparative advantages over other organisations. Examples include strategic Situation Analyses, systematic advocacy campaigns, innovative pilot activities to advance the realisation of children's rights, and outlines of potential interventions and marketing of such proposals. These external perceptions are largely consistent with those held by UNICEF Suva staff as expressed at the strategic planning retreat in February 2001.
The Programme of Cooperation arrangement allowed for considerable flexibility in terms of financial management, as under-spending in one country could be compensated by the allocation of additional resources elsewhere. The disadvantage of the system was a certain lack of transparency for partners in different countries, as decision-making was essentially centralised in the UNICEF Office in Suva. This lead to the situation that national partners owned the Programme to a much lesser extent than is customary for UNICEF supported Country Programmes. Involvement of regional bodies owned by the PIC (e.g. the Secretariat of the Pacific Community and the Forum Secretariat) was also insignificant. Lack of ownership by national and regional partners is one of the most striking characteristics of the past Pacific Programme.
During the programme period, OR mobilisation was considerably less than the target. It amounted to US$ 2,342,631, hence a little less than 17 percent than the target (ceiling). This state of affairs reflects unrealistic expectations when defining the ceiling in the CPR and insufficiencies in the resource mobilisation process. In programming documents, there is no evidence that a resource mobilisation strategy was developed. It should be noted that Country Office Management cites several reasons for low OR mobilisation: the increase of Regular Resources in the middle of the programming cycle, the introduction of a new accounting system (PROMS), which slowed down implementation and problems of liquidation with partners holding up the release of additional funds. Lack of sufficient professional staff was also quoted as a contributing factor. The average rates of expenditure was almost 68 percent for RR and almost 72 percent for OR. These averages do not compare too unfavourably with rates of spending elsewhere in the region and in the world.
During the period of the Programme, the UNICEF programming process took as a starting point the assessment of the situation of children and women, which was to be updated regularly during the programme implementation period. Such Situation Analyses served as a reference for UNICEF programming and were often also used by development partners as a reference for their proper assessments and programmes. UNICEF's role in assisting with the identification of appropriate indicators, of guiding national data collection and assisting in the use of the findings for advocacy and programming is critical. It is an area that has not been of high priority focus in UNICEF's programme and needs to be strengthened. Each country needs to be able to do its own situation analysis for long term sustainability of this process. In the case of small countries, assistance with data collection and analysis may always be required. This could be achieved through sharing of resources in a regional arrangement (e.g. with the Secretariat of the Pacific Community).
The previous Programme has also generated a strong lesson, which should be taken into consideration for the formulation of the new Programme: effective programme implementation requires adequate field presence of UNICEF in the countries. This would have to involve assignment of at least one full staff member of UNICEF to each of the priority countries, whose competency profile should be managerial and whose authority would involve day-to-day management of projects and programmes.
There are strong indications that representation of UNICEF through volunteers (United Nations and bilateral) under the previous Programme was not effective in many ways. Even when they had the necessary professional background, they lacked the authority and recognition of professional UNICEF staff. Even small decisions required communication with the Suva Office of UNICEF, which in practice proved cumbersome and ineffective. A stronger field presence would also increase chances to reach less accessible islands within the countries concerned.
In practical terms, for UNICEF, community-based initiatives will prove to offer greatest chances of sustainability (e.g. the kindergarten project in Vanuatu). In some cases, other development programmes may promote certain innovations initiated by UNICEF on a larger scale, e.g. the VIP latrines in Vanuatu that are now part of a broader rural development programme funded by New Zealand. UNICEF needs to develop a strategy for handing-over or encouraging the uptake of its technology models by other donors rather than continuing to support technology models for extended periods of time at the expense of introducing new innovations. The quest for sustainability of activities initiated by UNICEF will thus require a creative cooperation not only with communities and governments, but also with larger bilateral donors and other international partners who are likely to maintain a permanent presence in the region.
At the national level: strengthen ownership of programmes by governments and NGOs as well as National Children's Councils, ensure adequate UNICEF field presence in priority countries and build more strategic partnerships and alliances;
At the regional level: cooperate more closely with regional institutions, e.g. the Secretariat of the Pacific Community, for the monitoring of the evolving situation of children, for human resource development and human capacity supplementation especially in the smaller and less developed countries and for improving communication (websites, radio etc.);
At the programme level: strengthen the design of the Programme as a whole as well as individual programmes and activities (logical model, IMEP), better articulate interrelationship between operational activities and advocacy, more strategically plan for partnerships and alliances based on a realistic understanding of UNICEF's comparative advantage and undertake better risk assessment and management.
Full report in PDF
PDF files require Acrobat Reader.