Evaluation database

Evaluation report

2009 Global: A Study of UNICEF Engagement in Global Programme Partnerships



Author: UNICEF Evaluation Office and Programme Division

Executive summary

Background

UNICEF engages in a broad range of partnerships and collaborative relationships. This study specifically focuses on UNICEF engagement with global programme partnerships (GPPs), an increasingly popular development cooperation modality for addressing pressing global issues and concerns.

Despite the fact that UNICEF has been engaged in GPPs for decades, GPP mention as a distinct partnership category was introduced for the first time in the midterm review of the medium-term strategic plan (MTSP) for the period 2006 to 2009 and in the new Strategic Framework on Partnerships and Collaborative Relationships (SFPCR), recently approved by the UNICEF Executive Board. Engagement in a growing number of GPPs war-rants that UNICEF better understand not only the opportunities offered by these instruments to advance chil-dren’s rights, but also the engagement challenges resulting from their numbers and diversity.

Purpose and objectives of the study

The lack of baseline information and the difficulty in evaluating the effect of UNICEF engagement in the absence of pre-existing policies and strategies led to the decision to modify the focus of the study and to place greater emphasis on assessing the evaluability of UNICEF engagement in GPPs. An evaluability assessment is a review of a given programme or intervention, at the early stages of or preceding an evaluation, to determine, among other things, whether the programme's objectives are adequately defined and its results verifiable. The study included several components: assessing information requirements and available knowledge in relation to a number of pre-identified evaluation questions; compiling and synthesizing baseline information to facilitate the planning of subsequent process and impact evaluations; developing a conceptual approach to guide and analyse UNICEF engagement; and appraising readiness for evaluation by assessing current engagement practices and procedures using the proposed conceptual model as an analytical template.

Methodology
 
This study comprised a range of methods: baseline definitions and portfolio development; interviews with UNI-CEF staff and external informants; GPP case and country sampling and selection; GPP cases selected for in-depth analysis; country visits to UNICEF regional and country offices; review of country programming documents, annual reports and GPP-related literature; guidance from a UNICEF steering committee; feed-back from a high-level advisory panel; and development of a conceptual model to guide UNICEF engagement in GPPs highlighting the management changes required for effective engagement. A detailed description of methods and advisory mechanisms can be found in Annex 1 of the full report.

Key findings and recommendations

Finding 1: GPPs are expanding the current system of international cooperation from being primarily inter-governmental to being tripartite (state, business and civil society).

Recommendation

1.1 UNICEF should advocate externally for the need to further refine and harmonize definitions, terminologies, typologies and desirability criteria associated with GPPs. These collaborative efforts will eventually lead to international consensus on the total number of GPPs and their individual characteristics, on the pros and cons of different organizational models, and on criteria to be used by decision makers when considering creating new GPPs. All these elements could then form the basic constituents of an international policy framework on GPPs.

Finding 2: GPPs have the potential to advance children’s rights.

Recommendations

2.1 UNICEF should develop, test and promote analytical methods and metrics for assessing the extent to which RBM and principles underpinning the Convention on the Rights of the Child are correctly reflected in the value chain that links partners’ inputs to expected partnership results.

2.2 Because of its involvement in a large number of GPPs, UNICEF can help build bridges across GPPs with the view to promote a whole-child approach through enhanced complementarities and attention to human rights principles.

2.3. When the World Bank/Organisation for Economic Co-operation and Development-Development Assis-tance Committee Sourcebook for Evaluating Global and Regional Partnership Programs (World Bank/OECD-DAC Sourcebook) is revised, UNICEF could lend its support to strengthen the gender equality and human rights components.


Finding 3: UNICEF engagement in GPPs is significant, but its evaluability is constrained by limited availability of historical data and by the lack of a conceptual framework.
Recommendations

3.1 UNICEF needs to develop, test and implement a conceptual framework to guide its engagement in GPPs in a manner consistent with the high-level objectives stated in the new SFPCR. A definition of ‘effective engagement’ and a conceptual framework are proposed in this report.

3.2 To strengthen the evaluability of its engagement, UNICEF needs to establish a central database serving as a repository of critical information on UNICEF engagement in GPPs. The structure of the database should be consistent with the structure of the conceptual framework eventually adopted by UNICEF and should identify indicators, data sources, data management processes, information products, dissemination and use. The design requirements of the realization phase of the new UNICEF Virtual Integrated System of Information (“Vision”)United Nations (UN) Vision Project on Global Public Policy Networks should take into account the specific data needs of this central database. The guidelines on annual reports should be updated to include information on UNICEF engagement in GPPs at country, regional and global levels.

Finding 4: Improving the evaluability of ‘relevance’ as an evaluation criterion requires that engagement decisions be carefully weighed against selectivity criteria, subjected to a vetting process and communicated internally and externally.

Recommendations

4.1 While the idea to engage in a GPP may be promoted within UNICEF by anyone, the ultimate decision to commit the organization should be made by UNICEF senior management and informed by the result of a stan-dardized appraisal process involving relevant in-house expertise. Being clear about what constitutes the authorizing environment is an important condition to ensure unity of purpose across management levels.

4.2 Establishment of a vetting and authorizing environment should be linked to the formulation of criteria and standards for the identification, appraisal and funding of UNICEF engagement in GPPs. To the extent possible, internal guidelines should be harmonized with those of other organizations with similar GPP portfolios.

4.3 UNICEF does not have methods and tools to support trade-off analysis of engagement benefits versus transaction costs. Given the complexity involved in developing such methods, this may best be achieved through collaborative efforts with organizations also involved in GPPs and equally concerned by these issues.

4.4 Engagement decisions should be properly documented and recorded, and should be communicated to external partners and internal stakeholders.

Finding 5: Improving the evaluability of ‘effectiveness’ as an evaluation criteria will require managerial as well as behavioural changes.

Recommendations

5.1 Once engagement decisions have been made by UNICEF senior management, appropriate action should be taken and documented to translate institutional commitments into multi-year engagement strategies and a results framework spanning the MTSP time-frame. These should be appropriately reflected in annual work plans and budgets. When deciding on engagement targets, consideration should be given to the types of results that UNICEF would consider as reflective of an effective engagement.

5.2 UNICEF should undertake a process of alignment between the urgent need to sharpen the strategic focus of UNICEF engagement within GPPs and organizational capacities, internal structures and business processes. This process of alignment should be undertaken as rapidly as possibly, and its outcome should be communicated to internal stakeholders and external partners.

5.3 UNICEF should review the process of appointment of UNICEF representatives serving on the boards and steering committees of GPPs. UNICEF representatives should have terms of reference and should declare potential or real conflicts of interest. They should be correctly briefed and identify follow-up actions.

5.4 UNICEF needs to produce specific guidance on engagement in GPPs. Relevant manuals, most importantly the Programme and Policy Manual, should be updated and complemented with issue-specific guidance notes. These guidelines should be developed with the participation of internal stakeholders and harmonized with the guidelines, when they exist, of partner organizations. A conflict of interest policy addressing UNICEF engagement in GPPs should also be developed.

5.5 The UNICEF Risk Reference Guide should be updated to include the management of risks associated with UNICEF engagement in GPPs.

5.6 Portfolio review procedures should be developed and implemented. They should favour proactivity over reactivity and allow assessments of performance at the individual partnership level and the aggregate level.

Finding 6: Evaluating institutional engagement in GPPs is constrained by the lack of internationally agreed criteria and methods.

Recommendations

6.1 UNICEF should consider collaborating with other interested organizations, including the World Bank and OECD-DAC, on the development of a standard methodology for evaluating institutional engagement in GPPs. This could be carried out as part of the revision of the World Bank/OECD-DAC Sourcebook or as a separate but harmonized product.

6.2 When designing evaluations of the GPPs in which it participates, UNICEF should continue to promote the use of the indicative principles and standards contained in the World Bank/OECD-DAC Sourcebook and ensure that the evaluation design includes an assessment of the performance of institutional partners.

6.3 UNICEF should review more systematically the evaluation results of the GPPs in which it participates, with the view to adjusting its own engagement strategies accordingly. Any decision to modify the nature of its en-gagement as a result of such internal reviews should be communicated to the partners and to the chair of the board or steering committee (in the case of formal partnerships). These internal reviews should be documented and records should be kept in a central file.

Finding 7: Strengthening the evidence base on GPPs and institutional engagement will require a collaborative effort.

Recommendation

7.1. In order to further strengthen the evidence base on GPPs as an instrument of international cooperation, and take advantage of the growing number of independent reviews and evaluations conducted using inter-nationally agreed norms and standards, UNICEF should consider commissioning a meta-analysis of pub-lished evaluations. This could be best done through a collaborative effort involving organizations sharing similar portfolios.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2009

Region:
Global

Type:
Study

PIDB:

Follow Up:

Language:
English

Sequence Number:

 

New enhanced search