2015 Albania: Evaluation of the Government of Albania and United Nations Programme of Cooperation (PoC) 2012-2016
Author: Alexander MacKenzie, Sabina Ymeri
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.
The goal of the Government of Albania (GoA) and United Nations (UN) Programme of Cooperation (PoC), 2012-2016, is to ‘promote sustainable and equitable development, social inclusion, and the adherence to international norms and fulfilment of international obligations, in support of the integration of Albania into the EU’ . At the request of the Government, the PoC aimed to expand and deepen the Delivering-as-One approach of the United Nations with coordinated, harmonized arrangements for joint, coordination, planning, monitoring, and progress reporting, together with country partners. Finalized in October 2011 between the Government of Albania and 20 participating UN agencies, the PoC for 2012-2016 contains a results framework with 11 outcomes and 41 outputs, an indicative budget of USD $132 million, and a joint resource mobilization strategy.
The evaluation of the PoC is a strategic, programmatic evaluation. It meets UN’s requirement for evaluation of UN development assistance frameworks once in their cycle. In 2015, PoC is in its penultimate year of implementation. The evaluation was scheduled for the start of 2015 in order to provide usable recommendations and lessons for the process to prepare the new PoC, which will begin from the last quarter of 2015. This means that the evaluation has a summative element focused on results achieved for the period 2012-2014 and a formative element that provides recommendations and lessons that are relevant for the remainder of the current PoC and for the preparation of the next.
The users of the evaluation findings are: the GoA, UN Agencies, international donors, and other partners in civil society and the private sector. The results of the evaluation will also be of interest to stakeholders that follow closely the evolution of the DaO strategy and one programmes at country level, including the United Nations Development Group, other pilot countries, and the Economic and Social Council of the UN.
The purpose of the evaluation has been to:
- Make conclusions about the progress of PoC implementation and demonstrate the UN’s accountability to the GoA for its contributions to national development priorities; and
- Identify important recommendations and lessons that can strengthen ongoing implementation of the PoC to the end of 2016 and the design of the next programme for the period 2017-2021.
The terms of reference compiled for the evaluation assignment provide two objectives:
- To assess the contribution of UN system to national development priorities through the actual results of the Programme of Cooperation (PoC). It will assess the effectiveness and efficiency by which PoC outcomes are being achieved, their sustainability and relevance to national priorities and goals.
- To assess the process of UN system support, through the PoC, to Albania’s national priorities and goals. It will assess the processes, mechanisms and procedures in the light of effective and efficient contribution to the national development efforts and capacity building.
The methodology applied contribution analysis to understand and assess how the outputs achieved through programme cooperation have contributed to expected outcomes and influenced progress towards the achievement of national development priorities and targets. The evaluation was conducted in a participatory manner, ensuring the involvement of key stakeholders in all phases of the evaluation.
To meet the objectives of the evaluation, the evaluation used the following criteria:
- Relevance. The extent to which the PoC expected PoC results and strategies were consistent with the vision, strategic priorities, and policies of the National Strategy for Development and Integration, 2007-2013 and the commitments related to Albania’s ratified treaties and international agreements.
- Effectiveness. The extent to which planned PoC results (mainly outputs) were achieved and how, if at all, they made a contribution to the expected outcomes and progress towards national development priorities and targets.
- Efficiency. The extent to which PoC results were achieved in an economic manner (i.e. avoiding waste and duplication) and with minimum transaction costs.
- Sustainability. The extent to which the PoC results are likely to continue beyond 2016 and how the complementarities and collaboration fostered by the PoC contributed.
Given that the PoC is still in its penultimate year of implementation, the assessment of impact has not been targeted.
The evaluation employed the following data collection methods:
- Document review focusing on all major the plans and reports related to the PoC and NSDI
- Semi-structured interviews with individual and small groups of key stakeholders from GoA, UN, civil society, and development partners
- Focus group discussions involving small groups and sub-groups of the stakeholders.
Multiple questions with different groups of respondents were used in order to triangulate findings and provide credible conclusions.
Findings and Conclusions:
- The evaluation concludes that the PoC was highly relevant to the priorities of the NSDI I, including those for EU accession. The PoC was verifiably effective in terms of the achievement of most planned outputs, and their plausible contribution to the expected outcomes, and to the progress towards country priorities.
- In terms of sustainability, several important PoC results and related strategies are likely to continue beyond 2016 including legal/policy advocacy agendas; mechanisms for addressing domestic violence; juveniles justice; territorial reform; public ‘ownership’ of vulnerability agenda; pre-school education; employment promotion programmes; integration for PwD; social protection, care, and inclusion.
- PoC was able to achieve results in an economic manner and with manageable transaction costs.
- PoC results and strategies were driven or at least informed by the standards and principles of Albania’s ratified human rights treaties.
- The programming process and results for gender equality from 2012-2014 represented a significant achievement.
- Many of the environmental related achievements are sectoral rather than relating to broader environmental mainstreaming.
- Regardless of efforts in developing new institutional capacities, there is no evidence that formal, structured capacity assessments were conducted as part of the PoC preparation, implementation, or for MTR.
- The revised PoC management structure and joint biennial work plans for the period 2015-2016 comply with the quality and standard operating procedures for DaO, targeting the gaps in the initial PoC results framework.
- The preparation of the PoC did not include a formal, structured process to review and consider the UNCT’s comparative advantages vis-à-vis those of other major partners working in Albania.
- GoA-UN PoC was, and continues to be, an effective vehicle for coordination and partnership, with a high degree of knowledge and ownership of cooperation results amongst partners.
- Analyse the existing and planned sector programmes to identify policy development and implementation areas best fitting with UN programme support.
- Liaise with GoA sector leads and EU partners to understand the full scope of the shift from sector working groups to Integrated Planning and Management Groups (IPMGs) and seek clarification on the capacity support needs of IPMGs.
- Based on above, define a role and expected contribution of the GoA-UN PoC to the overall sector programme approach.
- Engage with the GoA to discuss ways for greater cost-sharing by the GoA for ongoing programmatic efforts. Revise the outcome budgets for the period 2015-2016 to arrive at more realistic funding gaps.
- Document and disseminate up to 3 compelling examples of how the coherence fund (CF) enabled rapid and effective programmatic action aligned with national priorities.
- Consult further with all Joint Executive Committee members for PoC and donor partners to confirm their perceived value of the CF and their willingness to promote and use it as an important channel for UN development cooperation. Develop a concrete advocacy and communication plan for promotion of the CF.
- Given the complex resource mobilisation (RM) environment and the introduction of sector budget support by the EU and GoA, the UNCT should review and update (as needed) the Joint Resource Mobilisation Strategy and ensure more frequent inter-agency communication around RM efforts.
- Make a priority to: a) Analyse and develop a business case for additional Long Term Agreements (LTAs) for common categories of goods and services, such as vehicle management and maintenance; b) Develop and implement the common services plan for the one UN house.
- Provide updated data on (1) the total value of purchase orders raised against common LTAs, (2) the estimated cost savings from volume procurement for all 9 LTAs, (3) and the estimated transaction costs avoided for all LTAs since their inception.
A successful one-programme approach should not necessarily include joint implementation by two or more UN Agencies and partners, entailing added costs of developing joint programme documents or negotiation for fund management modalities.
The gender theme group, being a well-led and resourced inter-agency group, played a major role to support gender mainstreaming in the PoC and gender equality results were a significant achievement.
The positive role of the UN in helping the GoA to move the territorial reform agenda forward offers a strong example of how the UN’s neutrality and impartiality can be a major advantage when engaging in very complex and sensitive political challenges.
GoA-UN PoC focused strongly on legal and policy development based on international standards and on evidence. These are also areas where beneficiary capacity is relatively weaker and resources are scarcer.
The GoA-UN PoC achieved significant results where there was clear alignment with national priorities and strong ownership of the initiative by national authorities both during the planning and implementation stages.
UN Agencies targeting capacity development should not assume building the right kinds of capacity in the right ways without sufficiently understanding the actual capacity constraints and development needs that exist.
The burden related to UN Agency work planning, monitoring and reporting is still very large and the expected simplification of procedures from a DaO approach has not yet happened. The existing quality in reporting has come at the expense of UN staff time and energy.
Consolidation of results and indicators in one programme results frameworks should not come at the expense of clarity in the results chain and theory of change. A clear and compelling story line should be traced.
Similar to middle and upper income countries, cash assistance to government does not appear to be an essential ingredient for effective partnership and programme ownership.
Full report in PDF
PDF files require Acrobat Reader.