We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2018 Eritrea: Summative Evaluation of the Strategic Partnership Cooperation Framework (SPCF) between the Government of Eritrea and the United Nations 2013-2016 (“Driving towards MDGs”)



Author: Craig Naumann

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as ‘Part 4’.

Background

This summative evaluation report of the SPCF 2013-2016 in Eritrea presents evidence-based findings and recommendations. The evaluation mission, jointly organized by the UN Country Team (UNCT) and the Government of the State of Eritrea's (GoSE) Ministry of National Development (MoND), was carried out by an independent international evaluation expert. The report covers the implementation of the SPCF over the entire period 2013-2016. It consists of a descriptive background section of the country and programming context, the purpose and objective of the report, the methodology applied and related analytical limitations; analytical chapters and a final section with recommendations geared towards utility for the current, on-going programming cycle which started over 1.5 years ago. 

Purpose/Objective

The SPCF 2013-2016 implementation ended on 31 December 2016. Given that the new SPCF (2017-2021) is already being implemented, the conduct of the final evaluation for the previous SPCF (2013-2016) is critical since the findings and recommendations will provide the needed lessons learned for the implementation of the current SPCF (2017-2021) and future ones. Thus the purpose of undertaking the evaluation is to:
• Learn about what has worked, what has not and why, in the specific context of Eritrea and in the particular collaborative efforts of the UNCT;
• Verify results achieved within the revised framework of the SPCF;
• Assess the effectiveness of the strategies and interventions used; and
• Focus on aspects related to the implementation of the SPCF with the express aim of formulating lessons learnt and providing recommendations for improvements in the implementation of the current SPCF and UN Agencies’ programme cycle, including strategy, design and implementation arrangements.

The evaluation will assess the work of UN agencies in Eritrea along the following three dimensions:
a) Accountability: Assess the contribution of the SPCF 2013-2016 to national priorities and results, based on qualitative (perception) and quantitative evidence, articulating the changes that have occurred in a logical chain of results, which speaks to the accountability of the UN system.
b) Explanatory factors: Identify the factors that have affected the UNCT's performance and its contributions, answering the question of why the performance is as it is and explaining the enabling factors and critical bottlenecks; and
c) Continuous improvement: Assess the effectiveness of the UN in implementing the SPCF, particularly the extent to which progress has been achieved and what difference it has made.  The evaluation is expected to generate actionable recommendations for improving UN contributions under the current programme cycle. 

Methodology

The evaluation confronted two meta-sets of data; i) qualitative data gathered through interviews and ii) quantitative and qualitative data identified through the desk review and data archive searches as directed by Ministry counterparts. This is commonly referred to as a mixed method approach. Data collected through various collection tools was continuously and systematically triangulated against other information sources so that findings could be exposed to different perspectives and challenges whenever needed. The validity, reliability and comprehensiveness of findings gradually emerged throughout initial data collection, taking shape from vague contours to sharp features towards the end of the evaluation process.

A participative approach was applied to the largest degree possible, and the main stakeholders at UN and Government level were involved in collecting and classifying empirical data and contributing to the analysis of achievements.

During the data collection phase, the evaluation engaged the largest possible number and various sets of key informants. Interviews were conducted with key informants both in group settings and bilateral discussions at the levels of outcomes and outputs. Interviewees comprised the upstream policy making and oversight level, programme/project managers and operational management. To the extent possible, efforts were made to also include as many field level counterparts and end level beneficiaries among the interviewees.

All the data was confronted and exposed to cross-validation by sequential, iterative interview sessions. Explicit and implicit hypotheses and theories were tested by confronting them to statements and evidence gathered from among other stakeholders.

Systematic efforts were made to grant key informants opportunities allowing various stakeholders to voice concerns and critique. Thus, a clearer picture emerged that allowed to establish a better understanding of achievements and issues, problems or challenges.

Findings and conclusions

Relevance: SPCF objectives showed a quite satisfactory degree of consistency with Eritrea’s needs, national priorities, and international/regional commitments. The general planning process was challenging given the absence of a long-term strategic national master plan. As an alternative to high level nationally defined targets, the SPCF used the MDGs as overall “guiding stars” in the sense of macro-level goals.

Effectiveness: The general achievement of set targets against Outcomes can be assessed as “good” (average grade B-) based on actually measurable progress as per performance indicators.

Efficiency: Overall, 83 per cent (or USD156.8m) of the total planned budget was met at the end of the implementation period. The general efficiency would have benefitted from a joint SPCF oversight and coordination mechanism in the form of a Steering Committee. A comprehensive SPCF M&E and RBM oversight mechanism was sorely lacking. Programme and project implementation, typically at single agency level, drove SPCF implementation across most outputs.

Impact: The national implementation of internationally agreed commitments and UN Conventions and Treaties was supported via a number of activities, including those related to women’s rights and anti-SGBV via CEDAW and human rights via the UPR. Results achieved under Outcomes 1, 2, 7 and 8 contributed the most towards MDGs. Interests of the most vulnerable (refugees, children, orphans, physically handicapped etc.) were safeguarded through work under Social Protection and Education in particular. Gender aspects were generally mainstreamed across SPCF Outcomes.

Sustainability: There is evidence that the SPCF programme improved long-term institutional capacity along the parameters of technical expertise, financial independence and participation of rights holders in process.

Recommendations

1. Ensure that the UN-internal RC/UNDP firewall is fully in place
2. Consider inviting WB and AfDB to join UNCT
3. Integrated budgeting and expenditure tracking through RGs (along with reporting against SPCF indicators)
4. Full implementation of DaO SOPs
5. Fast-track BOS implementation
6. Consolidate UNCT agency presence within main UN compound
7. Provide procurement training and practical support (HACT etc.) to GoSE
8. Joint Monitoring through RG-based multi-stakeholder joint monitoring missions
9. Align draft joint integrated implementation plan
10. Joint UNCT/GoSE workshop to brainstorm on novel JPs around UNDP SP
11. Design more integrated SPCF Pillars/RGs including integrated result chains, leading to shift from Outcome-based coordination and implementation practices to integrated RG-driven approach
12. Increase level of ambition and widen scope of (existing) “Data and Planning” and “GEWE” JPs across most if not all Outcomes
13. Shift from separate CPDs and CPAPs to common programming, management, reporting
14. Organize Strategic Planning Capacity Building Training and Brainstorming Workshop on SDGs
15. Introduce customized Theory of Change at RG and Outcome levels
16. Design SMART DaO performance indicator/tracking matrix
17. Joint UN/GoSE RBM training linked to revision of RRF/M&E logframes
18. Ensure that WP indicators are logically linked to SPCF output metrics
19. Introduce basic experimental research for recurrent programme features
20. Carry out an impact evaluation and applied agro-anthropological research on SLM Serejeka project



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year: 2018

Office/Country: Eritrea

Region: ESAR

Type: Evaluation

Theme: Program Excellence (Cross-Cutting); Country Program Evaluations; UN Reform and UNDAF evaluations; donor portfolio evaluation

Language: English

Sequence #: 2018/001

New enhanced search