2012 Macedonia: Enhancing Inter-Ethnic Community Dialogue and Collaboration
Author: Robin Philip Sluyk
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."
This report presents the findings of the Final Evaluation (FE) of the UN Joint Programme “Enhancing Inter-Ethnic Dialogue and Collaboration”, implemented by UNDP, UNICEF and UNESCO in the former Yugoslav Republic of Macedonia. The FE took place in June and July 2012, including a field mission from June 7-20.
The goal of this FE is determined by the Monitoring and Evaluation framework of the MDGF, which is set out in the Monitoring and Evaluation Strategy and the Implementation Guide for Joint Programmes under the Millennium Development Goals Achievement Fund.
The FE focuses its attention on three levels: design, process and results. It involves gathering data from existing documentation and monitoring data, complementing those data through surveys and/or interviews with Programme staff, partners and beneficiaries, and comparing the findings with baseline data (where feasible). The report will use the findings of this process to reach conclusions and make recommendations.
The FE has the following specific objectives:
1. To measure to what extent the joint programme has contributed to solve the needs and problems identified in the design phase;
2. To measure the joint programme’s degree of implementation, efficiency and quality delivered on outputs and outcomes, against what was originally planned or subsequently officially revised;
3. Measure to what extent the joint programme has attained development results to the targeted population, beneficiaries, participants whether individuals, communities, institutions, etc;
4. To measure the joint programme contribution to the objectives set in their respective specific thematic windows as well as the overall MDG fund objectives at local and national level. (MDG-s, Paris Declaration and Accra Principles and UN reform);
5. To identify and document substantive lessons learned and good practices on the specific topics of the thematic window, MDG-s, Paris Declaration, Accra Principles and UN reform with the aim to support the sustainability of the joint programme or some of its components.
The evaluation methodology is based on ratings of the OECD-DAC established evaluation criteria: relevance, efficiency, effectiveness, impact and sustainability. Additionally, the evaluation will study the extent of partnership and cooperation, its effects on project implementation and possibilities of duplication. The evaluation was carried out in three phases:
• A desk research phase including the identification of the evaluation questions;
• A field phase including interviews with a representation of project beneficiaries and other stakeholders; and finally
• The drafting phase of the Final Report.
The mission was divided into field visits in Skopje as well as in the municipalities included in the programme: Struga, Kicevo and Kumanovo. The fieldwork in Skopje provided an opportunity to meet and discuss the programme with relevant actors from government, UN agencies, educational institutions (universities, schools), media partners, civil society partners, international organisations and donors. The field work with the partner municipalities provided input for analysis of the programme achievements, its gaps and opportunities that were gained from visits to programme sites, interviews and focus groups with relevant local government stakeholders, programme team, partners and beneficiaries.
Findings and Conclusions:
Overall, the efficiency of the JP was good.The data and programme records show that overall the activities and related disbursements reflect a steady built-up to a peak in performance around the middle of the programme life cycle, and an equally steady decline towards programme’s end.
The exception is the programme’s behaviour under Outcome 3, whose activities seems to have started peaking as of December 2011 and is not yet finished, although its overall financial map does follow the curves of Outcomes 1 and 2. This is partly explained by the fact that the Year 2 activities for Outcome 3 were more cost intensive than its set for Year 3.
Additionally, the JP did start rather late; the Project Document (i.e. the contract between the MDG-F and the JP Partner agencies) was signed in July 2009, while programme activities started in December 2009. The effectiveness of the JP is undeniable but mainly based on verbal feedback from stakeholders and counterparts during the interviews, which was generally positive. Additionally the programme has made progress towards achievement of objectives contributing to Millennium Development and thematic window goals; particular contribution was made through the production of valuable studies on the state of capacity gaps, LSBE, mediation and gender.
However, measuring the ‘real’ effect of the JP is imperilled due to the lack of SMART indicators and a corresponding monitoring system, as well as through the absence of pre- and post testing mechanisms for the delivered training, necessary to establish achieved capacity development and increased awareness.
The JP had a significant impact during its life time, especially on the output level as well as within its efforts on Education. The JP prospects of sustainability are mixed. The Programme made strenuous efforts to assure ownership amongst its counterparts. It could not ensure sufficient capacity to safeguard sustainability.
The analysis on Effectiveness brought to light that solid interventions may not have the claimed effect without solid programme logic, illustrated by a logical framework matrix and SMART indicators. It is recommended that for future initiatives, better use is made of the principles of Project cycle management. A sound logframe enhances better monitoring (e.g. reporting system through templates that relate to indicators and reflect the hierarchy of the management model). The use of (simplified) Gantt charts to keep track on high volume intervention can be considered as well.
On the area of Impact and Sustainability, the programme took great strides to follow up and implement the MTE recommendations on finding creative ways to reach out to the private sector and mobilise private resources by targeting and including private universities and (some of the) media. This is a recommendable achievement, and deserves continued and (where feasible) intensified efforts to further sustain impact and long-term sustainability. The communication and advocacy strategy could be a good mechanism to support this.
Furthermore, the JP followed the MTE findings in strengthening the local-national link through the promotion of joint activities, inclusive workshops or other alternative mechanisms that ensured contact and exchanges between the local and national levels of implementation. Again this is recommendable, and it deserves continuation until the very last stage of implementation as well as following up post-project where feasible.
The implementation of this JP showed that the overall coordination (in fact: the Programme Manager function) can be carried by the RC, as opposed to recruiting an external Programme Manager. It obviously has functional as well as financial advantages. JP management structures are rather complicated by their nature with the participating agencies adhering to their own administration rules. Professional knowledge of such, as well as personal relationships within an existing country team, may add very well to overall synergy, which should be a trademark for any JP. To the FE it seemed more illustrative of the strength of the years long (pre-JP) cooperation between partner agencies, which allows for personal relationships to foster professional cooperation. It may very well have been the better solution for the JP, considering that a Programme Coordinator in a JP does not have full managerial authority over the partners. A well-known and established representative may overcome these gaps. Nevertheless, this may not work in all JP-s. Perhaps this was situational, but it provides grounds for further study. In any case, this position requires solid knowledge of and experience in project cycle management.
Despite the JP’s efforts to follow an inclusive approach and aim at integrated platforms, ownership was still fragmented. Perhaps this is a ‘natural’ result for a programme covering so many different levels; still it is not ideal. The MTE proposed role of the C&A strategy to overcome this was not clear by project’s end despite JP efforts. Steps to overcome this fragmentisation include strengthening of vertical lines.
Full report in PDF
PDF files require Acrobat Reader.