We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2017 Macedonia: Evaluation of UNICEF Roma Health Mediators Programme



Author: Zoran Stojanov, Boge Bozinovski

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

The object of the evaluation was the UNICEF supported Roma Health Mediators Program 2013-2015. The Roma Health Mediators program was initiated with a goal to improve health conditions of Roma people and their health care access, acting as a link in the chain for improving communication between the Roma population and the health system in order to ease access to health care, establish trust between patients and health care providers, and develop habits regarding self-care and care of others, which is also a constitutional obligation of all citizens.
Roma suffer from a lower health status due to different reasons. The number of self-declared Roma, according to the 2002 population census, is 53,879, i.e. 2.6% of the total population. Estimates of the Roma population provided by the authorities, researchers and NGOs vary from 80,000 to 260,000, i.e. 4%-13% of the total population. Their living conditions, often sub-standard, make them susceptible to health hazards. Those without personal documents have difficulties accessing medical and gynecological services. Data on the health status of Roma in Macedonia is worrying. Life expectancy of Roma is ten years shorter than the national average. Infant mortality rate is almost double that of the general population.
UNICEF supported Roma Health Mediators program in all three main components. One component is related to development of policy framework, the second component focused on capacity building of RHM and the third component focused on monitoring and evaluation of the program.

Purpose/Objective:

Evaluation Objectives: The main objectives of the evaluation are to:

  1. assess the relevance, efficiency, effectiveness, sustainability and to the extent possible impact of RHM work;
  2. identify and document lessons learned and the contribution of UNICEF to these systems or impact changes; and
  3. provide recommendations to guide the RHM program for the next program cycle.

Methodology:

To achieve the evaluation objectives, the evaluation framework was developed which identifies indicators for each question. It also identifies the sources of information, evaluation methods, the range of documents reviewed and the key informants interviewed for each question. The methodology comprised a mix of face-to-face semi structured interviews, focus group discussions, desk-based research and review of existing reports, documents and available secondary data. An extensive desk review summarized available documents and data collected through the field work, aimed at provision of a concise but thorough synthesis of activities completed over the past 5 years.  In-depth interviews with key informants qualitatively informed implementation successes, problems with program management and co-ordination, and options for change in future.  The purpose of FGDs with beneficiaries was to gauge the extent to which project might have contributed to improvement in quality of services and healthy behaviors, as well as utilization of services. Furthermore, FGDs identified key bottlenecks/challenges in access to health and social protection services. The principal topics to pursue as part of these FGDs included: i) degree of accesses to services; ii) their perception on the service quality; iii) bottlenecks, challenges.

Findings and Conclusions:

The RHM program has been aligned with the relevant national development strategies and polices. It has also been harmonized with the main International human rights treaties (CRC, CEDAW, CPRD) and addresses priorities of the main stakeholders in the country. The program has been aligned with the Ministry of Education prescribed curricula and the capacity building activities were relevant for Roma Health Mediators. Data collection and monitoring activities developed with UNICEF support were to a large extent relevant for the country context.
The program planned results were partially achieved. As specified in the 2011 Strategic Framework, the main goal of the RHM program was to provide services to 75% of Roma population living in 16 municipalities with highest proportion of Roma. This goal is partially achieved with 12 mediators providing services in 9 municipalities. The program has been effective in strengthening monitoring and reporting capacities in RHM and MoH. Still, majority of interviewed stakeholders and mediators expressed an opinion that additional training is needed in this field for in order to achieve high level of skills required in practice.
The program has achieved the results with optimal efficiency, with resources invested in capacity building being used in efficient manner. The activities were coordinated and conducted in close cooperation with all stakeholders ensuring efficient implementation.
The program was successful in improving practice and quality of services provided by RHM and the practices and quality of services provided to Roma in general.
The program contributed to improvement in allocation and use of resources in the RHM program. It also promoted ownership by Government partners over different program activities. However, the Government has limited capacity to sustain the RHM program components established with UNICEF support.

Recommendations:

Identify optimal model and institutionalize the RHM. After over five years as a project activity, some serious decisions on the future of the program need to be made. It is highly recommended for the RHM program to continue, not as a project activity but as an institutional program within MoH.
Increase the number of RHM, to cover large Roma communities in the country. Expanding the RHM network is one of priorities for all stakeholders. This is particularly important for Prilep, Bitola, Kicevo and Veles, municipalities with some of the largest Roma communities in the country.
Extend the RHM coverage to the tertiary level health institutions.
Revise RHM job description and mandate. The RHM job description should clearly state that the RHMs mandate is to identify and provide services to the most marginalized households and individuals within Roma communities.
Continue with mentoring support in more structured and systematic manner. Mentoring support needs to be included as an element of the RHM continuous in-service capacity building process as it has powerful effect on motivation, quality of work and achievements of less experienced mediators.
Upgrade the data collection system by making it more user friendly and useful for RHMs. Data collection system needs to be developed to provide useful information for RHMs, while keeping time and efforts for data entry at the minimum possible. Situation of recording same data twice is highly demotivating and it is strongly recommended to avoid such an approach.   
Increase visibility of RHM program and the achieved results. Positive results and achievement of the program need to be communicated with general public, with local level institutions, academia, and international organizations and with potential donors. Results published in the 2015 IPH report present RHM in very positive light and should be utilized for promotion of the program.

Lessons Learned:

High turnover rate of mediators is one of the major obstacles preventing the RHM program to achieve higher success. High turnover is caused by different factors, both subjective and objective in their nature and it affects program implementation in various aspects and levels.
Human rights based approach in work, knowledge of relevant laws and primary health care programs are crucial for ensuring success in the RHM work. RHM are human rights workers with primary mandate to protect human rights in the health and social sector. The HRB approach is the best tool they have to assist their clients, applicable in all situation of discrimination.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2017

Country/Office:
Macedonia

Region:
CEE/CIS

Type:
Evaluation

Theme:
Health

Language:
English

Sequence #:
2017/003

New enhanced search