We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2017 Macedonia: Evaluation of the Early Literacy and Numeracy Programme



Author: José Noijons, Beti Lameva, Ana Mickovska Raleva

Executive summary

Evaluation of the Early Literacy and Numeracy Programme in the former Yugoslav Republic of Macedonia

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as ‘Part 4’.

Background:

This Evaluation Report presents an evaluation of the Early Literacy and Numeracy Programme in the former Yugoslav Republic of Macedonia (the ELN Programme) 2009-2015. The UNICEF country office in Skopje instructed DUMA Consultants to conduct an evaluation that is primarily summative with a purpose of assessing the results of UNICEF support in raising the quality of teaching and student achievements through the ELN Programme. The evaluation was undertaken at the end of the UNICEF country programme that was signed with the Macedonian Ministry of Education and Science (MoES). The ELN Programme has been fully funded by UNICEF with a total budget of $1,140,000.

The ELN Programme (2009–2015) was developed in response to repeatedly low student achievement in international assessments, such as TIMSS, PIRLS and PISA, and in national assessment tests; to the reactions of secondary schools on the low level of numeracy and literacy of students entering secondary schools, as well as to the weak mechanisms in place to support learners lagging behind or not performing to the best of their abilities. The education system and the teachers in particular, had generally low expectations for students. Little attention was paid to those not mastering the language of instruction (such as Roma children), or having gaps in their mathematics or language knowledge. The ELN Programme was implemented on a national level in all primary schools in the Former Yugoslav Republic of Macedonia.

Purpose/ Objective:

The main objectives of the evaluation were:

  • Assessment of the relevance, efficiency, effectiveness, sustainability, coherence and, to the extent possible, the impact of the project interventions;
  • Identification of any unintended outcomes;
  • Identification and documentation of lessons learned in relation to strategies and interventions used to address certain critical bottlenecks.

The evaluation has focused on three key areas of the ELN Programme:

  1. Capacity building of all involved in the Programme;
  2. Management and support provided by the responsible institutions; and
  3. Implementation of the ELN Programme in schools.

Methodology:

The basic principles of the evaluation methodology were the following:

  • Its quality assurance system serves to guarantee, reveal and confirm the quality and relevance of the evaluation;
  • Stakeholders take full responsibility for their responses;
  • Stakeholders’ own strategies and targets are taken into account in the assessment process;
  • The Evaluation Team has been formed in conformity with the ToR and adheres to these.

The methodology has responded to the set objectives and questions of the evaluation assignment and taken into consideration the defined ELN Programme indicators in its Logical Framework mentioned above. Baselines, progress measurements, annual reports and other available reliable data and materials collected through monitoring and evaluation activities have been considered. The chosen evaluation methodology demonstrates impartiality and lack of bias by relying on a cross-section of information sources and on the use of a mixed methodological approach to ensure triangulation of information. The participation of key stakeholders has been obtained in all phases of the evaluation.

The evaluation has used an established, non-experimental design, based on a Standard Evaluation Protocol (SEP). The evaluation has applied the United Nations Evaluation Group (UNEG) norms and standards and the UNEG ethical guidelines. Care has been taken that the interests of various ethnic and language communities were respected. Although every effort has been made to do justice to the multifaceted nature of the ELN Programme under evaluation, the evaluation has not been able to cover every situation or answer every question that arose, through lack of relevant data, time constraints and/or financial constraints.
The evaluation is based on documents provided by UNICEF and participants in the ELN Programme, such as baselines, progress measurements and annual reports, and other data and materials collected through monitoring and evaluation activities.

Findings and Conclusions:

The evaluation showed that the ELN Programme has been successful to a sufficient extent to improve the achievements of students, although the absence of end-of-project student assessment and the implementation of other programmes during the same period makes it difficult to ‘extract’ the specific effects of ELN on student outcomes.

  • Most teachers consider that the Formative Assessment training influenced their teaching positively but they feel they need more time to prepare for lessons that include the use of formative assessment. They would need support for their further implementation in the next few years. This is particularly true for teachers working in schools with a majority of Roma students.
  • Teachers are positive about the trainings in the Recording of Good Practices, but for teachers in schools with a majority of Roma students it is difficult to manage classes and to make sure everything goes as planned. It seems that lessons conducted by applying the approaches learned at the training require more time. Teachers feel the need for additional support in the next few years.
  • Stakeholders have been aware of most of the programme expected results, be it that they sometimes phrase them differently. In various official reports on the ELN Programme different formulations were used.
  • The ELN Programme can be seen as relevant to the teachers’ needs and related to the national curriculum. However, the alignment with the mathematics curriculum has been problematic at times and in places.

Recommendations:

The Evaluation Team’s recommendations are given in order of priority per category.

SWOT

  • BDE to apply the new ELN teaching methods to the higher grades as well.   
  • Continued financial support to be assured to materially implement some of the teaching methods at national and local level.

Formative Assessment trainings

  • All grade teachers should be given a basic training in formative assessment. Teachers should attend additional meetings with their colleagues from the region and the BDE advisors in order to exchange experiences and deepen their understanding.
  • Future teachers should be trained in formative assessment during their pre-service preparation.

Trainings in the Recording of Good Practices

  • The recorded good practices should be posted on the BDE web page and linked to other websites which teachers use. More teachers should be involved in the training of recording good practices
  • Teachers should receive continued assistance from the BDE in integrating the new teaching approaches in the new Math curriculum.
  • Teachers instructing in Albanian language, and those teaching in majority Roma schools should be provided extra assistance for integrating the new literacy and numeracy approaches in the relevant curricula.
  • BDE to organize trainings in relating new teaching techniques to existing curricular goals.
  • BDE to conduct an evaluation study on teachers’ pedagogical knowledge.
  • The National Examination Centre to conduct repeated national assessment of early numeracy.
  • The country to participate in TIMSS and PIRLS for the assessment of effects.

Lessons Learned:

  • The introduction of new educational programmes is more than familiarizing and training teachers to implement new methodologies. It is also about facilitating stakeholders in terms of funds, logistical and content support.
  • The introduction and the implementation of the ELN Programme seems to have been a rather top-down process. Stakeholders will be more motivated if they are involved in the introduction of new programmes right from the start and are encouraged to come up with suggestions to improve it.
  • There is a danger that stakeholders look at new educational programmes selectively, i.e. that they focus on their direct interests without sufficient realisation of the general aims of a programme. Thus it is important to regularly present them with the complete picture.
  • Introducing new relevant teaching techniques can have a positive influence on the cooperation between teachers and support staff.
  • Recordings of good practices can be shared on Facebook pages which teachers use to exchange information on education-related issues.
  • Dedicated programmes like the ELN Programme can have unintended positive effects, such as approaches being transferred to other subjects.
  • The data collected in the course of the ELN Programme provide a very good overview of the training outputs and enable making analyses very easily. This is a practice which can and should be replicated by future programmes.
  • The developed fidelity tools need not be limited to the ELN’s programme activities but can easily be applied in the everyday monitoring of teachers’ lesson planning, classroom practice and providing specific feedback.
  • Trainings in formative assessment are useful in improving and motivating teachers and students. Such trainings are useful not only at the lower (ELN) grades but in all grades in order to accomplish the goal of improved student achievement.


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2017

Country/Office:
Macedonia

Region:
CEE/CIS

Type:
Evaluation

Theme:
Education

Institution:
DUMA Consultants

Language:
English

Sequence #:
2017/002

New enhanced search