We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 Montenegro: Evaluation of the project “Using textbooks in DAISY format - Primary school teacher training in using audio textbooks"



Author: Salvador Bustamante Aragonés

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

With the aim of providing accessible and adapted teaching and learning materials for CwDs, in June 2013, UNICEF Montenegro Office, the Ministry of Education, Textbook Publishing Agency and the Resource Centre for CwDs initiated the first ever production of audio textbooks in DAISY format for children with visual impairments and children with learning difficulties. The initiative was implemented with the generous contribution of the Drama Academy, whose students recorded the books on a voluntary basis. Six audio textbooks/primary school readers in Mother tongue have been produced so far and approved by the National Education Council. This initiative is ongoing and the production team at the Resources Centre continues to work on making more textbooks accessible in DAISY audio format.  However, further challenges are linked to scaling up the efforts, regular updates of the material and ensuring their effective use in classrooms.

Purpose/Objective:

The purpose of the evaluation was to document the results of the project “Using textbooks in DAISY format – Primary school teacher training in using audio textbooks”. The final objective of the project is to contribute to the development of an effective inclusive education system in Montenegro. This is a long-term objective that requires the support and collaboration of different stakeholders and frameworks. The 1st specific objective identified is “Teaching process improved in 24 inclusive schools and the Resource Centre”. It specifically refers to the contribution of the project to develop education staff skills and attitudes to ensure an inclusive and effective teaching environment. The achievement of this objective requires both the use of new technologies (DAISY) and classical educational techniques for CwSEN. Educational staff includes teachers, education specialists and directors, but parents also play an important role by supporting their children in the learning process. The 2nd specific objective is to improve the learning process for all students in 24 inclusive schools and the Resource Centre. This objective refers to the contribution of the project to developing an improved learning environment, focusing on the changes produced on students. Students (especially CwSEN) should experience an improved learning environment and develop positive attitudes toward school and individual learning at home through reliable modern learning technologies. Students should be accepted among them as social members of the teaching groups. Learning process improvement could be measured based on changes in attitudes towards school, emotional satisfaction and usefulness concerning the use of DAISY, engagement in students’ own learning, and the development of learning opportunities for all students. The two specific objectives are intimately intertwined but for evaluative reasons, these two aspects have been separated and studied individually.

Methodology:

The evaluation is guided by the following criteria: a) Relevance; b) Efficiency; c) Effectiveness; d) Impact; e) Sustainability; and f) Cross-cutting issues. Additionally, the evaluator included a programme theory approach. This perspective states that every project is built upon explicit or implicit models on how the intervention will lead to the desired results. To identify these models, a theoretical model of the programme was designed to establish the underlying intervention rationales, the causal linkages between all elements in the programme that produce results. This theoretical model includes all the relevant aspects of the programme: the results and objectives, processes and activities undertaken to achieve them and the means and resources that are in place to carry out these activities. This approach of studying the intervention based on all its elements is known as the systemic perspective. The relevance of this approach lies in the usefulness it provides in designing an evaluation, allowing a better understanding of the programme, including the cause–effect links generated at all levels. The methodological approach includes quantitative and qualitative strategies. Both quantitative and qualitative techniques have been used to obtain the information needed to answer the evaluation questions. Particular emphasis was given to cross-validation of data and an assessment of plausibility of the results obtained. The methodological mix included a desk review of existing literature, semi-structured individual interviews, focus groups, a survey and direct observation.

Findings and Conclusions:

The project objectives are in line with national strategies and policies on inclusive education. The project design and rationale reflect international and national norms and strategies on human rights and gender equality. Training sessions, materials and produced DAISY audio-visual textbooks are suitable for the intended users. Production issues were detected lessening the potential impact. Implementing partners carried out activities in a highly efficient manner. A lack of information and awareness on the potential of DAISY textbooks prevent teachers and parents from being more involved. A lack of technical equipment at schools has been identified as a widespread threat to the effective introduction of DAISY textbooks. Early evidence of DAISY sessions improving group dynamics and CwSEN participation in class identified. The project rationale and objectives are relevant to national priorities and policies on inclusive education, and suited to addressing the main causes of exclusion for CwSEN. Despite the fact that the project design leads to the achievement of results during the implementation period, the current results-based framework does not effectively reflect the expected objectives, and lacks specific indicators and standards whereby to measure their level of completion and that will negatively affect the project in the future. The intervention was implemented in a highly efficient manner. The project relied on schools to inform and motivate parents to join training, but this was not effectively conducted. Promoting the project is essential in ensuring engagement of the direct beneficiaries and students. The evaluation could not establish clear and solid evidence of the project’s contribution to its medium- and long-term objectives. Implementing partners developed a level of ownership and responsibility towards the achievement of results that will potentially ensure the continuation of activities even in case of an eventual withdrawal of external funding.

Recommendations:

  • Recommendation 1 (Very High priority) - Implementing partners should: a) Develop specific training for parents, focusing on their particular role in supporting the process of learning; b) Training sessions and materials should highlight the adequacy and relevance of using DAISY textbooks with all students, not only CwSEN, and include a practical session on how to conduct a class with DAISY textbooks with a focus on inclusiveness; c) Produce a manual on the use of DAISY based on the training experiences and lessons learned, including best practices; d) To the extent possible, the project should address the identified production issues (inadequate intonation and modulation, voices by role, page information and other interruptions, etc.); e) Concerning copyright issues, a standard procedure to engage authors should be developed; and f) The intervention should ensure adequate equipment is available at participating schools for the effective introduction of DAISY audio-visual textbooks.
  • Recommendation 2 (Very High priority) - Implementing partners should extensively conduct promotion and awareness-raising activities in the areas of intervention, addressing education staff, associations of parents, other families and students (especially children with no special education needs).

Lessons Learned:

  1. Students with special education needs face difficulties and constrains to accessing their education needs in mainstream schools in Montenegro. Moreover, the evaluation process identified that differences in equal access to the project and its benefits are issues that also affect children based on their level of disability. When conducting the analysis of gender mainstreaming issues related to the project, triangulation of information obtained from different stakeholders showed that the unbalanced access to educational needs in mainstream schools is based on the students’ degree of disability, but is not due to their gender. No limitations in accessing those rights could be attributed to gender inequalities. This lesson should be taken into consideration when developing scaling-up strategies and studying other contexts and situations. Other potential constraints should be also studied, especially those that could affect marginalized groups.
  2. Several difficulties are encountered when introducing pilot projects in any kind of context. The lack of previous experience and knowledge about the intervention are factors that can seriously hinder the involvement of stakeholders at all levels – an essential condition to ensuring an effective and sustainable intervention. This affects not only the target group and indirect beneficiaries, but also the implementing partners. The level of commitment among implementing partners may vary, especially if an intervention is based on voluntary support, as is the case in the DAISY project. This situation has enabled a highly efficient management of resources in achieving results. The downside of this approach is the difficulty of securing stakeholders’ commitment in the medium and long term. Implementers should be open to sacrificing efficiency levels to some extent in order to ensure the full commitment and availability of all relevant stakeholders, especially those involved in the production process.


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2015

Country:
Republic of Montenegro

Region:
CEE/CIS

Theme:
Education

Type:
Evaluation

Partners:
Ministry of Education, the Resource Centre, the Institute for Textbooks and Teaching Aids, primary schools, Associations of Parents of Children with Disabilities and Children without Disabilities

Language:
English

New enhanced search