Through improving assessment and evaluation practices to reduce educational inequalities

An international conference was recently held in Sarajevo, presenting 10 key recommendations from the OECD (Organization for Economic Co-operation and Development) Review of Evaluation and Assessment in Education in Bosnia and Herzegovina, and discussed in more detail some of the policies and practices that can help improve quality. The conference, organized by UNICEF Bosnia and Herzegovina and the OECD in partnership with the Bosnia and Herzegovina Ministry of Civil Affairs and with the support of the Norwegian Embassy in Bosnia and Herzegovina, brought together participants from Bosnia and Herzegovina education authorities, universities, international and non-governmental organizations, teachers' associations and other organizations involved in the education sector.

We spoke about the Review and key recommendations with Caitlyn Guthrie, one of the co-authors of the Report and a policy analyst at the OECD. At the event, Caitlyn Guthrie presented to the audience 10 key recommendations, of the many specific recommendations contained in the Review.

You have conducted research about evaluation and assessment practices in education in many education systems worldwide. What makes Bosnia and Herzegovina unique and what is common? What are the advantages and what are the constraints in Bosnia and Herzegovina?

There are certainly things Bosnia and Herzegovina has in common with other education systems - such as an emphasis on competence-based education, which is something that a number of countries around the world are trying to introduce. Bosnia and Herzegovina also has a state-level expert agency (APOSO) that is responsible for setting standards and evaluating education in the country – and this type of agency exists in many education systems in the Western Balkans and in OECD countries.

At the same time, Bosnia and Herzegovina has one of the most complex education systems in the world, with different levels of responsibility for education spread across the state- entity- and canton levels. An advantage of this approach – as in other decentralised systems like Germany, the USA, Canada, and Australia - is that it can allow government partners to respond more effectively to local needs. However, there are also challenges, including disparities in resources for education – or difficulties in developing and implementing systemic reforms.

Something makes Bosnia and Herzegovina especially unique, compared to other countries where decisions about education are not taken at the central or state-level – is the limited degree of cooperation across government partners. Other decentralised systems have found ways to set high-level education goals; collect comparable data; share resources about instruction and have some measures of accountability for improving the quality of education - and do all of this while maintaining the autonomy of individual education authorities. But we saw very few examples of this type of cooperation across Bosnia and Herzegovina, which makes it hard to scale-up improvements in teaching and learning and even collect comparable and reliable data on student learning outcomes.

According to UNICEF U-Report, over 80 per cent of young people in Bosnia and Herzegovina feel the grades are not the adequate measure of their learning. How can we help teachers to assess the learning of their students more effectively?

Our report sets out several ways that governments and education actors can help teachers more effectively assess student learning. These include covering student assessment in teacher initial preparation and professional development, creating clear assessment criteria and providing teachers with resources to boost their assessment literacy, like examples of marked student work that shows what achievement looks like at different levels, rubrics, or samples of diverse types of assessments that they can adapt and use in their classroom practice. Another important recommendation we make is to increase the use of standardised external assessments - which can help teachers benchmark their classroom assessments. For now, this type of external assessment data only exists in 1-2 administrative units in Bosnia and Herzegovina, representing a missed opportunity for students to get an objective and reliable measure of their learning.

In the OECD Review of Evaluation and Assessment of Education: Bosnia and Herzegovina the OECD recommends a set of recommendations to improve teacher appraisal processes. Why is this so important? How to establish systems that motivate for improvement?

Teacher appraisals are important because they provide teachers with feedback on their performance –that can inform their professional development, hold teachers accountable for the quality of instruction they provide, motivate them and ultimately help raise student outcomes.

The most important part of any teacher appraisal system is having clear professional standards that set out expectations for what teachers should know and be able to do in their role. The second step is to use these standards as the basis of appraisal processes. One of the most important ways appraisals can help improve teaching is by using feedback from teacher self-evaluations, as well as regular appraisals by school leaders and/or peers – to reflect on their practice and identify areas for improvement.

Another thing that many OECD countries do is link teacher standards with teacher career paths whereby promotions and higher salaries are linked to merit, additional responsiblities and excellence in teaching, rather than years of teaching experience alone. Bosnia and Herzegovina systems already have established teacher career paths – but these need to be linked to standards and appraisal processes if they are to help motivate and reward effective teaching.

The report highlighted 10 key recommendations. How many recommendations does the report contain, and to what extent is it realistic to expect them to be implemented? How important is it to start all the proposed reforms as soon as possible?

We actually have around 26 recommendations in the report, and a number of specific policy “actions” as well. While all of these measures have the potential to raise the performance of Bosnia and Herzegovina education systems, we chose to focus on the top 10 in the event of 12 May that we thought would speak to the largest amount of participants. Some of our recommendations could be called “low-hanging fruit” or things that different actors can do immediately – like creating networks and resources for teachers to use in their classrooms.

Other recommendations are likely going to be more difficult to implement, like re-vitalising the Conference of Education Ministers to set goals and common performance indicators at the country level. It is really up to Bosnia and Herzegovina governments, international partners and communities to determine which recommendations are the most urgent, feasible and relevant for their individual systems. We were encouraged when the MoCA announced that progress has been made to establish a state-level EMIS – this type of initiative can help generate data to hold governments and education systems accountable for providing quality education to all students.

There is much discussion about external matura. What is the OECD recommendation for this area?

At present, there are no external exams at the state-level in Bosnia and Herzegovina and only Tuzla canton has an external exam at the end of secondary education (ISCED 3). There are some examples of standardized exams at different levels in other parts of the country – but the reality is that many administrative units lack the financial resources and technical capacity to develop a rigorous exam system.

Our review team recommended that competent education authorities and APOSO work together, with support from the donor community, to develop an optional external examination of core competences at the end of secondary education. We know this has already been discussed in Bosnia and Herzegovina education debates, but we see it as an important means for addressing grade inflation and raising the value of secondary qualifications.

Our report goes into detail about the different arrangements that could help operationalize a Bosnia and Herzegovina matura within the country context. For example, by having competent education authorities determine the weight the exam would have as a share of their secondary graduation requirements.

What is the difference between formative and summative assessments and how to improve both in Bosnia and Herzegovina?

Summative assessment refers to assessment of learning – or summaries of learning that already took place. Formative assessment is about using assessment to shape instruction and help improve the learning process. Both types of assessment are important and education systems regularly revise their assessment policies and practices in an attempt to find a balance between the two. Bosnia and Herzegovina has a lot of experience in summative teacher assessments – and these help track students into different secondary pathways (i.e. vocational or general). However, available data suggests that a large share of students in the country go through school without mastering basic competences – so our report puts forward recommendations for strengthening the formative value of assessments – to help identify learning gaps early on and address them before them become problematic.

The report also recommends many activities to establish and improve school evaluation. How can schools be supported to evaluate their work? How to ensure that we support the schools that are in the most need?

Bosnia and Herzegovina governments can help schools evaluate their work by developing a set of indicators of school quality. The indicators should address shared concerns, including student progress and learning outcomes (e.g. rates of student absenteeism, advancement and/or graduation), school processes (e.g. teaching methods, guidance and support for students compliance with regulations) and contextual features that affect school performance (e.g. geographic location, number of shifts or use of multi-grade classrooms, socio-economic situation, etc.).

In the absence of an external school evaluation system, schools can conduct self-evaluations – and reflect on how their school performs vis a vis the quality indicators. School leaders and expert advisors from pedagogical institutes will have an important role to play in supporting the self-evaluation exercise and helping schools to develop improvement plans based on the results.

To identify and support schools with the most need, our report recommends that competent education authorities develop a methodology to identify at-risk schools using the school quality indicators. Then they should direct more resources and targeted support towards these schools, such as intensive, hands-on coaching from pedagogical institutes or by creating a formal networking programme that pairs at-risk schools with schools that are doing well. Education authorities should also consider publishing summary reports of how well schools are doing according to the indicators rather than individual school results to avoid putting undue pressure on school staff and help reinforce the school improvement focus of the school quality indicators.

Bosnia and Herzegovina has 12 education systems but not enough data on the quality of those systems. How to improve system-level evaluation to have the evidence-based reforms in place? How have you personally coped with so many different education systems in one country?

Bosnia and Herzegovina was certainly one of the more challenging systems we have covered but our review team tried to understand each of the individual systems we sampled – as well as the extent of cooperation at the entity or state level. Comparable data is undoubtedly important to support system accountability and improvement. In this area, we recommend aligning data collection with international data standards and protocols and committing to future cycles of international assessments – to collect data that can allow authorities to set measurable goals and monitor progress over time. We also recommend investing in the data capacity of administrative units – thorough peer learning initiatives and continuing to establish the state-level EMIS.   

How can international large scale assessments help in establishing evaluation and assessment mechanisms? And how can Bosnia and Herzegovina work towards improving the capacity to conduct its own assessments?

As mentioned, there are only a few (around 3) competent education authorities in Bosnia and Herzegovina that currently conduct standardized learning assessments. This means that there is very little reliable data on learning outcomes in Bosnia and Herzegovina and there is no way to compare learning outcomes across the country to conduct rigorous evaluations and help monitor and reduce system inequities. In the absence of local external assessments, we recommend that Bosnia and Herzegovina formally commit to the long-term participation in major international surveys like PISA, TIMSS and PIRLS, which can allow authorities to set measurable policy goals, and help to review performance over time.

What are some COVID-19 lessons in relation to assessment and evaluation? What is the future of evaluation and assessment practices in relation to technology and online learning environments?

During the pandemic, different administrative units in Bosnia and Herzegovina made use of a diverse range of approaches to remote learning, including using TV, radio and online learning platforms. While the majority of students in the country were able to benefit from these opportunities, the most vulnerable students struggled to adapt, especially those from socio-economically disadvantaged backgrounds and Roma communities, as well as SENs.  The pandemic highlighted the need for better strategic planning and stronger support for individual student learning. Technology certainly has a role to play in this regard, but preventing the learning crisis from becoming a learning catastrophe will require that Bosnia and Herzegovina use assessment and evaluation practices to address educational inequities that have been exacerbated by the pandemic.