We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 South Africa: RUBRICATE: Independent Evaluation Final Report



Author: Khulisa Management Services

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

In 2015, Khulisa was commissioned by UNICEF on behalf of the Department of Basic Education (DBE) to conduct an independent evaluation of RUBRICATE, a Commercially-Produced, Integrated Assessment Platform for the Schooling Sector (LPRS-LMA-9115026). RUBRICATE was seen as an assessment platform (which boasts additional features such as recording and analysing learner data) which could potentially serve to contribute to the vision and objectives set-out in the DBE’s “Action Plan to 2014: Towards the Realisation of Schooling 2025” (the Action Plan is currently being revised to cover the period to 2030 in line with the National Development Plan) along with the 2004 White Paper on e-education, both of which advocate for the use of ICTs in schools to enhance teacher development and learner outcomes.

Purpose/Objective:

The evaluation was directed by two over-arching primary objectives:

  • The extent to which teachers have used RUBRICATE in provinces that have invested in the platform
  • The extent to which learning, teaching and assessment have improved as a result of teachers’ exposure to and use of RUBRICATE

Methodology:

The evaluation took a primarily qualitative approach, using a case-study methodology, based on stakeholder observations, perceptions and experiences gathered through Focus Group Discussions (FGDs) and Key Informant Interviews (KIIs). In total, 24 FGDs were held with teachers in the Gauteng, Western Cape and North West Provinces (three of the four provinces in which RUBRICATE has been implemented) and 81 teachers were included in FGDs. 18 KIIs were held with stakeholders across district, provincial, and national offices in four Provinces; Gauteng, Mpumalanga, Northwest, and Western Cape. Some quantitative data was collected during the evaluation to both contribute to, and support, the qualitative data. In total, 81 teachers completed Rapid Educator Surveys (RESs). In general, data was collected from various sources to allow for triangulation purposes and the validation of findings.

Findings and Conclusions:

There is limited use of RUBRICATE in Gauteng, and only the updated RUBRICATE Junior has been distributed to schools. Commonly reported reasons for the lack of use were the limited training provided by the district, and a general lack of user-friendliness which deterred teachers from using RUBRICATE

Of those schools where RUBRICATE is being used, teachers only use it to develop assessment tools. In these schools, the benefits were limited at best, and many of the teachers did not report any noticeable changes to teaching and learning as a result of RUBRICATE.

There is limited use of RUBRICATE in Western Cape, and there have been challenges regarding distribution of the CAPS-aligned version as a result of the need to adhere to specific security requirements. As such, the majority of schools appear to have outdated versions. This, along with the limited training and general lack of user-friendliness has contributed to the minimal use in the Western Cape.

Of those schools where RUBRICATE is being used, teachers only use it to develop assessment tools. In these schools, the benefits were limited at best, and many of the teachers did not report any noticeable changes in teaching and learning as a result of RUBRICATE.

The evaluators were unable to locate a single school in the Northwest which had used RUBRICATE. The district offices do not have complete records of the schools who received training, and it was clear that training was never rolled-out across the entire province.

There is very little information available, or existing knowledge within the provincial department regarding RUBRICATE in Mpumalanga. Based on an interview with a provincial official, however, the belief was that there is currently no use of RUBRICATE in the province.

Recommendations:

As a result of the above, it is recommended that the DBE should not invest in RUBRICATE. The potential gains (which for the most part there were no evidence to either substantiate or deny) would likely not justify the up-front licensing costs, as well as additional costs for upgrades, support, and training.

Lessons Learned:

A number of lessons were learned during the course of the evaluation which could apply to future interventions, particularly those with an ICT component. The lessons learned were grouped according to the following areas:

  • Project Management; the need for a systems-approach to broad scale interventions, location in the correct provincial and district departments, ongoing documentation and knowledge management, and constant follow-ups and monitoring,
  • Training; a well-orchestrated training model from top to bottom to ensure minimal to no loss of skills and knowledge when cascading; and
  • Software; the importance of user-friendly and reliable applications which require minimal additional work and continuous upgrades for use.


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
South Africa

Region:
ESAR

Type:
Evaluation

Theme:
Education

Language:
English

Partners:
teachers, HODs, principals, district officials, developers of RUBRICATE and provincial officials in Gauteng, Mpumalanga, Northwest, and Western Cape, as well National officials

Sequence #:
2016/012

New enhanced search