We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Uganda: UNICEF – Ugandan Ministry of Education and Sports BRMS Mentorship Project Evaluation



Author: Martin Prew

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

UNICEF Uganda, in partnership with the Ministry of Education and Sports (MoES), has been implementing an innovative model of the Coordinating Centre Tutor (CCT) support based on a mentorship model. It is designed to support the Ugandan government’s 13 Basic Requirements Minimum Standards (BRMS) implementation, which are designed to increase child participation and survival rates in primary schools and improve the quality of primary education through innovative projects. The success of BRMS depends on the capacity of the primary school system to implement it and improve. This means that the focus is on the four support systems impacting on primary schools. These are:

1. The district education office, which mainly acts as a coordination centre and liaises between education and local government in the district;
2. The 23 colleges of education which have outreach capacity provided through the CCTs who are based in the colleges but work in schools;
3. The Directorate for Education Standards (DES) which inspects schools and (at least in theory) advises on improvements to the quality of the learning experience. Their inspectors are based national levels to implement this mandate. 
4. District education authorities who also conduct school inspections at municipal and district level through the District Inspectors of Schools (DIS) and the Municipal Inspectors of Schools (MIS).

UNICEF has focused in practice on the second of these levels – the Primary Teachers' College (PTCs) with their cohort of CCTs. This was a logical decision as the over 500 CCTs are a critical human resource, who have the mandate to work with school managers, governors and staff on improving the school. However, while the appointment of CCTs in the late 1990s was an innovative attempt to increase school effectiveness, lack of funding and mobility has greatly reduced their impact.

Purpose/Objective:

The formative evaluation of the UNICEF Uganda – Ministry of Education and Sport Project known as ‘the Basic Requirements and Minimum Standards (BRMS) Mentorship Project’ was commissioned by UNICEF Uganda to achieve a number of outcomes. The desired outcomes included:

• Providing a formative evaluation half way through the delivery period of the project;
• Providing a more summative evaluation of the work of the mentors based in the Core Primary Teachers’ Colleges (CPTCs);
• Providing baseline metrics for the future measurement of the project, as no specific project baseline survey was conducted at the start of the project ; and
• Assessing the methods and extent to which the impact of the project can be sustained and replicated.

The outcomes expected from the BRMS mentorship programme in summary can be expressed as:
1. CCTs to improve their engagement with schools to provide professional development and support (rather than acting as materials distributors and inspectors);
2. Improvements in teacher performance and community engagement with the school which in the long term leads to improved completion rates and exam results;
3. Improvements in coordination of district education partners (district office, inspector, school, teacher training college).

Methodology:

This evaluation is designed to both provide a formative evaluation of the Project, a provisional summative evaluation of the work of the mentors on the project and a basis for setting metrics and evaluation process for future evaluations (in the absence of a baseline survey when the project started).
In the process of undertaking this review the evaluator:

• Reviewed the Theory of Change and the assumptions that underpin it
• Reviewed the programme design and implementation process against the original objectives
• Developed some draft metrics for interim outcomes and measured them
• Reviewed the programme plan and budget to indicate cost of delivery.

The evaluation was conducted in early 2014. The approach included a review and analysis of the available documentation, observations and interviews in the field, and the collection of data to inform a set of metrics.
While the evaluator is interested in failure and challenges faced by the mentors and the colleges in implementing the programme, the main focus was on conditions where the project has succeeded and achieved real impact. This allows for the description of a proposed model in section 9 of this report. However, this positive approach does not mean that problems faced in the planning and implementation of the project have been ignored as this would serve no purpose and potentially lead to poor value for money in replicating the project elsewhere.

Findings and Conclusions:

The overall finding of the report is that the BRMS Mentorship Project was well conceived and, given the constraints affecting it, had significant impact and serves as a potential model for sustained school improvement.

This can be seen in the findings below:

• 6.9 percentage point increase in P3 NAPE literacy results in the 73 target districts by 2012
• 6.7 percentage point increase in P3 NAPE numeracy results in the 73 target districts by 2012
• Most target districts saw increases in Division 1 and 2 PLE results
• 3 percentage point (4.6%) increase in primary school completion rates in the 73 target districts
• 313 CCTs were able to provide improved support to over 2000 schools and 20,000 educators
• 100% of sampled focus schools indicate ‘better’ or a ‘lot better’ support from their CCTs
• 100% of CCTs report improved ability to support schools as a result of the project
• 70% of head-teachers in sampled schools have improved practice according to their staff
• 93% of sampled schools ‘happy’ with support they gets from their CCT; 0% not happy
• 94% of CCTs believe relations between CCTs and between CCTs and their DPO have improved

Recommendations:

The report recommends that UNICEF in partnership with the MoES implements a two strand process over the remaining two years of the project. This is detailed in Section 11 of the full report. It involves support of the intervention and impact to date in most of the colleges through a low-cost peripatetic support process, while implementing a much more structured intervention in the remaining 8 core PTCs based on the strengths of the model. The aim of the second strand would be to extend and explore the model further and document it with rigour from the start so that there is reliable data to show impact on schools, the CCTs and the CPTCs. This documentation is critical for future advocacy and replication of the model.

Lessons Learned:

The report concludes that from the intervention a model of successful school improvement based on colleges of education and CCTs can be defined. While this is a relatively expensive model it is judged to be relatively cost effective given the impact it has had and can potentially have.

However, the report also finds that:

• UNICEF and the MoES could have played a more structured and coordinated role in planning and managing the project and its finances with positive effect on the project’s coherence and impact. Had the project had a strong shared project plan supported by a specific project baseline survey and a mid-term process evaluation, with clear project targets which were shared by all stakeholders, the project would have achieved even more impact. 
• The lack of baseline data specific to the focus schools and CPTCs means that it is almost impossible to measure the impact of the project on these schools and the CPTCs.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Uganda

Region:
ESARO

Theme:
Education

Type:
Evaluation

Language:
English

New enhanced search