Evaluation database

Evaluation report

2011 South Africa: Evaluation of the Safe and Caring Child-Friendly Schools Programme



Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

 

Introduction

 

This ‘Evaluation of the Safe and Caring Child-Friendly Schools (SCCFS) Programme 2007-2010’ has been produced by the Centre

for Education Policy Development (CEPD) for the United Nations Children’s Fund (UNICEF) South Africa. The SCCFS Programme is a

UNICEF-supported intervention targeting South African public schools; the evaluation focused on four South African provinces in

which the Programme was implemented and in which 616 public schools participated. Although fieldwork commenced in June

2010, a teachers’ strike in the South African public school sector in the third quarter of 2010 led to the postponement of much of

the fieldwork to the first quarter of 2011. The evaluation was completed in June 2011. The findings of the evaluation may be of use

to the South African Government, in particular but not only the Department of Basic Education (DBE), and to UNICEF South Africa

(in relation to its bridge Country Programme for 2011 and its Country Programme for 2012-2014). The results of the evaluation

may also support the promotion of the Child-Friendly Schools (CFS) framework in other countries. The report is available in English

only.

Summaries of each chapter of the report are presented below. The following annexes to the report should also be noted, these

are available as a separate report:

 

««Annex 1: The terms of reference of the evaluation

««Annex 2: The research instruments

««Annex 3: The qualitative dataset

««Annex 4: Qualitative assessment of the sampled schools in terms of the six SCCFS Programme principles

««Annex 5: Sampled schools ranked by qualitative score.

««Annex 6: CFS cost projections scenario 01

««Annex 7: CFS cost projections scenario 02

Chapter 1: Introduction

Chapter 1 provides the background to and context of the evaluation. A review of relevant documentation identifies a high degree

of consistency between the UNICEF-supported SCCFS Programme in South Africa and CFS programmes in other countries. The

review also shows that CFS programmes have been effective globally and are supported by research into school effectiveness. The

core Programme framework suggests that a school should be:

 

««A rights-based and inclusive school

««An effective school

««A safe, protective and supportive school

««A health-promoting and health-seeking school

««A gender-sensitive and gender-promoting school

««A partnership-building school.

 

The review shows a high degree of synergy between the SCCFS Programme goals and South African Government strategies

in crime prevention, provision of health services and improvement in school education, all of which stress the importance of

community involvement, as does the SCCFS Programme. The review pays particular attention to the synergy that exists between

the Programme and the ‘Action Plan to 2014 Towards the Realisation of Schooling 2025’ of the South African Department of Basic

Education (DBE, 2010a). However, relevant cautions must be noted, related to serious barriers in the way of effective Programme

implementation, such as the lack of integration of functions within and across Government departments and the need to

strengthen schools’ internal capacity for teaching and learning, as well as management and leadership.

The review also shows, in the summaries of the baseline studies that were undertaken prior to Programme inception, that the

Programme was highly relevant in the selected schools because of the difficult socio-economic circumstances in which they

operate, and the resultant social problems they experience. Finally, the review has summarised the various strategies that were

adopted by the Programme service providers to address these challenges from 2007 to 2010.

 

Chapter 2: Purpose, objectives and scope of the evaluation

 

In Chapter 2 we note that the evaluation assesses the relevance, efficiency, effectiveness, impact, sustainability and replicability

of the SCCFS Programme. The purpose of the evaluation is to assess whether the implementation of the Programme and specific

interventions are aligned with Government’s plans and existing programmes, and to support the Government’s five-year plan

(2011-2015), UNICEF’s bridge Country Programme for 2011 and its Country Programme for 2012-2014. The results of the evaluation

may also support the promotion of the CFS framework in other countries. The specific objectives, scope, evaluation criteria and

intended outputs of the evaluation are presented in this chapter. Ethical issues are also addressed generally and with specific

reference to the participation of children in the research.

 

Chapter 3: Evaluation methodology

 

Chapter 3 describes the evaluation methodology in which the central feature was a case study approach that enabled systematic,

in-depth analysis of 37 schools, of which six were non-Programme schools to facilitate comparison with schools which received

little or no external support. Both qualitative and quantitative data were gathered. The quantitative approach provided a basis

for statistical analysis of school characteristics, while the qualitative approach provided deeper and more meaningful insights

into relevant school phenomena. Both approaches have helped to distinguish between the successes (and success factors) of

Programme and non-Programme schools in terms of the SCCFS principles. It is noted that a limitation of the evaluation methodology

is that the sample frame was relatively small, but that the small sample enabled a thorough, in-depth investigation of the selected

schools using a mixed-methods approach. A further limitation is that verification of Programme reach was not possible, but again,

the depth of interpretation achieved outweighs the disadvantages of lack of data on Programme reach. A final limitation is that

it was agreed by the Evaluation Steering Committee in the design stage that the evaluators would assume that Government

expenditure is relatively uniform in schools in the lower quintiles, and that we would therefore examine, in the financial analysis,

the potential value added by external partners rather than Government funds allocated to participating schools.

 

Chapter 4: Government, UNICEF and implementing partner respondents

 

Key Programme role players and stakeholders were interviewed, including personnel from the DBE, UNICEF and provincial

department of education personnel, as well as Programme implementing partners and district officials. Many of these key

respondents agreed that essentially the Programme framework is intended to support Government policy for school education,

and streamlines and integrates various aspects of policy. However, key difficulties were experienced during Programme

implementation in terms of the roles of Government: changes in personnel, especially senior managers, adversely affect the level

of Government participation; aspects of the Programme framework such as school safety were located in different directorates

in the various provincial education departments (PEDs), and were often regarded as an ‘add-on’ function; and there was variable

success at district level and officials with appropriate competence were not always assigned to the Programme. Variation in

Programme strategy was reported per province: provincial circumstances required different emphases, and the capacity and focus

of service providers differed in each province. It was widely felt that the Northern Cape PED had been the most committed to the

Programme. Nevertheless, most respondents reported pockets of success and an overall positive impact. The fact that the Care

and Support for Teaching and Learning (CSTL) Programme has incorporated the SCCFS Programme principles and goals shows

high awareness of the importance of the Programme at national level, and several respondents reported increased awareness at

provincial level also.

The challenges reported by the respondents include: fragmentation of functions in education departments; lack of buy-in by some

PEDs and many districts; weak school governing bodies (SGBs); weak school leadership in many schools; and lack of authentic

child participation. The respondents reported the following lessons learned: higher levels of ownership are needed by the PEDs

and districts; the Programme should become an integral part of the operations of PEDs and districts; increased resources are

needed at school level, including human resources, to improve the likelihood of Programme success; and incentives should be

contemplated for schools that perform well in terms of the Programme principles.

 

With regard to Programme sustainability, it was felt that: the capacity of PEDs and districts needs to be considerably enhanced,

and the Programme needs to be institutionalised in PEDs to ensure sustainability; district services need to be streamlined and

integrated; multi-level training is needed at provincial, district and school levels, the latter including SGB members; increased

school-based support is required, and this would enhance capacity for building relationships with the community; and where

school-based SCCFS committees have been established there is an increased likelihood of sustainability. Finally, it is important

to note again that the CSTL Programme has incorporated the SCCFS Programme principles and goals and is now the primary

mechanism for ensuring sustained SCCFS-related interventions.

 

Chapter 5: Presentation of the quantitative data

 

Chapter 5 presents the quantitative dataset obtained through questionnaires completed by 789 grade 11 learners at 21 secondary

and combined schools and 549 teachers at 37 schools (22 secondary, two combined and 13 primary) situated across four South

African provinces, including six control schools that did not participate in the Programme. The dataset shows that learners in

Programme schools have significantly more positive perceptions of their schools’ performance in terms of the Programme principles

than learners in non-Programme schools, and that ratings of the schools’ performance with respect to rights and inclusivity are

significantly higher at Programme schools than at non-Programme schools for both educators and learners.

 

Chapter 6: Presentation of the qualitative data

 

Chapter 6 presents relevant aspects of the qualitative dataset for selected schools: the six schools performing best in terms of

the Programme principles, the five worst-performing schools, and the seven schools where the Programme strategy was most

effective. Full descriptions of all 37 sample schools are presented in Annex 3.

 

Chapter 7: Findings

 

Chapter 7 presents the findings of the analysis of the quantitative and qualitative datasets. The quantitative analysis shows that the

views and perceptions of learners at high schools where the Programme was implemented are significantly more positive than

those of their peers at schools where the Programme was not implemented. This holds for all six facets of the SCCFS Programme.

Learners at SCCFS Programme high schools are therefore much more likely to believe that their rights are respected and that the

school is inclusive; that the school is effective in managing resources and supporting learners; that the school offers them a safe

environment for learning; that attention is given to the health of learners by the school; that boys and girls receive equal treatment;

and that the school is forging partnerships with its parent body and local community. Educators at SCCFS Programme schools

are significantly more likely to believe that their schools are inclusive of all types of learners and that learners’ rights are respected

and protected. In relation to the other five Programme principles, however, the views of educators about their schools do not

differ from those working in non-Programme schools. Overall, the SCCFS Programme therefore appears to have impacted on the

perceptions of learners to a far greater extent than on those of educators.

 

“... learners in Programme

schools have positive

perceptions of their

schools’ performance...”

 

The qualitative analysis shows that Programme relevance is high across all the sample schools. It also shows that Programme

strategies were highly variable both across provinces and across schools within a given province, ranging from intensive schoolbased

support to largely off-site training. In four of the sample schools the low-intensity, largely off-site Programme strategy has

been a reasonably effective (and in some cases very effective) source of support, and in three of the sample schools where the

high-intensity, on-site strategy was implemented the Programme has also been a reasonably effective (and in some cases very

effective) source of support.

 

Because of the variability in Programme strategies, and because the variability in strategy was not consciously aligned to the

management capability of the schools, Programme effectiveness and impact varied greatly and it was not possible to assess

Programme efficiency (addressed in Chapter 8). The features of high-performing Programme and non-Programme schools include:

a strong principal (and in some cases a strong school management team (SMT)) who interacts well, consults with stakeholders

(including learners) and delegates functions and tasks effectively; caring, committed teachers; learners who are involved in

decisions that affect them; high morale among learners and teachers; an active, supportive SGB that helps to develop good

relationships with the community; strong partnerships with non-governmental organisations (NGOs) and Government service

providers, including police who adopt a developmental approach to their relationship with the school; partnerships that are

aligned to ensure efficient support; buildings and grounds that are safe, secure and welcoming; close supervision of learners

during break; and willing and able district and PED officials.

 

The features of low-performing Programme and non-Programme schools include: the presence of an unassertive principal, or a

principal who does not believe in certain of the learners’ rights, or is unable to report how many learners are socially vulnerable,

or does not know what the Programme (or the Programme coordinator) is doing at the school, or makes decisions without

consulting stakeholders; teachers who are not punctual or are not committed to teaching and learning, do not supervise learners

during break, and send learners on errands to sites that are dangerous; teachers who reportedly have sexual relationships with girl

learners; school staff who blame the community and learners for the school’s troubles; unsafe, permeable school grounds and a

laissez-faire approach to learners leaving and local residents entering the grounds; weak partnerships with NGOs and Government

service providers; police who respond violently when called into the school; district officials who stress bureaucratic requirements

in their relationship with the school rather than the Programme principles; and poor maintenance and cleaning of school buildings

and grounds, which are not welcoming.

 

The key factors in terms of Programme success are: high-intensity support in the form of an active, committed and empathetic

Programme facilitator on-site in schools where the staff are willing, but struggling to improve the school environment; and, in

schools that have many of the features listed above as effective (particularly, strong management), low-intensity support that

provides a framework that the school can use to improve its performance.

 

Important criticisms of the Programme (even in cases where it had been very successful) are: lack of attunement of the Programme

intervention to the most pressing needs of the school; insufficient buy-in among stakeholders; training of selected teachers

without high levels of Programme awareness among the rest of the staff; lack of training of the SGB; insufficient on-site support;

no provision of key resources, such as fencing to protect the school and learning resources; lack of attention to improved teaching

and learning, with a frequent emphasis on safety at the expense of other Programme principles.

 

The key factors in terms of Programme sustainability are: a strong, committed principal or SMT; staff who are committed to the

Programme principles; the creation of relevant school structures (such as health committees); the incorporation of the Programme

principles in school policies; the recruitment of a welfare assistant to sustain some of the Programme activities; PED and district

officials who are committed to the Programme, are aware of where responsibility for the Programme is located and have sufficient

time to monitor and support Programme activities; and post-Programme follow-up to ascertain schools’ ongoing needs. These

factors are not present in the majority of the sample schools.

 

The SCCFS Programme is clearly replicable, but should be replicated with modifications that are discussed in Chapters 8 to 11.

Replication is advisable, since the documentary review presented in Chapter 1 shows that the Programme is supportive of South

African Government strategies in crime prevention, provision of health services and improvement in school education. The review

shows a particularly strong synergy between the Programme and the ‘Action Plan to 2014 Towards the Realisation of Schooling

2025’ of the South African Department of Basic Education (DBE, 2010a).

 

Chapter 8: Financial analysis

 

Chapter 8 presents the financial allocations per service provider, the breakdowns of service providers’ expenditure and a comparison

of expenditure by category across service providers over the Programme period. The chapter, read together with the Chapter 7

findings, shows that the Programme is replicable but that modifications in strategy are needed. Two distinct models of Programme

delivery were found to be effective in seven schools: the first is a high-intensity model that provides for on-site support; the

second is a low-intensity model that focuses on off-site training with little on-site support. Both models cost approximately the

same. Three hybrid scenarios that combine the best features of the two implementation strategies are presented and costed,

with adjustments such as increased expenditure on project management and monitoring and evaluation to provide for a more

centralised approach to both functions and reduce the potential for unintended variation in the Programme strategy. The three

hybrid models all contain provision for similar levels of off-site training and for different degrees of on-site support, and range in

cost from R166 million to R310 million to cover 2,000 schools over a three-year period. An ‘exit strategy’ scenario is also presented

in which the level of intensity of on-site support decreases each year over three years in 2,000 schools. The total estimated cost of

the ‘exit strategy’ scenario is R227 million.

 

There is no clear evidence that any given Programme strategy was efficient, for the following reasons:

With the exception of the Northern Cape, «« respondents in the sample schools reported a high degree of inconsistency

in Programme strategy (see the Chapter 7 findings). The most extreme example of variation is found in the Eastern

Cape, where in some schools a full-time coordinator was deployed. However, since expenditure per school is not

available, it is not possible to assess Programme efficiency. Conversely, in the Northern Cape, where a much more

consistent strategy was implemented by the service provider, Programme success varied depending on the quality of

school management (again, see Chapter 7).

 

«« The above suggests, paradoxically, that an efficient approach to Programme implementation would be to ensure that

variation in Programme strategy is deliberate, not unintended, and that the strategy should vary from high-intensity

approaches in schools with weak management and/or dire socio-economic conditions to low-intensity approaches in

schools where management is strong. For this reason, three hybrid models of Programme implementation have been

presented and costed in this chapter.

 

Chapter 9: Conclusions

 

The Programme framework, as noted by key respondents (see Chapter 4), is effectively a user-friendly tool for implementing

what is already contained in Government policy and plans for more effective schooling. Many school-based respondents noted

that the Programme principles represent a simple approach to the plethora of policies and circulars that they have to respond

to; indeed, the simplicity of the Programme framework is one of its greatest virtues. We can also conclude that, while simple, it is

also a multi-faceted and very comprehensive framework for school improvement; in several schools, where strong management

and leadership was in place, merely being introduced to the Programme principles prompted school managers and teachers to

undertake very successful transformation processes in their schools in collaboration with external stakeholders.

It must be noted that the Programme was implemented in schools that operate in extremely difficult socio-economic circumstances,

and that its relevance in all of the sample schools is indisputable. All of the schools have a history of severe challenges, ranging

from extreme poverty to drug abuse and violence. The environment in which the Programme was implemented was therefore

difficult at best, but the difficulties were compounded in poorly managed schools, where much higher levels of on-site support

are needed. Despite the challenges, two clearly distinct Programme strategies were found to be effective or very effective in seven

of the sample schools, one weighted towards on-site support and the other towards off-site training with little on-site support.

Since the latter strategy was effective only in well managed schools, we conclude that greater attention needs to be paid in the

preparation phase to assessing the state of school management and viewing this as an important driver in the choice of strategy.

We also note that greater attention is needed to identifying the specific challenges that particular schools are facing.

 

We conclude that the Programme strategies were highly variable, in some cases even across schools within a particular province.

As discussed in Chapter 8, this made an assessment of Programme efficiency impossible. It should be noted that monitoring and

evaluation was also highly variable across provinces and across schools within provinces. Many schools reported infrequent visits

to monitor the effectiveness of Programme implementation, with the exception of the Northern Cape, where regular school visits

were effected to monitor what the service provider called the Programme ‘barometer’. A related defect is that the baseline studies

were very inconsistent in their design, as is evident in the documentary review presented in Chapter 1; moreover, data per school

are not contained in the baseline reports, which makes measurement of progress per school impossible.

 

An important conclusion derived from the quantitative research is that the impact of the Programme, particularly in terms of the

perceptions of learners, was significantly greater in Programme than in non-Programme schools, which indicates an important

success in terms of the creation of child-friendly school environments. However, we have noted that, when examined from a

qualitative comparative perspective, the impact of the Programme was highly variable and we conclude that this is because of

the variability in Programme strategies.

We also conclude that much more attention needs to be paid in the planning phase to sustainability and follow-up activities.

Programme sustainability seemed guaranteed in only a small number of the sample schools, where there was strong management

and where Programme-related activities had become a routine function. As key respondents noted with regard to sustainability

(see Chapter 4): improved government capacity is needed; district services need to be streamlined and integrated; increased

ownership of the Programme by Government is essential and a longer preparatory phase is advisable to provide for this; multilevel

training is advisable for all role players and stakeholders; and, ultimately, the responsibility for sustaining the Programme rests

with Government, not service providers. The Programme needs to be institutionalised as a Government responsibility, especially

since a key finding noted in Chapter 7 is that it is closely aligned to Government plans and provides a robust framework and

platform for improving school effectiveness.

 

The evaluation findings (Chapter 7), together with the analysis of costs and efficiency (Chapter 8), show that the Programme is

replicable and that a range of hybrid models (combining off-site training with different degrees of on-site support) is advisable

and a worthwhile investment (approximately R227 million over three years, covering 2,000 schools). Replicability is subject to

important modifications to the Programme strategy. These modifications include: combining high- and low-intensity strategies

that entail centralised training coupled with a strong on-site presence in each school where management is below what we have

referred to as a ‘threshold of functionality’; adopting a more centralised and standardised approach to project management and

monitoring and evaluation; aligning Programme interventions more closely with schools’ most pressing needs; and strengthening

project planning and preparation to ensure that the sustainability factors described in Chapter 7 are present.

 

Finally, we note that there is a particularly strong synergy between the Programme and the ‘Action Plan to 2014 Towards the

Realisation of Schooling 2025’ of the South African Department of Basic Education (DBE, 2010a) . The fact that the CSTL Programme

of the DBE has incorporated the SCCFS Programme principles and goals is an extremely important development that will support

Programme sustainability and replicability. However, this evaluation shows that Government capacity at provincial and district level

needs to be improved and school support services streamlined and integrated if the Programme is to be replicated effectively.

Moreover, two relevant cautions contained in the Chapter 1 document review must be noted: first, there are serious structural and

process-related barriers in the way of effective Programme implementation, such as the lack of integration of functions within and

across Government departments, the presence of excessively bureaucratic requirements that hinder effective delivery and weak

monitoring and evaluation systems; and, second, serious attention is needed to schools’ internal capacity for teaching and learning

and management and leadership. Since these cautions are expressed in South African studies, they must be taken very seriously

as the Programme is replicated.

 

Establishment by law

and understanding

of the mandate

 

Chapter 10: Recommendations

 

Recommendations are presented in Chapter 10 related to:

 

«« The need for greater intensity in school-based support

«« The potential for low-intensity support in well managed schools

«« The need for a standardised model of delivery

«« The need for more centralised programme management

«« The need for research into the school management ‘threshold of functionality’

«« The advisability of attuning support to schools’ needs

«« The need for greater effort to ensure stakeholder buy-in

«« The need for more intensive training

««A more intensive focus on teaching and learning

«« Enhanced monitoring and evaluation and consistency in baseline studies

«« The need for post-programme follow-up activities

«« The importance of PED and district support

«« The advisability of Government investment in school safety

«« The need to ensure continued support for schools

«« Future investment in SCCFS

«« Programme replication.

 

For each recommendation, actors and required actions have been identified.

 

Chapter 11: Lessons learned

The broad lessons presented in Chapter 11 are related to:

«« Variability in programme strategy

«« The need to strengthen monitoring and evaluation

«« The need to manage the expectations of role players and stakeholders

«« The management capacity of schools

«« The need for future ‘hybrid’ programme strategies

«« The need to monitor expenditure per school

«« Programme design and evaluation

«« The weighting of programme principles

«« The need to respond to schools’ most pressing needs

«« Programme sustainability.

 

The lessons learned from the evaluation are related in Chapter 11 to the programme planning and implementation domains in

which they should be applied. Notes are provided to guide future actions.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search