“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
Introduction
This ‘Evaluation of the Safe and Caring Child-Friendly Schools (SCCFS) Programme 2007-2010’ has been produced by the Centre
for Education Policy Development (CEPD) for the United Nations Children’s Fund (UNICEF) South Africa. The SCCFS Programme is a
UNICEF-supported intervention targeting South African public schools; the evaluation focused on four South African provinces in
which the Programme was implemented and in which 616 public schools participated. Although fieldwork commenced in June
2010, a teachers’ strike in the South African public school sector in the third quarter of 2010 led to the postponement of much of
the fieldwork to the first quarter of 2011. The evaluation was completed in June 2011. The findings of the evaluation may be of use
to the South African Government, in particular but not only the Department of Basic Education (DBE), and to UNICEF South Africa
(in relation to its bridge Country Programme for 2011 and its Country Programme for 2012-2014). The results of the evaluation
may also support the promotion of the Child-Friendly Schools (CFS) framework in other countries. The report is available in English
only.
Summaries of each chapter of the report are presented below. The following annexes to the report should also be noted, these
are available as a separate report:
««Annex 1: The terms of reference of the evaluation
««Annex 2: The research instruments
««Annex 3: The qualitative dataset
««Annex 4: Qualitative assessment of the sampled schools in terms of the six SCCFS Programme principles
««Annex 5: Sampled schools ranked by qualitative score.
««Annex 6: CFS cost projections scenario 01
««Annex 7: CFS cost projections scenario 02
Chapter 1: Introduction
Chapter 1 provides the background to and context of the evaluation. A review of relevant documentation identifies a high degree
of consistency between the UNICEF-supported SCCFS Programme in South Africa and CFS programmes in other countries. The
review also shows that CFS programmes have been effective globally and are supported by research into school effectiveness. The
core Programme framework suggests that a school should be:
««A rights-based and inclusive school
««An effective school
««A safe, protective and supportive school
««A health-promoting and health-seeking school
««A gender-sensitive and gender-promoting school
««A partnership-building school.
The review shows a high degree of synergy between the SCCFS Programme goals and South African Government strategies
in crime prevention, provision of health services and improvement in school education, all of which stress the importance of
community involvement, as does the SCCFS Programme. The review pays particular attention to the synergy that exists between
the Programme and the ‘Action Plan to 2014 Towards the Realisation of Schooling 2025’ of the South African Department of Basic
Education (DBE, 2010a). However, relevant cautions must be noted, related to serious barriers in the way of effective Programme
implementation, such as the lack of integration of functions within and across Government departments and the need to
strengthen schools’ internal capacity for teaching and learning, as well as management and leadership.
The review also shows, in the summaries of the baseline studies that were undertaken prior to Programme inception, that the
Programme was highly relevant in the selected schools because of the difficult socio-economic circumstances in which they
operate, and the resultant social problems they experience. Finally, the review has summarised the various strategies that were
adopted by the Programme service providers to address these challenges from 2007 to 2010.
Chapter 2: Purpose, objectives and scope of the evaluation
In Chapter 2 we note that the evaluation assesses the relevance, efficiency, effectiveness, impact, sustainability and replicability
of the SCCFS Programme. The purpose of the evaluation is to assess whether the implementation of the Programme and specific
interventions are aligned with Government’s plans and existing programmes, and to support the Government’s five-year plan
(2011-2015), UNICEF’s bridge Country Programme for 2011 and its Country Programme for 2012-2014. The results of the evaluation
may also support the promotion of the CFS framework in other countries. The specific objectives, scope, evaluation criteria and
intended outputs of the evaluation are presented in this chapter. Ethical issues are also addressed generally and with specific
reference to the participation of children in the research.
Chapter 3: Evaluation methodology
Chapter 3 describes the evaluation methodology in which the central feature was a case study approach that enabled systematic,
in-depth analysis of 37 schools, of which six were non-Programme schools to facilitate comparison with schools which received
little or no external support. Both qualitative and quantitative data were gathered. The quantitative approach provided a basis
for statistical analysis of school characteristics, while the qualitative approach provided deeper and more meaningful insights
into relevant school phenomena. Both approaches have helped to distinguish between the successes (and success factors) of
Programme and non-Programme schools in terms of the SCCFS principles. It is noted that a limitation of the evaluation methodology
is that the sample frame was relatively small, but that the small sample enabled a thorough, in-depth investigation of the selected
schools using a mixed-methods approach. A further limitation is that verification of Programme reach was not possible, but again,
the depth of interpretation achieved outweighs the disadvantages of lack of data on Programme reach. A final limitation is that
it was agreed by the Evaluation Steering Committee in the design stage that the evaluators would assume that Government
expenditure is relatively uniform in schools in the lower quintiles, and that we would therefore examine, in the financial analysis,
the potential value added by external partners rather than Government funds allocated to participating schools.
Chapter 4: Government, UNICEF and implementing partner respondents
Key Programme role players and stakeholders were interviewed, including personnel from the DBE, UNICEF and provincial
department of education personnel, as well as Programme implementing partners and district officials. Many of these key
respondents agreed that essentially the Programme framework is intended to support Government policy for school education,
and streamlines and integrates various aspects of policy. However, key difficulties were experienced during Programme
implementation in terms of the roles of Government: changes in personnel, especially senior managers, adversely affect the level
of Government participation; aspects of the Programme framework such as school safety were located in different directorates
in the various provincial education departments (PEDs), and were often regarded as an ‘add-on’ function; and there was variable
success at district level and officials with appropriate competence were not always assigned to the Programme. Variation in
Programme strategy was reported per province: provincial circumstances required different emphases, and the capacity and focus
of service providers differed in each province. It was widely felt that the Northern Cape PED had been the most committed to the
Programme. Nevertheless, most respondents reported pockets of success and an overall positive impact. The fact that the Care
and Support for Teaching and Learning (CSTL) Programme has incorporated the SCCFS Programme principles and goals shows
high awareness of the importance of the Programme at national level, and several respondents reported increased awareness at
provincial level also.
The challenges reported by the respondents include: fragmentation of functions in education departments; lack of buy-in by some
PEDs and many districts; weak school governing bodies (SGBs); weak school leadership in many schools; and lack of authentic
child participation. The respondents reported the following lessons learned: higher levels of ownership are needed by the PEDs
and districts; the Programme should become an integral part of the operations of PEDs and districts; increased resources are
needed at school level, including human resources, to improve the likelihood of Programme success; and incentives should be
contemplated for schools that perform well in terms of the Programme principles.
With regard to Programme sustainability, it was felt that: the capacity of PEDs and districts needs to be considerably enhanced,
and the Programme needs to be institutionalised in PEDs to ensure sustainability; district services need to be streamlined and
integrated; multi-level training is needed at provincial, district and school levels, the latter including SGB members; increased
school-based support is required, and this would enhance capacity for building relationships with the community; and where
school-based SCCFS committees have been established there is an increased likelihood of sustainability. Finally, it is important
to note again that the CSTL Programme has incorporated the SCCFS Programme principles and goals and is now the primary
mechanism for ensuring sustained SCCFS-related interventions.
Chapter 5: Presentation of the quantitative data
Chapter 5 presents the quantitative dataset obtained through questionnaires completed by 789 grade 11 learners at 21 secondary
and combined schools and 549 teachers at 37 schools (22 secondary, two combined and 13 primary) situated across four South
African provinces, including six control schools that did not participate in the Programme. The dataset shows that learners in
Programme schools have significantly more positive perceptions of their schools’ performance in terms of the Programme principles
than learners in non-Programme schools, and that ratings of the schools’ performance with respect to rights and inclusivity are
significantly higher at Programme schools than at non-Programme schools for both educators and learners.
Chapter 6: Presentation of the qualitative data
Chapter 6 presents relevant aspects of the qualitative dataset for selected schools: the six schools performing best in terms of
the Programme principles, the five worst-performing schools, and the seven schools where the Programme strategy was most
effective. Full descriptions of all 37 sample schools are presented in Annex 3.
Chapter 7: Findings
Chapter 7 presents the findings of the analysis of the quantitative and qualitative datasets. The quantitative analysis shows that the
views and perceptions of learners at high schools where the Programme was implemented are significantly more positive than
those of their peers at schools where the Programme was not implemented. This holds for all six facets of the SCCFS Programme.
Learners at SCCFS Programme high schools are therefore much more likely to believe that their rights are respected and that the
school is inclusive; that the school is effective in managing resources and supporting learners; that the school offers them a safe
environment for learning; that attention is given to the health of learners by the school; that boys and girls receive equal treatment;
and that the school is forging partnerships with its parent body and local community. Educators at SCCFS Programme schools
are significantly more likely to believe that their schools are inclusive of all types of learners and that learners’ rights are respected
and protected. In relation to the other five Programme principles, however, the views of educators about their schools do not
differ from those working in non-Programme schools. Overall, the SCCFS Programme therefore appears to have impacted on the
perceptions of learners to a far greater extent than on those of educators.
“... learners in Programme
schools have positive
perceptions of their
schools’ performance...”
The qualitative analysis shows that Programme relevance is high across all the sample schools. It also shows that Programme
strategies were highly variable both across provinces and across schools within a given province, ranging from intensive schoolbased
support to largely off-site training. In four of the sample schools the low-intensity, largely off-site Programme strategy has
been a reasonably effective (and in some cases very effective) source of support, and in three of the sample schools where the
high-intensity, on-site strategy was implemented the Programme has also been a reasonably effective (and in some cases very
effective) source of support.
Because of the variability in Programme strategies, and because the variability in strategy was not consciously aligned to the
management capability of the schools, Programme effectiveness and impact varied greatly and it was not possible to assess
Programme efficiency (addressed in Chapter 8). The features of high-performing Programme and non-Programme schools include:
a strong principal (and in some cases a strong school management team (SMT)) who interacts well, consults with stakeholders
(including learners) and delegates functions and tasks effectively; caring, committed teachers; learners who are involved in
decisions that affect them; high morale among learners and teachers; an active, supportive SGB that helps to develop good
relationships with the community; strong partnerships with non-governmental organisations (NGOs) and Government service
providers, including police who adopt a developmental approach to their relationship with the school; partnerships that are
aligned to ensure efficient support; buildings and grounds that are safe, secure and welcoming; close supervision of learners
during break; and willing and able district and PED officials.
The features of low-performing Programme and non-Programme schools include: the presence of an unassertive principal, or a
principal who does not believe in certain of the learners’ rights, or is unable to report how many learners are socially vulnerable,
or does not know what the Programme (or the Programme coordinator) is doing at the school, or makes decisions without
consulting stakeholders; teachers who are not punctual or are not committed to teaching and learning, do not supervise learners
during break, and send learners on errands to sites that are dangerous; teachers who reportedly have sexual relationships with girl
learners; school staff who blame the community and learners for the school’s troubles; unsafe, permeable school grounds and a
laissez-faire approach to learners leaving and local residents entering the grounds; weak partnerships with NGOs and Government
service providers; police who respond violently when called into the school; district officials who stress bureaucratic requirements
in their relationship with the school rather than the Programme principles; and poor maintenance and cleaning of school buildings
and grounds, which are not welcoming.
The key factors in terms of Programme success are: high-intensity support in the form of an active, committed and empathetic
Programme facilitator on-site in schools where the staff are willing, but struggling to improve the school environment; and, in
schools that have many of the features listed above as effective (particularly, strong management), low-intensity support that
provides a framework that the school can use to improve its performance.
Important criticisms of the Programme (even in cases where it had been very successful) are: lack of attunement of the Programme
intervention to the most pressing needs of the school; insufficient buy-in among stakeholders; training of selected teachers
without high levels of Programme awareness among the rest of the staff; lack of training of the SGB; insufficient on-site support;
no provision of key resources, such as fencing to protect the school and learning resources; lack of attention to improved teaching
and learning, with a frequent emphasis on safety at the expense of other Programme principles.
The key factors in terms of Programme sustainability are: a strong, committed principal or SMT; staff who are committed to the
Programme principles; the creation of relevant school structures (such as health committees); the incorporation of the Programme
principles in school policies; the recruitment of a welfare assistant to sustain some of the Programme activities; PED and district
officials who are committed to the Programme, are aware of where responsibility for the Programme is located and have sufficient
time to monitor and support Programme activities; and post-Programme follow-up to ascertain schools’ ongoing needs. These
factors are not present in the majority of the sample schools.
The SCCFS Programme is clearly replicable, but should be replicated with modifications that are discussed in Chapters 8 to 11.
Replication is advisable, since the documentary review presented in Chapter 1 shows that the Programme is supportive of South
African Government strategies in crime prevention, provision of health services and improvement in school education. The review
shows a particularly strong synergy between the Programme and the ‘Action Plan to 2014 Towards the Realisation of Schooling
2025’ of the South African Department of Basic Education (DBE, 2010a).
Chapter 8: Financial analysis
Chapter 8 presents the financial allocations per service provider, the breakdowns of service providers’ expenditure and a comparison
of expenditure by category across service providers over the Programme period. The chapter, read together with the Chapter 7
findings, shows that the Programme is replicable but that modifications in strategy are needed. Two distinct models of Programme
delivery were found to be effective in seven schools: the first is a high-intensity model that provides for on-site support; the
second is a low-intensity model that focuses on off-site training with little on-site support. Both models cost approximately the
same. Three hybrid scenarios that combine the best features of the two implementation strategies are presented and costed,
with adjustments such as increased expenditure on project management and monitoring and evaluation to provide for a more
centralised approach to both functions and reduce the potential for unintended variation in the Programme strategy. The three
hybrid models all contain provision for similar levels of off-site training and for different degrees of on-site support, and range in
cost from R166 million to R310 million to cover 2,000 schools over a three-year period. An ‘exit strategy’ scenario is also presented
in which the level of intensity of on-site support decreases each year over three years in 2,000 schools. The total estimated cost of
the ‘exit strategy’ scenario is R227 million.
There is no clear evidence that any given Programme strategy was efficient, for the following reasons:
With the exception of the Northern Cape, «« respondents in the sample schools reported a high degree of inconsistency
in Programme strategy (see the Chapter 7 findings). The most extreme example of variation is found in the Eastern
Cape, where in some schools a full-time coordinator was deployed. However, since expenditure per school is not
available, it is not possible to assess Programme efficiency. Conversely, in the Northern Cape, where a much more
consistent strategy was implemented by the service provider, Programme success varied depending on the quality of
school management (again, see Chapter 7).
«« The above suggests, paradoxically, that an efficient approach to Programme implementation would be to ensure that
variation in Programme strategy is deliberate, not unintended, and that the strategy should vary from high-intensity
approaches in schools with weak management and/or dire socio-economic conditions to low-intensity approaches in
schools where management is strong. For this reason, three hybrid models of Programme implementation have been
presented and costed in this chapter.
Chapter 9: Conclusions
The Programme framework, as noted by key respondents (see Chapter 4), is effectively a user-friendly tool for implementing
what is already contained in Government policy and plans for more effective schooling. Many school-based respondents noted
that the Programme principles represent a simple approach to the plethora of policies and circulars that they have to respond
to; indeed, the simplicity of the Programme framework is one of its greatest virtues. We can also conclude that, while simple, it is
also a multi-faceted and very comprehensive framework for school improvement; in several schools, where strong management
and leadership was in place, merely being introduced to the Programme principles prompted school managers and teachers to
undertake very successful transformation processes in their schools in collaboration with external stakeholders.
It must be noted that the Programme was implemented in schools that operate in extremely difficult socio-economic circumstances,
and that its relevance in all of the sample schools is indisputable. All of the schools have a history of severe challenges, ranging
from extreme poverty to drug abuse and violence. The environment in which the Programme was implemented was therefore
difficult at best, but the difficulties were compounded in poorly managed schools, where much higher levels of on-site support
are needed. Despite the challenges, two clearly distinct Programme strategies were found to be effective or very effective in seven
of the sample schools, one weighted towards on-site support and the other towards off-site training with little on-site support.
Since the latter strategy was effective only in well managed schools, we conclude that greater attention needs to be paid in the
preparation phase to assessing the state of school management and viewing this as an important driver in the choice of strategy.
We also note that greater attention is needed to identifying the specific challenges that particular schools are facing.
We conclude that the Programme strategies were highly variable, in some cases even across schools within a particular province.
As discussed in Chapter 8, this made an assessment of Programme efficiency impossible. It should be noted that monitoring and
evaluation was also highly variable across provinces and across schools within provinces. Many schools reported infrequent visits
to monitor the effectiveness of Programme implementation, with the exception of the Northern Cape, where regular school visits
were effected to monitor what the service provider called the Programme ‘barometer’. A related defect is that the baseline studies
were very inconsistent in their design, as is evident in the documentary review presented in Chapter 1; moreover, data per school
are not contained in the baseline reports, which makes measurement of progress per school impossible.
An important conclusion derived from the quantitative research is that the impact of the Programme, particularly in terms of the
perceptions of learners, was significantly greater in Programme than in non-Programme schools, which indicates an important
success in terms of the creation of child-friendly school environments. However, we have noted that, when examined from a
qualitative comparative perspective, the impact of the Programme was highly variable and we conclude that this is because of
the variability in Programme strategies.
We also conclude that much more attention needs to be paid in the planning phase to sustainability and follow-up activities.
Programme sustainability seemed guaranteed in only a small number of the sample schools, where there was strong management
and where Programme-related activities had become a routine function. As key respondents noted with regard to sustainability
(see Chapter 4): improved government capacity is needed; district services need to be streamlined and integrated; increased
ownership of the Programme by Government is essential and a longer preparatory phase is advisable to provide for this; multilevel
training is advisable for all role players and stakeholders; and, ultimately, the responsibility for sustaining the Programme rests
with Government, not service providers. The Programme needs to be institutionalised as a Government responsibility, especially
since a key finding noted in Chapter 7 is that it is closely aligned to Government plans and provides a robust framework and
platform for improving school effectiveness.
The evaluation findings (Chapter 7), together with the analysis of costs and efficiency (Chapter 8), show that the Programme is
replicable and that a range of hybrid models (combining off-site training with different degrees of on-site support) is advisable
and a worthwhile investment (approximately R227 million over three years, covering 2,000 schools). Replicability is subject to
important modifications to the Programme strategy. These modifications include: combining high- and low-intensity strategies
that entail centralised training coupled with a strong on-site presence in each school where management is below what we have
referred to as a ‘threshold of functionality’; adopting a more centralised and standardised approach to project management and
monitoring and evaluation; aligning Programme interventions more closely with schools’ most pressing needs; and strengthening
project planning and preparation to ensure that the sustainability factors described in Chapter 7 are present.
Finally, we note that there is a particularly strong synergy between the Programme and the ‘Action Plan to 2014 Towards the
Realisation of Schooling 2025’ of the South African Department of Basic Education (DBE, 2010a) . The fact that the CSTL Programme
of the DBE has incorporated the SCCFS Programme principles and goals is an extremely important development that will support
Programme sustainability and replicability. However, this evaluation shows that Government capacity at provincial and district level
needs to be improved and school support services streamlined and integrated if the Programme is to be replicated effectively.
Moreover, two relevant cautions contained in the Chapter 1 document review must be noted: first, there are serious structural and
process-related barriers in the way of effective Programme implementation, such as the lack of integration of functions within and
across Government departments, the presence of excessively bureaucratic requirements that hinder effective delivery and weak
monitoring and evaluation systems; and, second, serious attention is needed to schools’ internal capacity for teaching and learning
and management and leadership. Since these cautions are expressed in South African studies, they must be taken very seriously
as the Programme is replicated.
Establishment by law
and understanding
of the mandate
Chapter 10: Recommendations
Recommendations are presented in Chapter 10 related to:
«« The need for greater intensity in school-based support
«« The potential for low-intensity support in well managed schools
«« The need for a standardised model of delivery
«« The need for more centralised programme management
«« The need for research into the school management ‘threshold of functionality’
«« The advisability of attuning support to schools’ needs
«« The need for greater effort to ensure stakeholder buy-in
«« The need for more intensive training
««A more intensive focus on teaching and learning
«« Enhanced monitoring and evaluation and consistency in baseline studies
«« The need for post-programme follow-up activities
«« The importance of PED and district support
«« The advisability of Government investment in school safety
«« The need to ensure continued support for schools
«« Future investment in SCCFS
«« Programme replication.
For each recommendation, actors and required actions have been identified.
Chapter 11: Lessons learned
The broad lessons presented in Chapter 11 are related to:
«« Variability in programme strategy
«« The need to strengthen monitoring and evaluation
«« The need to manage the expectations of role players and stakeholders
«« The management capacity of schools
«« The need for future ‘hybrid’ programme strategies
«« The need to monitor expenditure per school
«« Programme design and evaluation
«« The weighting of programme principles
«« The need to respond to schools’ most pressing needs
«« Programme sustainability.
The lessons learned from the evaluation are related in Chapter 11 to the programme planning and implementation domains in
which they should be applied. Notes are provided to guide future actions.
PDF files require Acrobat Reader.