We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Kenya: Evaluation of the Integrated Intervention within the CFS framework



Author: FHI 360

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

In 2013-2014, UNICEF partnered with the Ministry of Education Science and Technology (MOEST) and the United Kingdom’s Department for International Development (DFID) to establish a policy and institutional framework to provide quality and child-friendly education services for nomadic children. By 2016, the program benefited an estimated 3.5 million boys and girls aged 4–18 years in arid counties in Kenya. The program consisted of four discrete school-based interventions:

  1. Water, sanitation, and hygiene (WASH), including constructing or connecting improved water sources to schools for hand washing, drinking, bathing, and sanitation; building gender-sensitive school latrines with hand washing facilities; conducting hygiene promotion training for teachers and school children; and building capacity of Schools Boards of Management (BoM).
  2. Solar lighting for two dormitories (boys and girls), ablution facilities, and two upper primary classrooms in each school, to maximize afterschool time and improve security.
  3. Child-Friendly Schools (CFS) capacity development training developed teacher skills in child-centered pedagogy; addressed issues of gender inclusiveness; developed teaching and learning materials; and trained primary school head teachers to improve school management.
  4. Communication for Development (C4D) aimed to build community capacity to meaningfully engage and participate in analyzing challenges and barriers and identify priority solutions for education access and quality issues in schools. The intervention used local channels to communicate and promote understanding of the legal framework of education governance. Community members were inducted in CFS principles and school management and disaster risk reduction (DRR), and engaged parents, teachers and pupils.
  5. Combination schools received a combination of more than one of the above interventions, or received all four.

Purpose/Objective:

The study examined schools affected by four different types of interventions, including water, hygiene and sanitation (WASH), solar panels, child-friendly schools (CFS), and community involvement through communication for development (C4D), as well as their combinations. The evaluation further examined program effects, implementation challenges, and perceptions of the beneficiaries on program effectiveness.

Methodology:

FHI 360 was contracted to carry out an external evaluation of the program in 2014. Due to the nature of selection for interventions, and the prior presence of similar program interventions in the target areas, a quasi-experimental methodology (matching on the propensity score) was selected as the primary method for this evaluation. In addition, the study factored in the timing and duration of program interventions, and was able to account for the differential effects of the interventions resulting from longer duration.  The study drew on data from 2,136 student interviews, 321 head teacher interviews, and 26 student/community focus group discussions, to examine program effects, implementation challenges, and perceptions of the beneficiaries on program effectiveness. In addition, classroom observations were conducted at 54 CFS and C4D intervention schools provided a gauge of time use and the application of child-friendly instructional practices. The endline reached 321 out of 349 schools (92%) across all eight counties, as some schools were dropped due to security concerns.

Findings and Conclusions:

WASH – Both the descriptive and regression analyses show that schools that implemented WASH the longest (since prior to June 2014) exhibited the largest gains in enrollment relative to all other schools. Specifically, enrollment among schools that implemented WASH the longest increased by approximately 20 percent relative to similar schools in the constructed comparison group that never received the WASH intervention. Attendance rates are also highest among WASH schools with the longest duration.

Solar power – Regression results show that enrollments are only modestly higher - by about 8.6 percent among UNICEF Solar schools than similar schools in the constructed comparison group. These schools also exhibited greater study time, which was one of the intended outcomes of the solar intervention.

CFS – Regression results suggest that enrollment growth among schools that implemented CFS prior to June 2014 as well as those that implemented after June 2014 is 8.6 percent higher than the constructed comparison group. However, the schools that reported having the CFS intervention, but were not supported through the UNICEF/DFID program, that exhibited the largest gains in enrollment. It is likely that these differences were driven by differences in the level of implementation.

C4D – The C4D intervention was largely implemented in conjunction with other interventions. Only six schools report receiving only C4D from UNICEF, therefore, we are unable to isolate the impacts of C4D from that of other interventions. Further, none of the schools report receiving C4D prior to June 2014.

Combination Schools – Schools that received two or more UNICEF interventions were more likely to show positive effects on attendance and enrollment. Schools that received solar power and C4D exhibited 10.5 percent higher enrollment relative to the constructed comparison group.

Recommendations:

Dosage and duration of the treatments play a significant role when determining intervention impacts. In this report, we provide evidence that interventions that were implemented for a longer duration exhibited larger effects on enrollment when compared to schools that implemented the same intervention for a shorter duration. This is especially true with the WASH and CFS interventions.

  • Further research is required to ascertain the level of program implementation fidelity in each intervention. When examining the impact of the CFS or C4D interventions, Overall results were either relatively small in magnitude and/or provided some evidence that similar interventions supported by other organizations had larger gains. In both cases, the findings point to a certain amount of untapped potential gains in enrollment that could be had. As such, context-specific elements of implementations should be investigated to determine where and how certain practices could be improved from a programming standpoint.
  • The investments in WASH, CFS, and Solar panels yield positive returns on investment over the long-term and to varying degrees. It is important to be cognizant of the assumptions of the cost-benefit analysis, especially with the discount rate, which acts as a proxy for the value of future money. In addition, wage based returns are estimated and do not account for any unforeseen shocks to the economy at least in the ASAL region.

Lessons Learned:

The overall findings support the hypothesis that certain programmatic effects require some time to manifest themselves in education outcomes.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
Kenya

Region:
ESAR

Type:
Evaluation

Theme:
Education

Partners:
MoEST, DFID

Language:
English

Sequence #:
2016/007

New enhanced search