Evaluation database

Evaluation report

2010 jamaica: Evaluation of Project “Strengthening Diversion Opportunities, Rehabilitative and Reintegration Programmes for Children Deprived of their Liberty and Not in the Formal School System: An Integrated Livelihood and Remedial Education Project”



Author: Jennifer Jones and Audrey Brown. Partners: NGO Western Society for Upliftment of Children

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:

The framework under which UNICEF funded this programme included child rights, child justice and child protection outcomes, which would cover children in conflict with the law benefitting from diversionary and community based programmes, and out-of-school children benefitting from programmes to rehabilitate and reintegrate them into the formal school system. Older youth were equipped through remedial education, skills based learning, livelihood skills and increased work opportunities.
The project‟s key result is identified as follows:
“By December 2009 in the parishes of St. James, Hanover, Trelawny and Westmoreland, 424 vulnerable and out of school children (12-14) and (15-17+) (21% of five-year CPAP target), especially boys equipped with remedial education and 70% of this total (15-17+) equipped with livelihood (i.e. prevocational) skills and increased work opportunities; fostering the rehabilitation and reintegration of 195 children into formal school systems, work opportunities (for 15-17+), apprenticeships and skills training programs (for 15-17+).
Tasks and expected results or outputs, with appropriate indicators and their sources, are set out. The tasks fall under two main headings: 1) School Programme and 2) Livelihood (i.e. Prevocational) Skills Training and Job Placement with a third task, 3) A Baseline Assessment, to be carried out in 2007 (it was actually done in 2008) as well as a final evaluation to be carried out in 2009 (started in February 2010) of which this report is an outcome.

Purpose/Objective:

To conduct an evaluation of the effectiveness and scope of the services of the Western Society for the Upliftment of Children in the areas of remedial education (literacy and numeracy), personal and family development, livelihood (i.e. pre-vocational) skills development, reproductive health, conflict resolution, mediation, guidance counseling and re-integration into the formal school system.

Methodology:

In evaluating the Western Society for the Upliftment of Children, a framework for the evaluation was developed from the original multifaceted activity proposal, and the following methodologies used:
a. School Programme
- A desk review of records and reports that included the WSUC Baseline Assessment July 2008; the UNICEF Multifaceted Activity Proposal Form completed by WSUC for a project to run start from July 2007-Dec 2009; and Quarterly Progress Reports submitted by WSUC to UNICEF from Dec 2007 to Sep 2009. Missing is the final report for Oct-Dec 2009.
A review of initial intake, assessment and placement, reasons for coming to WSUC, involvement in child labour, and relationship to person responsible for the child, from secondary data from all 110 current individual student files. These files should all have contained an Initial Interviewing Sheet, a Registration Data Entry Form, including entry performance in literacy and numeracy, a Reproductive Assessment Test and Score, administered on entry, periodic Performance Assessment Forms for numeracy and literacy and skills training areas, and forms recording learning objectives and their achievement. Forms also recorded parent attendance at meetings. These records were manually transposed onto an Excel database by the evaluators and then analysed using Intercooled STATA 9 software.
- A review of literacy and numeracy levels on entry and periodically after 12 months, 2 years and 3 years, utilizing secondary data from the database developed from current student files.
- A review of the number of learning plans in place and an assessment of the percentage of objectives met as contained in the current student files on the database.
- A review of attendance and drop-out rates for the terms September-December 2009 and January-March 2010.
- A review of the students‟ assessment of their school and of their own progress, as well as the person with the most positive influence in their life. These views were obtained by utilizing primary data from a survey that was administered to a stratified random sample of 36 students currently registered at WSUC, entered and analysed in an Statistical Package for Social Scientists (SPSS) 17.0 database, and from three Focus Group Discussions (FGD) with a) an all male group of four current students 13-17 years old, b) an all female group of six current students 12-19 years (see Appendix 1 for Current Students FGD guide), and c) a past students group of three males (it was intended to be a mixed group) of 15-28 years of age (see Appendix 2 for Past Students FGD guide).
Parents‟ experience and evaluation of WSUC and of the challenges of educating children were collected through interviews with five parents (all mothers), selected randomly (see Appendix 3 for Parent Interview Guide).
- Interviews were conducted with Board Members and with experts in the field of education and child protection, most of whom are very familiar with the WSUC (see Appendix 4 for list of interviewees).
b. Livelihood (Prevocational) Skills Training and Job Placement
- An assessment of livelihood skills was undertaken by reviewing secondary data from the database of 110 current student files, which identified the skills training undertaken by each student as well as periodic performance assessments. .
- An assessment of children reintegrated into the formal school system, or transitioning into the National Youth Service (NYS), the Human Resource Employment and Training (HEART)/National Training Agency (NTA), the Jamaica Foundation for Lifelong Learning (JFLL) and similar educational and training programmes, or placed in apprenticeships or jobs, in relation to Level 4 students in the last complete academic year of 2008/9. This data includes the number of drop-outs from that level during the year.
- Phone interviews were conducted with two HEART staff members (see Appendix 5).
c. Beneficiary Data
- Personal data on all current students is on the Excel database referred to above.
- Data on family, housing, occupation and income of main wage earners in household, child‟s assessment of family financial situation, stay in children‟s homes, involvement in activities that are in conflict with the law, substance use and abuse, exposure to violence, family disciplinary measures and the child‟s assessment of these, and sexual behaviour and health, were collected through a survey using a random sample of 36 current students, stratified by class level. It was entered and analysed using SPSS 17.0
d. Staff Demographic Data and Evaluation of Quality of Teaching
- Personal data on staff, their age, education, work experience and current professional development activities, along with salary and benefits, including a comparison with government salaries and benefits, has been collected from interviews and records.
- An assessment of the quality and relevance of the teaching methodology, curriculum and implementation and teacher to student ratios through observations of teaching staff classroom activities, reviews of lesson plans, classroom records and tests and assignments, and teacher interviews, was done over a three day period by a trained graduate teacher with a master‟s degree in counselling and over 20 years teaching and counselling experience, including work with HEART and JAMAL (Jamaica Movement for Literacy, now JFLL). The time was spent conducting teacher evaluations in the classroom, interviewing teachers, and reviewing staff records, lesson plans (from September 2009 to February 2010) and student data (tests and assignments for the corresponding period as staff records and lesson plans). Students‟ views on their teachers and teaching methods emerged strongly in the focus group discussions referred to below as did parent views from their interviews.

Findings and Conclusions:

a. Beneficiary Data from Current Student Files
In the current school population male students outnumber female students by almost 6 to 1
(Figure 1), a situation which does not sit happily with the girls nor with the male past students.
However some current male students claimed teacher favouritism “Girls can do anything in
class”, “Yeah, we ever wrong, but girls dem neva wrong!” The Baseline Assessment recorded
118 students attending of whom 80% were male and the general impression from interviews is
that the male population of the school, while always higher than the female, has steadily been
rising, although the reasons for this increase are not clear.
The most popular age range for registration at the WSUC, based on the current student data, is between 12-14 years, with a significant number also registering as late as 15 years old (Table 2).
Approximately 26% of the current student population are between the ages of 12-14, while 63% are between the ages 15-17 years (Table 3). When compared to the students recorded in the 2008 Baseline Assessment, the current population is somewhat older with only 26% between the ages of 12-14 years compared to 38% two years ago.
b. Beneficiary Data from Student Survey
The age distribution of the survey of 36 students is similar to that of the full student body (Table 6), although leaning slightly more towards the younger students. The gender distribution was more skewed more towards males with only 11.1% female students represented (full student body has 14.5% females).
Almost two out of every three students (61%) in the sample had been in WSUC for less than two years; in the full student body this proportion is even higher at 71% (Table 7). Of the rest just over a half had been at WSUC for two to just under three years, while the other students had spent at least three years at WSUC. Two had spent four or more years. This profile is more skewed towards students who have spent a longer time at WSUC than the full student body....etc. please see more in the Report
“Although at present there is no system to accurately assess the school drop-out rate, the declining ratio of boys to girls at secondary level suggests a higher drop-out rate among them. Students who are functionally illiterate fall further and further behind and it is difficult for schools to cope without specialised remedial programmes. However many of these students need specialised remedial education as well as other types of support of a psychological and emotional nature. It is a serious challenge for the education system. It is therefore very important to support those NGOs and special schools that assist such children.” (Cabinet Office of Jamaica 2008: 143).
WSUC is one of those NGOs that is reaching the boys who drop out, it is reaching the functionally illiterate, it is offering specialised remedial education as well as support of a psychological and emotional nature, even though this latter is weaker without a guidance counsellor to lead it. The majority are leaving to take up places in mainstream high schools, to continue their education or to take up employment or self-employment, The conclusion of this evaluation is that the WSUC is offering a critically needed service that has pulled children back from academic failure and low self-esteem, and all its consequences, from futures of hustling and unemployment and a life of poverty, or from the false lure of criminals and their crime, even though not all will – or perhaps can - take advantage of it. It is offering a successful service but the institution‟s inefficiencies are putting in jeopardy its short-term survival and its long term sustainability.
Below is a list of short and medium term recommendations directly related to the Evaluators‟ analysis. They depend on the successful design and implementation of a sustainability plan.

Recommendations:

Short – term Recommendations (6-9 months)
1. The Board needs to secure funds to complete the new building. It should immediately carry out its plan to approach large private sector companies in Montego Bay even as it begins consideration and action around a sustainability plan (see next section).
2. Hire a male Guidance Counsellor.
95
3. Make greater use of government services such as the Child Guidance Clinic and the Child Development Agency for referencing of students with behavioural and/or emotional problems that need specialised professional help.
4. Formalise the link with the CDA, particularly in light of the possibility that the Agency could send some of the children in Places of Safety who need remedial education to the WSUC.
5. Restart skills training as soon as possible.
6. Ensure the system of learning plans, and objectives achieved, is continued in order to motivate students and help them to focus on the long term.
7. Display students‟ work.
8. Use the reproductive test administered on entry to guide the teaching of reproductive health.
9. Organise some form of physical recreation for each student at least once a week, using an appropriate external venue such as Jarrett Park.
10. Ensure that there are written policies and practices for the institution including health and safety procedures, behaviour management, expulsion and that teachers, parents and children are familiar with them.
11. Update all current student files.
12. Ensure that all the appropriate forms are completed when a child is registered.
13. Ensure files of students who register but do not turn up are removed from current student files following an appropriate period.
14. Ensure the standard literacy and numeracy tests are given periodically, at minimum annually, and recorded in student files.
15. Ensure that skills training instructors complete tests for each student on a regular basis and that the results are recorded in student files.
16. Update all current staff files including proof of certification, TRN number, NIS number etc.
17. Contact the Ministry of Education, the Jamaica Foundation for Lifelong Learning and any other appropriate body to find out about accessing professional development seminars, workshops and courses to ensure continuous professional development for staff.
18. Open a tuck shop serving nutritious snacks.
19. Fit the school with fire extinguishers.
96
Medium Term Recommendations (10 – 18 months)
20. Once the new building is complete, seek funding from the Ministry of Education as a private school taking children in instances where there are insufficient places in public schools.
21. Link with HEART re. offering evening programmes and exploring certification for the top levels in the school‟s skill training programme
22. Hire a suitable security guard to man the entrance during daytime hours.
23. Restart the cooked lunch programme.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search