2011 Somalia: Impact Evaluation of the Child to Child Clubs Project Somaliland, Puntland and South Central Zones
Author: Dr. Marla Stone
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
The Child to Child (CtC) Clubs Project has been implemented in the 3 zones of Somalia under the Strategic Partnership (SP) since 2007, funded by the United Kingdom’s Department for International Development (DfID) in partnership with United Nations Children’s Fund (UNICEF) and United Nations Education Scientific and Cultural Organization (UNESCO). The clubs are extracurricular and are aimed at providing a forum for children’s involvement and participation in rights based activities emanating from clubs within their schools that will improve their lives and the lives of other children in the school and families in the community.
UNICEF has identified 5 implementing partners that have established 155 CtC Clubs in primary schools, 45 in Somaliland, 50 in Puntland and 20 each in 2 regions and 1 district of the Central South Zone. The clubs usually consist of 20-40 children, totaling 5,334 members, of different ages who represent their respective classes.
These children use information learned about child rights and skills of organization and leadership to attract more children to school and tackle identified areas of need in their school and community. The Child to Child approach in club operations encourages children to be change agents, taking responsibility for their situations, principally by teaching them a problem solving method called 6 Steps.
This evaluation is for the purpose of assessing the implementation, outcomes and sustainability of the Child to Child Club Project in all 3 zones.
The evaluation is done against 5 stated objectives of the project and 1 stated goal of the SP. The CtC Project goals are:
1. Excluded children will be attracted to enroll in formal schools.
2. Girls and boys will be equally motivated to remain in school until completion of primary school.
3. Children will acquire and be able to apply essential life skills in their daily lives.
4. Children will be able to gain better understanding of selected topics.
5. Children will be able to act as change agents in school, family and community levels.
The SP Objective is:
Increased enrolment and completion of basic education with particular emphasis on girls.
School visits where interviews and focus groups with club members, non-club members, head teachers, club catalysts/facilitators and adults in the community provided part of the information for this report. Interviews were also conducted with UNICEF field staff, 4 of the 5 implementing partners (SOLSA, PSCA, SHEDU, and HAPO Child) and with Ministry of Education officials. Two club meetings were observed, 1 in Somaliland and 1 in Puntland.
Schools to visit in SL and PL were selected by the evaluator, using information about the schools provided by the implementing partners. Characteristics such as small/large, town/village, older/newer clubs were represented in the sample. In SL, 20% (9 of 45) participating schools were visited. In PL 18% (9 of 50) were visited. UNICEF staff identified representatives from 1 school each and 1 representative each of the implementing partners in Middle Shabelle and Hiran Regions for the evaluator to interview. In total 9 instruments were administered to 20 schools with CtC Clubs. In total 221 people participated in the school and partner interviews and focus group discussion.
The evaluation has been organized under 7 evaluation criteria provided by UNICEF. These are Efficiency, Impact, Coverage, Coherence, Relevance, Efficiency and Sustainability. The data is reported in detail in separate sections for each criterion.
Findings and Conclusions:
1. Children in the CtC Clubs have knowledge of their rights and are actively pursuing those rights for themselves, others in the school and non-school going children in the community.
2. Abundant anecdotal evidence suggests that CtC Clubs are drawing more children to school and keeping others in school, both by action and by example of the members. However, the actual number of children who have enrolled, returned or stayed in school as a result of the clubs is not formally recorded.
3. Changes in enrolment indicate that female enrolment, though increasing, is losing in percentage against male enrolment. Enrolments of both genders are increasing, but in many cases the influx of boys is greater than that of girls. In several schools where a female majority existed, there was a shift to a male majority.
4. Children in schools with CtC Clubs are viewed as agents of change by themselves, fellow students, teachers, community members and leaders and Ministry of Education staff. The activities and projects undertaken by club members have solved or significantly improved the health, environs, social interactions and education of their schools, families and communities.
5. There are profound positive changes in self-perceptions by most club members. Club members possess a sense of empowerment and confidence to act as leaders of their peers and to speak to and interact with adults in healthy ways. An augmented self efficacy now fosters the belief that they as children have knowledge and skills that can make a difference.
6. Monitoring of the project has not been undertaken by UNICEF and not by the MoEs in a formal way. LNGOs lack knowledge and need guidance in how to monitor catalyst/facilitator and club performance. The UNICEF staff in both Nairobi and in the field have been remiss in providing guidance to the implementing partners on how to monitor the CtC Clubs against more the than numerical data for the objectives of the project (number of female club members, increased school enrolments, number of beneficiaries). Actual monitoring of the qualitative technical performance of the LNGO staff and the cat/facs on the important CtC approach is missing. The UNICEF Indicators for Monitoring CtC guide sheet could have been instructive, but it apparently has little meaning for the implementing partners, who are not educators.
7. When the unit of evaluation is the zone (SL, PL, CS) there is evidence that there are gains in enrolment in schools with CtC Clubs. However when the unit of analysis is the region or the school the same evidence is inconclusive. Within a zone some regions have a negative change in enrolment, while some have a positive change in enrolment. Within a given region that has 7-8 schools in the project changes in enrolment may range from -47% to 81%. Four of 8 schools may experience a decrease in enrolment and 4 may enjoy an increase in enrolment.
8. On average the project covers 19% of schools. Excluding Hiran Region, the average coverage is 10%. In SL and PL approximately 16% of all children who go to school have access to a CtC Club.
9. The goals of the CtC Project are clearly in alignment with the goals and policies of the Puntland and Somaliland Ministries of Education. Examples are the intent to increase enrolment in school, encourage completion of primary education and fostering student participation in their education, as well developing the education sector.
10. Schools with CtC Clubs are markedly and positively different from schools that do not have CtC Clubs. Club children claim ownership of the school and lead the student population in caring for the school grounds, respecting each other and making positive changes in their community. Awareness by children that they can have opinions and express them and can use their abilities to be positive agents of change is a highly relevant change that will improve their lives.
11. The extensive gains achieved by the CtC Project are greater than any planned outcomes. Though documented numerical increases in enrolment and female enrolment are not clearly substantiated, the changes in individual children, their families, their communities and their schools are well documented in the data collected. The extent of change is dramatic, as recognized by the children themselves, their classmates, the school staff, community members and even MoE staff.
12. There is a consensus among individuals and organizations involved in the CtC Project that the CtC Clubs are sustainable from within, with or without any further funding. Fourteen of 14 catalysts/facilitators responded yes when asked if the clubs will continue when funding ends. Five of 5 MoE staff in SL and PL were confident that student ownership of the clubs would propel the continuation. The implementing partners recognize some challenges to sustainability, but insist that the benefits experienced will be the basis for sustainability.
13. Preparations for handover to the Ministries of Education, both by UNICEF and the implementing partners, are currently inadequate to expect that the Ministries will adopt the project in a meaningful way. There are few indicators that either a financial responsibility for or a technical leadership of the clubs will be assumed by the Ministries. A few individuals are enthusiastic, but there is no broad-based knowledge and planning to adopt the project.
14. In 4 of 5 geographic areas, CTC Clubs are facing significant risks to their clubs. The 3 areas in CSZ are under threat from local authorities who do not approve of the CtC approach and potential school closures following the exit of NGOs who once paid teacher salaries. SL is in a transition from SOLSA catalysts to teacher facilitators who may not have the same commitment and willingness to function in their roles without a monthly stipend.
1. Attend to training needs of catalysts/facilitators (cat/facs) before the project closes. A significant number of cat/facs have expressed a need for additional assistance in both general and specific training to be able to function effectively in their roles. Their concerns should be addressed in either one-on-one or group remediation methods by LNGO supervisors/coordinators and UNICEF staff.
2. Distribute the Somali versions of the Facilitator Manual and the Student Workbook to the schools before the project closes. These print materials in Somali language provide practical means of support to the cat/facs that will help solidify understanding and use of the CtC approach and will be essential tools when support from the LNGO staff is no longer available.
3. An additional teacher, particularly in CS schools where the head teachers are the cat/facs, should be engaged to serve as co-cat/facs with the head teachers. The CS head teachers should be given permission to share how to facilitate a CtC club with another school staff and share the responsibility in leading the club with that teacher.
4. In SL and PL more effective strategies to better involve the Ministries of Education in the handover of theCtC Project must be developed and implemented immediately. In order to increase the likelihood that the Ministries of Education(MoEs) in SL and PL will assume ownership of the CtC Project, a more strategic approach is needed to effectively make the various echelons in the Ministries aware of the principles and benefits of the clubs. The changeover of crucial staff in PL has hampered what PSCA had done in terms of training MoE staff and SOLSA’s reliance on one individual in the Ministry has limited the SL Ministry’s awareness and understanding. Both organizations must substantially increase their efforts in this area, particularly in awareness and training. UNICEF staff, both in Nairobi and in the field, must martial their influence to exert pressure on the Ministries to meet their responsibilities to pick up the financial and administrative oversight of the project.
5. Initiate a process to adopt the CtC Clubs into the primary school curriculum. The alignment of the CtC approach and objectives to the MoE policies and objectives has been established in this study. UNICEF should use the already existing complementarity as a basis for incorporating the CtC Project to intercurricular portion of the primary school curriculum. What works so well in the clubs would work just as well in the regular classrooms with magnified change in the quality of education.
6. Encourage the expansion of the CtC Clubs to more schools. Something as powerful as the CtC Clubs should not be denied to other children. Implementing partners should invite head teachers, key teachers and especially children from non-participating schools to attend the club meetings and club activities as often as possible before the project closes. With firsthand observation and interaction and the availability of the new CtC manuals, more schools could catch the vision of what the CtC Clubs could do for their students and start – without any financial support.
The data reported in this study regarding effectiveness and impact should be shared with the Ministries of Education and local partners who should be encouraged to move forward in allowing more schools to initiate a CtC Club.
7. Consider establishing term limits for club members in order to permit more students to experience the benefits of being active members of the CtC Clubs. More children could benefit from club membership if term limits were set to two years at most. Those who are elected/selected for a second year should identify another child to shadow their activities and be groomed for potential election the next year.
Lessons Learned (Optional):
1. The CtC Clubs Project has exceeded all expectations in changing children’s perceptions of themselves and their ability to be agents of change. The spillover to the school and community has been beneficial in many ways to many groups of people.
2. The CtC Clubs Project has succeeded in some ways despite the lack of sufficient record keeping, monitoring and evaluation. Both the implementing partners and UNICEF have been lax in their duties to provide a sufficient measure of quality assurance.
3. The sustainability of the project in SL and PL has been jeopardized by insufficient planning for more direct involvement of the Ministries of Education in the development of the project and in assuring that project success is made known to a wider scope and higher echelons of the Ministries.
4. The cessation of funding of the CtC Project is not expected to mean a closure of the clubs. The children who are and have been members of the clubs, as well as their classmates, have an infectious enthusiasm for what the club has done for them. There is a strong likelihood that the clubs will continue to operate in some form or another. Ownership of the clubs is local and continuation is likely to be determined locally, rather than by the Ministries.
5. What has been a fantastically effective project has been kept exclusively for participating schools for much too long. The CtC Clubs should have been opened up to all schools that want them by now, offered and almost certainly accepted without any external financial support, such as a monthly stipend for the cat/fac. The provision of the Facilitator Manual and Student Workbook would help in lieu of formal training
Full report in PDF
PDF files require Acrobat Reader.