Evaluation database

Evaluation report

2012 Somalia: Developing a Local Model for The Delivery of Primary Education in Karkaar Region, Puntland



Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."

Background:

This Report presents the findings of the Final Evaluation (FE) of “Developing a Local Model for the Delivery of Primary Education in Somalia” implemented in Karkaar Region in Puntland by Save the Children, and funded by UNICEF, and hereafter referred to as the Primary Education (PE) Project.

The goal of the project is:  “to increase the number of children accessing and completing inclusive, quality and protective basic education in Puntland”. Its overall purpose is: “to develop a local model for the delivery of primary education in Somalia, which contributes to the overall goal of increasing the number of children accessing and completing inclusive, quality and protective basic education in Puntland”.  The specific purpose is: “to support the Puntland education authorities and communities to establish a regional level model of good practice in the delivery of quality primary education”.

The project had four expected outputs, namely: 
i)  A 120% increase in the number of children accessing and completing quality basic education;
ii) The capacity of Education Authorities and Communities 
to manage and monitor the delivery of quality primary education is strengthened; 
iii) A protective environment is in place in all SC supported schools;
iv) Resources available for basic education in Puntland increased.

Purpose/Objective:

The main objective of the evaluation was to assess the performance of the project, using the  standard OECD and UNICEF evaluation criteria, namely: Relevance; Effectiveness; Efficiency; Implementation process; Coverage, Coherence, Impact; Sustainability; and to give recommendations for action, based on the findings

Methodology:

Field work consisted of 12 days in Karkaar region where the project was implemented, and 5 days in Garowe, to continue with interviews with the MOE, and to plan and carryout a validation workshop. The workshop was intended to present topline findings and reach consensus with key stakeholders on the same, including the way forward for the project. The Final Evaluation team comprised the consultant, two (2) Project Officers of Save the Children (SC) and a local consultant. Out of the 5 districts in Karkaar 3 (60%), namely Gardo (urban), Rako (pastoral) and Bandar Bayla (fishing & pastoral) were reached. The FE population of interest included: i) School level respondents comprising: headteachers, teachers, Community Education Committee (CECs)/parents; and pupils ii) MOE officials at the regional level, namely: the Regional Education Officer (REO) Karkaar Region, District Education Officers (DEOs), District Education Boards (DEBs) and, the supervisors iii) the MOE  officials at the central ministerial level comprising the Ag. Director General (DG), the Program Director and heads of Primary Education, Non-Formal Education and School Supervision Departments. 

The evaluation adopted a mix of data collection approaches namely: Qualitative and Quantitative methodologies, in order to maximize results. The data collection tools used under the qualitative methodology were: FGDs and one-on-one discussions, using discussion guide notes. Semi-structured questionnaires were also used with some of the groups such as the REO, headteachers and teachers; MOE officers. Quantitative data such as school enrolment and retention trends; teacher establishment; school facilities of interest to the evaluation; etc were collected using a structured tool for quantitative data collection.

Simple random sampling approaches were used. A sample size of 36% (12) of the 33 beneficiary schools was reached. A total of 125 people were reached and interviewed during field work, with a gender proportion of 36% for females. 

Findings and Conclusions:

1  The Project Preparation & Design Process:
This sub-section seeks to establish the project preparation and design process; whether or not the activities were relevant, and if the process was participatory and inclusive. The key findings are that: i) the Primary Education Project in Karkaar Region was the result of intensive consultations between the key stakeholders and actors in educational development in Puntland, namely:  the MOE, UNICEF, SC; the REO, DEOs, and CECs (in Karkaar Region). ii) The process was preparatory and involved the key stakeholders in educational development in Puntland in general, and in Karkaar region in particular. The project was designed to address issues of access, participation and retention, especially of the disadvantaged children of the pastoral and fishing communities in Karkaar Region; gender disparities in enrolment; capacity strengthening; school based violence; child rights and; district level resource mobilization.

2  Relevance of the Project:
The problems facing educational development in Karkaar Region, and indeed Puntland in general,  were: i) Lack of access to and low participation in education ii) High and prohibitive cost of education iii) Poor teaching/learning environment, with dilapidated physical infrastructure iv) Weak capacities in educational management at the REO & DEO v) Weak capacities in the communities  to manage schools  vi) Child rights violation and poor child protection, manifested in rampant school based violence, and  viii) Low budget allocation for education, constituting only 1% of government budget at the time.

The findings of the FE are that: i) the design of the project directly addressed the problems facing the target beneficiaries, and, therefore, relevant to them; ii) The goal and purpose of the project are focused and definitive of the concerns to be addressed by the project; iii) The four (4) expected outputs, stated above, and their accompanying activities were specific to, and directly address the problems identified. The findings also show that the concerns addressed by the project were in keeping with the policy concerns addressed in the Puntland Education Policy White Paper which emphasizes on access and participation of the marginalized communities, school infrastructure development, and capacity development in educational management. 

3  Effectiveness of the Project: 
1.  The Extent to Which the Project Goal has been Achieved
The project goal (stated above under 4.2) had 2 tracking indicators, namely: Indicator 1:  120% increase (at least 13,635 children) enrolled in primary schools in Puntland: The findings indicate that the overall regional enrolment in Karkaar has increased from 7,501(41% girls) to 12,682 (44% girls), representing 69% enrolment increase in the region. However, based on the project target of “at least 13,635 children enrolled…” by the end of the project, the enrolment of 12,682 children represents an achievement of 93%. Indicator 2: Increased retention and completion rates of all children between Grades 1 to Grade 8:  The survival and transition (Grade 4-5) rates in the sample schools were assessed by district and gender, the key findings of which are averaged and summarized below, against the scholastic period, 2008/2009-2010/2011.

Retention rates are fairly good in the supported schools, indicated by the data collected from the sample schools. The average retention rate at lower primary is 84% (85% for boys & 82.5%) and 87% (85% for both boys & girls) at upper primary level in the sample schools. The data from the sample schools indicate that there is not much difference between the retention of boys and girls both at lower and upper primary levels. The main reasons given for this  are: i) the impact of the intensive community mobilization and sensitization which focused on the rights of the child (both boys and girls) , education being one of them, and;  ii) the DEB school teacher salary payment initiative, which benefited 580 needy children  who were exempted from school fee payment as a result. 

2. The Extent to Which the Overall and Specific Purpose of the Project have been Achieved The overall and specific purpose of the project is reflected under 4.2 above. 

Indicator 1: “Regional, District and community level education institutions demonstrate increased capacity to effectively manage and supervise the delivery of quality primary education”: The Key findings are that:  i) Managerial and supervisory capacities of the regional & district education authorities, and community based managers (CECs), were strengthened through training; ii) This has resulted into improved frequencies and quality of supervision in the schools in Karkaar Region, reported to be 1-2 visits per term, compared to the situation before where supervision visits by the education authorities were very rare. 

Indicator 2: “Physical environment and capacity to accommodate children is improved - 15 new schools constructed and 15 Schools rehabilitated”:  i) A total of 13 new classrooms were constructed in Karkaar Region (this represents 87% achievement of target); ii) 22 classrooms were rehabilitated in 17 schools (147% achievement of target); iii) 20 new latrines (9 for girls) were constructed in 16 schools; iv) 26 latrines (15 for girls) were rehabilitated in 22 schools; v) 8 water storage tanks and 4 berkeds were provided to 12 schools. These efforts have resulted in: increased enrolment and retention of children in general; increased enrolment &retention of girls; improved hygiene and sanitation situation in schools. 

Indicator 3: “80% of teachers demonstrates understanding of child protection and applies the principles in practice”:  i) All the 33 CECs and 66 teachers (2 per school)  from the supported schools were trained on child – rights issues, and on how to monitor, identify and report cases of child abuse in their schools; iii) Teachers’ Code of Conduct which was developed under the SIBES  Project was reproduced and distributed to the 33 supported schools in Karkaar region; iv) There are indications of drastic reduction (60% drop) in cases of school based violence in schools in Karkaar, and increased use of  counseling and other non-violent means of correcting wrongs in student offenders.   Indicator 4: “Increased external funding for teacher salaries”: i) The DEB Fund was established in all the 5 districts, and managed by the DEB (with the CECs representing the grassroots); ii) Part of funds raised by the DEBs was used to pay the salaries of 75(20 women) teachers, resulting in their increased retention and regular attendance; iii) Part of the fund was used for school physical infrastructural development (classrooms; latrines; water tanks, etc).  Indicator 5: % Reduction in average school fees: i) A total of 580 poor children (about 17 per school) were exempted from school fee payment, against the amount of money received for payment of the teachers, resulting into increased enrolment and retention of children in the supported schools.

3. Achievement of the Project against the Expected Outputs/Results, and Activities Specified in the Project Log Frame.

Achievement Against Output 1: 120% increase in the number of children in Karkaar region accessing and completing quality basic education.

Indicator 1: “120% increase in children enrolled in primary schools in Karkaar Region – from 7,501 (2007/2008) to at least 13,635 by 2011”:   As already articulated above, a total of 12,682 (44% girls) are enrolled in Karkaar as a result of the project efforts, representing 69% increase in enrolment. 

Indicator 2: “Average proportion of girls in all primary schools is 45% by 2011”: i) Of the 12,682 children enrolled in Karkaar Region, 44% are girls, representing an achievement of 98% of target at the regional level. ii) Of the 5 districts, Rako had the highest gender proportion for girls (47%), while Hafun and Bandar-Bayla had the lowest with 41% each. iii) There are, however, discrepancies in gender representation for girls as seen in the data of the sample schools visited (see Annex 1 for detailed presentation and analyses). Qalwo Primary school in Rako District topped with 52% gender proportions for girls, while Shahada in Gardo District had the lowest with only 34%. These figures point to the gender dynamics within the individual districts and schools. 

Indicator 3: “30 schools benefiting from new/improved classrooms and other physical improvements”:  i) All the 33 schools benefited from various physical development activities, namely new classroom construction and rehabilitation; construction & rehabilitation of latrines, and; construction of water storage facilities (as in 2 above). This is an achievement of 110% of target.                        

Indicator 4: “% increase of children with disability in schools”: The FE data shows minimal participation of children with disability. Of those enrolled in the supported schools, less than 10% of them were children with disability. Among the key reasons given for their meager participation were:  i) lack of awareness in communities/parents that children with mild disabilities could attend ordinary schools ii) Lack of disability friendly schools facilities iii) Lack of psychosocial skills in teachers in ordinary schools to handle these children. Because of the absence of data on the enrolment of children with disabilities in Karkaar Region, the performance of this indicator cannot be assessed.

The average achievement under Output 1 (based on Indicators 1-3) is 88.5% 

Achievement Against Output 2:  The Capacity of Education Authorities and 

Communities to manage and monitor the delivery of quality primary education is strengthened.

Indicator 1: “90% of REO and DEO staff demonstrate understanding of their roles and responsibilities”:   All the REO & DEO staff interviewed (Deputy REO, DEOs, supervisors) showed good understanding of their roles & responsibilities and were able to state them clearly. This is the result of their training in roles and responsibilities as already articulated above.  

Indicator 2: “Functional District Education Boards established in each District”: All the DEBs have been established, trained on their roles and responsibilities, and are functional in all the 5 districts.  

Indicator 3: ”All 66 primary schools in the Region have functional, trained Community Education Committees (CECs)”. All the CECs of the 33 supported schools have been trained and are functional. This represents target achievement of 100%. All the 12 schools visited had trained and functional CECs. Indicator 4: “Capacity of MoE, REO and DEO enhanced to disburse, monitor and manage education funding”. All the Regional Education Accountants; the 5 District Cashiers, 5 chairpersons and 5 secretaries of the DEBs, and Finance Officers from the MOE, were trained on the basic concept of block grants management, in preparation for the implementation of the DEB block grants, resulting into  proper management and disbursement of the DEB funds and block grants.  Indicator 5: “25 schools have functional school based clubs”  i)  About 60% of the 33 school based children’s clubs are functional; but some are still weak and in their formative stages. 

Achievement Against Output 3: A protective environment is in place in all SC supported schools:

Indicator 1: “Child protection and child rights training are institutionalized into the training curricula for all MoE, REO, DEO staff, teachers and CECs”:  i) This indicator was fully met. All curricular for training education authorities at all levels reflect child rights and child protection issues have incorporated child rights issues, translating into 100% achievement of target. ii) General awareness of child rights issues was evident during interviews with the various cadres of education authorities (REO, DEOs, DEBs, and CECs), headteachers and teachers.  Indicator 2: “% Reduction in number of children reporting school based violence.” i) The last Interim Report (March 2010 –March 2011) indicates that 60% of the teachers in the supported schools agreed with the principles of child protection, and were not using corporal punishment. ii) Basically all (100%) of the 21 teachers & headteachers, the 39 students and the CEC members interviewed said that cases of corporal punishment have reduced drastically, and that counseling and other forms of correction, were used to correct wrongs in student offenders.  Indicator 3 & 4: “% of teachers that agree to the principle of positive disciplining of children” (3) & “% of teachers not practicing corporal punishment” (4):   i) Based on the FE data and the interim reports, there are strong indications that more than 60% of teachers agree to the principle of positive disciplining of children and are actually practicing it in their operations as teachers. ii) All (100%) the 21 teachers (12 headteachers) interviewed indicated they agree to the principle of positive disciplining of children.

Achievement against Output 4: Resources available for basic education in Puntland increased: 

Indicator 1: “Government budget for education increased to at least 6% of total government expenditure”: i) Currently, the MOE budget constitutes about 2% of the total government spending. This is about 33% achievement of target. ii) In terms of actual figures, it increased from US$ 317,139 in 2008/2009 to US $ 460,049, constituting 45% increase.  Indicator 2: “District level mechanisms established to harness and monitor additional private/donor funds for teacher salaries in Karkaar”. i) This indicator was met through the establishment of the DEB Fund, and the (DEBs) fund raising efforts; ii) The operation of the fund’s was facilitated by the establishment of accounting systems at the District level, through the deployment and training of the District Cashiers stationed at the DEOs offices, and iii) the development of a simple and functional disbursement system at the DEO’s offices. Indicator 3: “School fees reduced as teacher salaries supported from other sources”: As already explained above, the funds raised by the DEBs were used partly for payment of teachers’ salaries, with the aim of reducing the burden of school fees from poor parents. As a result, a total of 580 poor children (50% girls) benefitted from school fee exemptions.  

4  Assessing the Efficiency of the Project:
1.  Performance Efficiency Assessment: The sub-section, above assessing the effectiveness of the project, indicates that most of the activities planned under the 4 expected outputs/results were implemented and completed with an overall performance score of 92.63%, for the project, and rounded off at 93%.  

2. Time Efficiency Assessment
The first year of the project was slow, with activity implementation starting 2-3 months later. Delay was also reported in the implementation of activities in 2009/2010, for about 3 months. However, the time lost was recovered in the course of implementation, by speeding up the implementation process. By the time of this evaluation, activity implementation was on course.  

Cost Efficiency Assessment:  On average the cost efficiency factor of most activities was very reasonable, considering their short, medium and long term benefits of the efforts. Of the cost centers assessed, two had the lowest per unit cost, but  with the highest outputs, namely: Activity 1: “Child Rights and Child Protection training materials incorporated into all training curricula”, (under Output 3), with   a cost efficiency factor of US $ 0.6, benefitting 12,632 children enrolled in schools in Karkaar region; and  Activity 2:  “Mobilization campaigns with communities to send all children to school” (under Output 1), with a cost efficiency factor of US S 4.00, reaching about 2,500 community members, and leading to the enrollment of 5,181 more children (over 2,000 or 44% of them girls). Two activities with the highest unit cost were: 

Activity 5: “Training and support to Community Education Committees (CECs)” under Output 2, with per unit cost of US $ 56, to train 460 CEC members (66 CECs) on roles and responsibilities, and; Activity 2: “Strengthen Systems for payment of teachers from MOE through Regional and District Education authorities” under Output 2, with per unit cost of US $ 25, facilitating the establishment of standardized systems for the payment of teachers, and benefiting 120 teachers paid under the project. However, considering the long term benefits of capacity development of CECs, and the continued benefits of standardized systems of teacher salary payments at the district level, in Karkaar region, these seemingly high unit costs are still considered efficient. 

5  Implementation Process:
This takes into account the strategies employed and challenges experienced during the implementation process. 

1.  A mix of strategies were employed, namely: i) Participatory and inclusive processes, involving the key stakeholders; ii) Institutional and technical capacity strengthening of education authorities; iii) Quality enhancement and standards control models, through learning space expansion & face-lifting of schools; and enhancement of the frequency and quality of school supervision and monitoring; iv) Gender responsive approaches to school development: Community mobilization & sensitization on child rights focusing on the right of girls to education; gender responsive hygiene and sanitation facilities; gender balanced leadership in school based children’s clubs; v) Policy Advocacy: for increased government spending in education (from 1% to 6% of the total government budget). The % increase in government spending in education was 2 % during this period.

2.  Some of the key challenges experienced: a) External Challenges: i) the prolonged drought; ii) long distances between the coastal districts of Hafun & Bandar-Bayla and the Regional Headquarters which slowed down the frequency of supervision & monitoring from the Regional Education Office in these districts; iii) The unpredictable security situation in Karkaar, which minimized the number of times the international education experts could travel to the rural districts; b)  Internal Challenges:  i) Gender disparity in enrolment is still a challenge in some districts and individual schools; ii) high attrition rates of girls in some of the schools, suggesting the persisting socio-cultural attitudes which militate against the education of girls; iii) Inconsistency in enrolment seen in sharp fluctuations (especially during drought).  

6 Coverage & Coherence: Target beneficiary groups:  i) Out of 7,437 (45% girls) additional school aged children targeted, 5,181 (44% girls) children were reached, constituting 69% coverage of the intended group; ii)  20 regional & district education officers were targeted, but this was surpassed by 3, comprising 115% coverage of the intended group. iii) 460 CEC members were targeted for training, but 462 were reached, constituting 100.4% coverage; iv)  2,350 pupils were targeted to be reached with sports equipments/facilities, and to benefit from the newly built and renovated classrooms, water and sanitation facilities. About 3,150 pupils (134% coverage) were reached with these facilities. 

7 Assessing Impact of the Project: Positive impact is evident from the data: i) Increased access to school and by extension, increased enrolment in Karkaar Region ii) ‘The one goat for one child’ innovation  had such remarkable impact on enrolment among the pastoral communities, who make the majority in Karkaar region. About 1,845(40% girls) were enrolled as a result; iii) Narrowing Gender gaps (44% proportion for girls up from 41% in 2008/2009) at the regional, district and school levels; iv) Increasingly child-friendly & protective learning environments in schools; vi) Community sourced & managed financial resource for educational development in Karkaar Region, resulting in expansion of learning spaces, improved hygiene & sanitation situation in schools; vii) Institutionalization of some of the systems & models developed under the project, into the mainstream, e.g.: Job description and protocols  and the school monitoring and supervision system. viii) The DEB model and the ‘One Goat for One Child’ innovation are being considered for replication nationwide by the MoE. 

8 Lessons Learnt: i) participatory and inclusive approaches to doing development projects, where all key stakeholders are involved, works to yield the desired results; ii) the DEB Fund and the achievements made through it show that it is possible to reduce the feeling of total dependency and helplessness in the local communities, to appreciate that they can achieve a certain level of sustainability; iii) For transparency and accountability as good practice to work in any organization, the necessary systems must be put in place to facilitate its practice iv) It is important to involve children as agents of change in matters affecting their lives, if change is to be effective at all levels.

9 Sustainability: There are prospects for sustainability in: i)  Institutional & technical capacity development ii) Institutionalization/mainstreaming of the Karkaar model iii) Minimal  financial sustainability in the DEB Fund model, for local level educational development, e.g. school maintenance; expansion of learning spaces; school fee exemptions, etc  iv) the ‘One goat for one child’ innovation as potential for in-kind payment of fees, using available non-cash local resources such as livestock, which can be changed into cash by the schools. 

Conclusions: The key ones are that:
1. For an initiative (whether educational development or other) to produce the desired results, models that encourage meaningful participation and inclusion of, and the mutual consultations with, the key stakeholder are the way to go; and in this particular case, in Puntland and elsewhere in Somalia; 
2.  Community empowerment, where communities (including children) are able to make decisions about, and act on their present and future, based on informed options and decision making skills, is an important feature that should, of necessity, be part and parcel of any development efforts in Puntland and elsewhere in Somalia; and that when communities are able to do this, it is what really constitutes real community empowerment; 
3.  Innovation in development efforts such as education are good, but unless they are grounded on local experiences and culturally acceptable practices, it may eventually add up to very little or nothing. The ‘One goat for one child’ innovation success leads to the conclusion that any innovation that seeks to improve local situations, and for it to be sustained, it should take into account culturally acceptable local practices;
4.  The PE in Karkaar Region model has demonstrated that the strategies employed in its implementation can work to yield the desired results. It is concluded, therefore, that the lessons learned in this project and the experiences gained should be used to replicate it as a viable model to help enhance educational development in other regions in Puntland, and elsewhere in Somalia.

Recommendations:

Action recommendations were arrived at through a 4 stage process, namely: i) The findings based on FE data analyses and conclusions; ii) views of the various respondent groups/key stakeholders iii) views of education experts/program officers (UNICEF & SC) iv) views of the Dissemination Workshop participants. Six (6)  key/priority recommendations are summarized below:

Recommendations Pointing to the Future Direction of the Karkaar Project
1.  PE in Karkaar model should be replicated in other regions in Puntland and elsewhere in Somalia (Action: UNICEF; SC & the MOE);

Recommendations on Strategies to Keep up the Tempo of Increasing Enrolment
2. Community mobilization and sensitization, back to school campaigns and enrolment drives should constitute a major component of every educational development program in Puntland and elsewhere in Somalia, which intends to facilitate increase in enrolment and retention/survival. (Action: UNICEF;SC; the MOE);

Recommendations Addressing the Under-enrolment and Participation of Children with Disabilities
3.  Given the minimal participation of children with disabilities in educational programs in Puntland, it is recommended that: i) The MoE, in collaboration with its educational development partners such as UNICEF, SC, and others, should make a deliberate effort to mobilize and sensitize parents on the right of children with disabilities to be given equal opportunity to go to school; ii) SC should discuss with the MoE the possibility of replicating  in Puntland, the on-going Inclusive Quality Basic Education Project being implemented (by SC) in Somaliland, in order to encourage the enrolment of children with disabilities, and especially those with mild conditions that can be handled by ordinary schools; (Action: SC & the MOE);
4.  Future basic education intervention programs should address the area of capacity development of teachers at the primary school level, on how to handle children with disabilities, and on psychosocial skills to help these children integrate into the school system; (SC & UNICEF);

Recommendations to Strengthen Child Participation & Protection
5. In order to enhance their effectiveness, school based children’s clubs should be provided with yearly/ periodic regional leadership training. A more cost effective approach would be to train trainers (e.g. Regional school supervisors/mentors) who would then carryout school based leadership training of children at minimal or no cost (Action: SC; UNICEF & the MOE);Recommendation on Continued Development & Strengthening of Managerial & Supervision Capacities of Regional & District Education Authorities and Schools/Community Level Managers  
6. Capacity development of education authorities is still one of the key strategies for enhancing quality assurance and quality monitoring at the school level. It is recommended that the MOE should ensure that  educational development programs by partners should, of necessity, have a capacity development component which addresses the various capacity development needs of the various categories of education authorities and the CECs (Action: UNICEF;SC & the MOE).



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search