We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Romania: National Intermediate Evaluation of the “School Attendance Initiative” Model



Author: Marian Preda, Lazăr Vlăsceanu,Laura Grünberg, Cosima Rughiniș and Marian Vasile

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labeled as ‘Part 2’ of the report.

Background:

When designed in 2010, SAI used as a rationale the principles and approaches specific to the Educational Priority Area model. SAI aimed to prevent dropout and increase school attendance mainly in the communities most affected by high absenteeism and dropout rates. Following the launch of the model in 38 communities (school year 2010-2011), it expanded (school year 2011-2012) to 103 additional communities with high risk of absenteeism and dropout and further, to reach 178 communities, including  93 new and 85 re-validated communities (school year 2012-2013). The main model activities are: (1) improving school management through school directors’ training for strategic planning; (2) improving teaching and evaluation methods and techniques; (3) strengthening parent – school relationships; (4) parental education to improve relations between parents and children; (5) providing positive role models, especially for Roma children, and developing their educational self-efficacy; (6) developing a network of community actors to prevent early school dropout; (7) training school mediators and teachers of Romani language. Expected outcomes are both quantitative (i.e. decrease by 50-60 % of dropout and absenteeism in SAI communities) and qualitative (i.e. preventive pedagogical practices, better parental skills and increased community civic engagement in problems of education).

Purpose/Objective:

As a national intermediate evaluation of the UNICEF model “School Attendance Initiative” the report reviews the interventions carried out in September 2012 – August 2013, corresponding to the third year of model implementation. While previous evaluations of SAI were focused on process analysis and advised on changes in management, the present report aims to assess the relevance, effectiveness, efficiency and sustainability of SAI interventions. Its objective is to inform adjustments so as to increase attendance and prevent dropout, particularly among the most disadvantaged children.

Methodology:

The evaluation methodology is designed by following a set of research imperatives:
1/Complementarity of quantitative evidence, documenting statistical patterns and correlations, and qualitative evidence, documenting social actors’ perspectives and the evolution of local social processes in depth;
2/Statistical validity and reliability by operating with a representative sample that ensures a comprehensive coverage of SAI interventions at local level;
3/Diversity of perspectives in approaching the absenteeism and dropout phenomena;
4/Comparable SAI and control communities.

The reference population is the school population of the 178 communities included in the SAI model in 2012-2013. The sample is drawn from a population of 62.441 schoolchildren that would implicitly be the target of statistical inferences. Probabilistic and multi-stadial methods of selection are used, with sampling in the first stratum according to the development region and the degree of social development of the locality. The final datasets include information on 55 schools and 943 individual cases (children and family members) in: 8 communities studied in-depth (including case studies and group discussions); 32 communities studied comparatively; and, 15 control communities.

The main sources of information are: (1) Statistical data gathered from school archives; (2) Structured interviews with children and their parents or caretakers; (3) Group discussions with school personnel and / or with parents; (4) Structured interviews with school teachers, school principals, school mediators, and, social workers; (5) Desk reviews of documents produced in SAI interventions; (6) Debriefing reports from field researchers that systematize their field observations.

Findings and Conclusions:

This evaluation documents a general increase in the number of absences in the last 3 years in all schools. This increase is visible for all categories of schoolchildren, but it is lowest in schoolchildren in the non-risk group in SAI communities. This indicates that the intervention could not compensate for the initial disadvantage (which was a condition for a school’s selection in the model).

At the same time, the proportion of schoolchildren with a decreased number of absences in 2012-2013 compared to 2011-2012 is the highest in the risk group. While the difference from the non-risk group in the SAI schools is relatively low, the disparity from the control community is statistically significant, which indicates that overall the model was successful in supporting a limited decline in absenteeism. Nevertheless, it could not compensate for the other determinants that generate higher absenteeism, especially factors pertaining to the social and cultural context of formal education.
 
The current organization of the model produced moderately positive results and changes are needed for generating sizeable progress towards the set objectives. Statistical and qualitative data indicate a measurable but limited influence, which is nevertheless explicable given the model’s choice to focus on the very limited educational causes of absenteeism and dropout.

While one way or the other SAI tackled some relevant individual and familial causes of school disengagement, sweeping social processes through which children miss classes and eventually drop out of school remain outside of the reach of the model (e.g. poverty and child labor). As most participants in this evaluation agree, SAI interventions are generally insufficient to make a quantitatively meaningful change, in a context where social determinants including poverty, child labor, low educational endowment of families, and precarious school environments reinforce one another.

Recommendations:

Focus on a much lower number of communities, while improving coordination among components;

Establish and use relevant 1) social and school indicators; 2) output and outcome indicators for monitoring SAI results; 

Invest most resources in direct activities with the right holders – the children and their families conducted in the communities; according to the data those activities generate best results;

Provide training to teachers based on a needs analysis and transform the training to address school and community specifics, as opposed to delivering generic inputs;
As socio-economic and family/community determinants of absenteeism and dropout massively exceed other types of causes, either 1) open the model to a cross-sectoral approach and build in interventions to deal with determinants previously unaddressed: material deprivations, low parental supervision of children, child labor; or 2) amend objectives to match the narrow range of determinants the current organization of the model does address;

Consider appointing various duty bearers as local coordinators depending on criteria such as the capacity to influence local determinants, permanent residence in the community, experience in working with socially excluded children, and, interest.

Integrate additional partnerships that could provide even simple poverty alleviation inputs to increase effectiveness by addressing the heftiest causes of absenteeism and dropout.
Link measures that discourage child labor, increase trust in the role of education for a better life and, after-school interventions helping children do homework: the rough living conditions, the low educational endowment of families support the recommendation of providing after-class support to at risk children; also, SAI should envisage supporting measures for reducing all forms of violence in schools, and for discouraging unhealthy behaviors to prevent parents from withdrawing children for safety reasons.

Lessons Learned:

A comprehensive approach is best suited to decrease absenteeism and drop-out in communities in which multiple and diverse determinants are simultaneously at play in generating those phenomena. Models of intervention should take into account the ensemble of determinants that lead to absenteeism and drop-out; e.g. education and health programs should be part of broader programs of community intervention, acknowledging that the relevance of formal education for pupils and families is strongly dependent on their circumstances of daily life, including available resources and competing priorities.

Models should take into account the recent history of similar inputs delivered locally, and their social context to avoid possible inflationary interventions that change their social meaning through extensive repetition.

In contexts in which multiple and diverse causes are at play, models should favor an in-depth rather than in-breadth approach, so as to identify and address local determinants properly, and avoid spreading out resources in interventions that do not achieve the required momentum and ownership for effective results.

To promote ownership, it is important to co-design activities with local stakeholders, to value and build upon local successes and local contributions to the design of programme deliverables. Continuously testing the interest in and local commitment to the intervention is essential for avoiding situations where beneficiaries relate to various outputs as “foreign” interventions.

It is important to develop and incorporate indicators for monitoring and evaluating effectiveness from the stage of model design. When possible, a randomized allocation of communities would enable better effectiveness and impact evaluations.

A focus on direct beneficiaries (children and their families) rather than on intermediaries (service providers) would ensure better ownership and increased effectiveness.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Romania

Region:
CEE/CIS

Theme:
Education

Type:
Evaluation Report

Language:
English

New enhanced search