Author: Oscar Solano
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report.”
"Return to Happiness" is a methodology used by UNICEF in its mission support the psychosocial recovery of children and adolescents living in areas affected by armed conflict in Colombia. This methodology is also used as a tool for preventing the recruitment of children and adolescents by illegal armed groups. Return to happiness is part of UNICEF's recruitment prevention projects in Cauca, Chocó, and Córdoba.
In Cauca, the evaluation process looked at phases I and II of the UNICEF—Association of Indigenous Councils of Northern Cauca (ACIN) joint work, between January 2006 and June 2008; in Chocó it reviewed UNICEF’s work with the Dioceses of Quibdó and Istmina between 2007 and 2008; and in Córdoba the MoU with the Diocese of Montería, implemented in 2007 and 2008.
This evaluation was conducted by an external evaluator, Oscar Solano, who was hired by UNICEF.
This consultancy embraces the technical criteria of pertinence, efficacy, efficiency, sustainability and impact suggested in the contract Terms of Reference, which appear in the Organisation for Economic Cooperation and Development (OECD) principles for evaluating development assistance and in the guidelines proposed in the Committee on Development Aid – CDA – (OECD, 1991), complemented by the special application of these criteria for cases of Evaluation of Humanitarian Action (EHA) (ODI, 2006).
The evaluation will operate along three central axes:
Does it meet its objectives?
Based on the declared objectives of the central protocol of the “Return to Happiness” strategy, we want to establish whether there is coherence between the announced objectives and the ones achieved – that is, whether each of the goals for achievement has been reached once the activities have terminated. To do this it is important to identify what was expected to be achieved and contrast that with what finally was obtained. Achievement indicators will be designed based on the plan of action, and beneficiaries will be consulted as to whether they perceive that the indicators are reflected in each one of the actions received. At this point it is important to triangulate the information with the beneficiaries’ environment, asking the participants’ significant adults – like parents, teachers, guardians, community leaders and social agents that interact frequently with the children and adolescents.
What has its impact been?
The impact of a project’s actions is measured by analyzing the difference between the beneficiaries’ living conditions before the intervention – that is, the reigning psychosocial condition – and the conditions following the project experience and participation in its actions. In other words the new, recuperated psychosocial condition. To measure the impact, the evaluation instruments will be designed to establish whether the children and adolescents perceive a change in their way of viewing and feeling their own lives in terms of identifying, controlling and overcoming their individual and collective emotional and relational affectations experienced as a result of political violence and natural disasters.
Are its results sustainable?
The measurement of the results is in consonance with the detection and identification of the new changes perceived in the individual and collective relations (the social fabric) found in the respective communities where the project has operated. For this purpose, instruments will be created to establish whether there are identifiable products that the project leaves the communities as a fruit of the activities and the participation of the beneficiaries.
At this point, we shall verify whether there is agreement among the sources in each of the evaluated communities as to the perception of a concrete product that is identifiable by the members of the community, and which they can name and describe in all its characteristics. The sustainability of the results will be evaluated beginning with the detection of who, within the communities, expresses commitment to the permanent and continuous dissemination of the methodology and the knowledge acquired, in addition to new and innovative proposals for psychosocial actions intended to be performed in communities, families and schools.
In addition, the commitment to sustainability will fall only on the community actors that have been involved and impacted by the Return to Happiness strategy. We also must take into consideration the participation – conscripted, if necessary – of official entities with missions of protection and promotion of the rights of children and adolescents, and of other actors that can be detected and that participate by contributing resources other than the ones provided by UNICEF.
The evaluation also must include the identification of good practices resulting from the successful execution of the project, the aspects that are proposed for improving it and the lessons learned in each intervention. For this purpose, the evaluation will have primary information obtained through structured interviews with the beneficiaries participating in each of the scenarios – children, adolescents and adults.
The evaluation incorporates elements of qualitative evaluation, based on tools of critical ethnography and participant observation . The most frequently used tools were focal groups, semi-structured interviews, conversation groups and observation. The evaluator found community assemblies (of 60 or more participants) in some municipalities of the three departments visited, with representatives of all the actors expected to participate in the projects: young volunteers from the Return to Happiness methodology, teachers, parents, and participating children. In addition, structured sessions were held with the leaders of the indigenous councils, both in Cauca and Chocó, to present the objectives of the evaluation mission and to answer their questions regarding UNICEF and its projects.
The evaluation had a limitation in its being performed ex post facto, with an average lapse of six months since the culmination of the agreements (June 2008). This is considered to be a limitation because there were actors, like teachers that know the projects and especially the Return to Happiness methodology, who were no longer in the zone or in the schools that served as project sites, because of transfers or simply having moved away, or because the directly benefited children and adolescents were no longer in the institution being visited, which especially was the case of some volunteers who had graduated from secondary school and had left their towns. Nonetheless, and to counter this limitation, spontaneous continuations of the projects’ recreational activities were found, making it possible to infer the methodology’s sustainability, such as the fact that youths were continuing their voluntary work with children, coordinated by a young woman who had participated in the activities.
The evaluation process included three weeks in the field, one in each of the participating departments. The visits were made between February and March 2009. The process also included an induction to the process by the Monitoring and Evaluation Officer and the Populations in Crisis Officer in UNICEF’s Bogotá office; establishment of basic agreements on the expected results and the methodology to be followed; a meeting with the two most important sponsors and defenders of Return to Happiness in UNICEF, both world-wide and in Colombia, Nydia Quiroz and César Romero; and, finally, direct coordination with each of the three departmental coordinators in order to coordinate each visit: Helena Romo in Cauca, Adriana Guerra in Chocó and Stella Díaz in Córdoba. Each coordinator decided which municipalities to visit and whom to visit, and informed each participating institution or actor of the upcoming evaluation visit.
General documentation of the established agreements was requested and received; and in the field more information was obtained in the form of books, readers, manuals and technical reports. The questions for facilitating the focal groups and group meetings were developed. The evaluator kept a field diary – a documentary centrepiece for permanently recording the experience of the current social context of the communities visited and their living conditions after having participated in and experienced Return to Happiness during the implementation of the projects. The data obtained were extracted and ordered; a critical analysis was made of the activities observed; all the material collected in the field, downloaded from the Internet and found at the UNICEF office in Bogotá was read and confronted and a triangulation was made of the compiled information, cross-referencing it to establish patterns, tendencies, constants – elements for use in preparing this report.
In the course of the evaluation in Cauca, seven municipalities and three reservations, the main headquarters of the ACIN, and six schools were visited; 22 individual interviews were performed; 5 community assemblies were convened, and five recreational learning demonstrations were observed. In Chocó six municipalities were visited, along with one indigenous community called Unión Wounaan, where a recreational learning campaign was observed and the evaluator stayed for two days. Four community discussions were held; two demonstrations with the Afro-descendant population and a focal group of parents were observed; and four semi-structured interviews were held. In Córdoba, four municipalities, one rural community, and five schools were visited, five semi-structured interviews were performed and five discussions were implemented with as many community assemblies.
Findings and Conclusions:
Regarding the social and human effects of Return to Happiness, this evaluation found positive evidence of criteria for belonging among the communities. The projects have met the psychosocial recovery needs of children and adolescents, since the communities in their entirety and the children themselves recognized it, and it was reflected in some children’s drawings produced under this methodology (Seen Annex No.3) with childish expressions far removed from themes of conflict or war. The concept that best indicates the pertinence of the activities is that of harmonization in Cauca, where the original approach in UNICEF’s version of Return to Happiness was adapted to the Nasa world-view, especially with regard to the adaptation to the cultural requisites of indigenous traditions in the area of health. In Chocó the projects also took care to adapt the parameters of intervention to the indigenous traditions that are most respected and adopted by the communities – as in the case of the strategy for nutritional and food security recovery, CARNUI, or Mobile Centre for Indigenous Nutritional Recuperation.
Of special interest is the methodology’s impact on the volunteer youths, the segment of beneficiaries that perhaps most vigorously defends the pertinence of Return to Happiness. Transversally – in all the departments – indigenous, rural peasant and Afro-descendant youths, after participating in the Return to Happiness as recreational therapists, have experienced an unexpected process of personal and social change in themselves and their environment, helping them to become aware of their role in defining their own life plans. This aspect was considered positively in the evaluation as an indicator of mental and psychosocial health in the community of youths. In other words, these young people express, organize, seek support for and prepare projects for life after secondary school. Also, this new psychosocial reality being experienced by the young persons has distanced them and protected them from the possibility of becoming involved in the illegal armed groups, which is one of the central objectives in each project’s strategy. In this regard, the testimonies and information collected positively mention the reduction of this tendency in these regions, previously marked by a high rate of recruitment.
With respect to the participation of girls and women, all the projects showed the participation of this sector of the population. There was no evidence of any kind of gender-based exclusion or discrimination. Among the youth volunteers, an important number of women were found to be participating in all three departments, both among the volunteers still in secondary school and among those that had graduated from school – some of them currently serving as voluntary coordinators of the groups of young persons that have wanted to continue their recreational therapy activities with the children in their municipalities. In Córdoba, women heads of household formed handcraft production units while accompanying the Return to Happiness processes in their children’s schools. These women admit to having participated during the planning stages in order to find ways of generating new income with the skills they had obtained.
Also found was the intention of involving girls in the projects to counter a situation that could lead to their becoming involved in the armed groups: romantic relations with members of the illegal armed groups. A pattern of men in arms seeking out adolescent girls has been clearly diagnosed in the project beneficiary communities in all three departments. In response, the Return to Happiness activities were oriented to protecting girls from this kind of menace by making explicit the facts associated with the unwanted involvement in armed groups and creating spaces for participation in the different groups of young persons, in order to foster cohesion and a sense of belonging to the community, which tends to have a protective effect on all the participants. When they join Return to Happiness, young men and women in the three departments form units of solidarity and mutual protection, endorsed by adults that have witnessed their process of personal development.
In terms of efficacy, the evaluated projects achieved the goals of the Return to Happiness’ intervention – an important achievement given the numerous elements required for implementing the methodology and the need to guarantee accompaniment and follow-up for optimal implementation of the strategy. It should be mentioned that the thematic “heart” of Return to Happiness, its reason for being, was understood by all the groups that learned about the strategy, were trained in it, used it and experienced it. In general, all the departments met the proposed levels of coverage of children and adolescents, with the creation of support groups with actors including parents and teachers and without the discrimination or exclusion of any group. In addition, triangulating the information made it possible to determine that Return to Happiness was able to fit functionally in the schools, either as a reference for the Community Education Plans (PEC), in Cauca, or for the Institutional Education Plans (PEI) in all the departments in general, or simply generating important changes in the way teachers work or in children’s demands that their classes be made more dynamic and enjoyable. The activities planned for implementing the Return to Happiness strategy were fully implemented in all the projects, indeed surpassing the level of execution of the other components.
Among the other components of the projects – other than the Return to Happiness strategy – it might be mentioned that no evidence was found to the effect that the goal of institutional strengthening was achieved, with respect to the role of municipal mayors’ offices or municipal secretariats of education, or active and evident support from State entities like the ICBF. In Chocó, the text of the agreements with the Dioceses reflects an expectation of working in network with multiple institutions like the SENA, the aforementioned ICBF or the personero’s and inspector general’s offices. (Although the priority of this evaluation was to determine the performance of the Return to Happiness strategy in the context of the selected projects, care was taken to gauge the community impact generated by working in association with the public and private institutions present in the regions.) Instead, the active and committed role of the participating schools prevailed – a role underscored spontaneously by the regional coordinators in the way they designed the agendas of the evaluation missions. In Córdoba, however, the evaluator noted the active role of the department’s public library network, and the University of Córdoba’s work in making complementary activities more dynamic and contributing to the impact of Return to Happiness, for example, by creating the Story and Reading Club (still functioning at the time of this evaluation) which included the publication of a book with a selection of stories by the children and adolescents. Also, at SENA’s El Porvenir training centre, graduates of the secondary schools of Tierralta are developing their life plans, training as interns in agricultural and livestock technicians.
The projects’ efficiency in using resources of time and money was satisfactory, supported by the degree of general satisfaction expressed by all the actors in the places visited, with respect to the projects’ impact, particularly Return to Happiness. Except for an isolated comment on the inconvenience caused by the delay of one of the disbursements by UNICEF in Cauca, the flow of resources ensured the implementation of the activities with no major hitches other than the ones to external causes like problems of public order or the condition of the highways due to the flood emergencies, which affected the rhythm of the technical teams’ orientation visits. As a reflection of the efficiencies encountered, we might mention the financial counterpart provided by the ACIN in Cauca, the assignation of teams of accountants (more than one functionary) to manage and protect the resources, and the clarity and organization of the information with which the members of the field teams performed their activities.
The projects are clear about their strategies for sustainability, and the evaluation missions was able to gauge how well the process is continuing, more than six months after the formally planned activities had ended. Thanks to the impact of the activities on the community, schools are spearheading the sustainability of Return to Happiness, or at least this has been the case so far. The strategy’s articulating axis is based on the identification and training of youth volunteers as recreational therapists, and we could see the initiative for continuing to train yearly cycles of 9th-grade youths who want to participate in the project, replacing graduating students. This is backed by the initiative of creating youth organizations which, among other things, will have a permanent line of work with Return to Happiness in the formalization of socio cultural animation activities for social and human development. Furthermore, indigenous authorities have given their backing to these UNICEF initiatives, since they have found them to be eminently respectful of their traditions and culture, as reflected in the respect and humane and professional quality of the technical teams in each region. There is functional and professional harmony between UNICEF’s functionaries and the local communities.
The limitation to sustainability that was detected is the low level of active participation by local and municipal authorities. Schools – all from the public sector – join the project, but not the leading authorities in the area. Although the Nasa indigenous organization in Cauca is strong, for example, the assignment of teachers for the reservations’ schools is performed by the Secretariat of Education in Popayán, and its inefficiencies in naming teachers or making school repairs, decidedly affected the sustainability of the projects in this component. The same can be said of health, since the projects, with the generation of the figure of the ACS – Community Health Agent – have the possibility of serving as an effective link for resolving problems in the area, as long as the respective public authorities understand and act in accordance. And the ICBF, the lead agency for public policies for children and youth, was perceived as being functionally distant from the indigenous communities. Nonetheless, the figure of the indigenous social pastorate of the Dioceses of Chocó is a significant resource for the communities.
This evaluation has collected enough information to affirm that the Return to Happiness strategy has had impacts on family, community and social life in the communities where it has been implemented. Once again, it should be emphasized that the youthful volunteers have played an important role as artifices of the general perception that the methodology does have an impact. with concrete results in terms of social cohesion, prevention of problems that are socially relevant to young people, facilitator of the development of life plans, social organization of young persons, involvement of parents and response by schools – because these young people contribute to raising the psycho-affectivity of children and adolescents, and feed back positively to their parents and teachers, encouraging them to pay more attention to them and thus affecting the quality of child rearing, on one hand, and the education that is offered, on the other. The projects have generated behavioural changes in the community and have made adults more aware of the rights of children and adolescents, and of their own role as protectors and guarantors of the integral development of the child population, even under conditions of emerging social violence. This awareness has increased among indigenous communities, where the authorities now include the themes of the rights of children and adolescents on the agendas for their Councils’ sessions.
Among the most important recommendations for these projects with respect to potential future developments, we may indicate the following:
1. They are projects with a very short average life, between 12 and 15 months, and processes of social transformation in areas of permanent conflict are constantly threatened. Longer cycles – for example, three years – would make it possible to create groups of beneficiaries whose options include guaranteeing the incorporation and formation of new generations of beneficiaries, providing a clear strategy for sustainability.
2. The projects evaluated along with the Return to Strategy have developed productive initiatives, with handcrafts for women or agriculture in the schools, especially in Córdoba. But there is no project for promoting the commercialization or marketing of products, or the generation of cooperatives or other instances of solidary economy, which threatens the sustainability of the productive efforts that were observed. This development component must be made more explicit in the overall consideration of future projects for fostering their integral effect on the psychosocial reality of the benefited communities.
3. Participation and coordination with State entities that have direct responsibility in the great majority of the project themes must be a functional coordination, and not only a declaration in official documents. In other words, it is necessary for the coordination to be visible and operational. These entities must guarantee their presence in the areas of influence with functionaries that are properly trained in matters of social and community development.
4. International technical cooperation in the areas visited is led by UNICEF, with a scarce presence of other organizations like Misereor or Diakonie. In these regions, more social and economic development must be promoted, since it is necessary to take advantage of the overcoming of children’s and adolescents’ psychosocial obstacles, where the communities have come together and there are leaders that understand how to work for the collective good.
5. The counterparts already are aligned with many of UNICEF’s objectives, in the broadest possible sense of promoting and protecting the rights of children. The ACIN and the dioceses are present in the territories and continue to work with the communities, and their multiplying effect is permanent. This guarantees the sustained appropriation of new knowledge and experiences derived from technical cooperation.
6. Return to Happiness is not necessarily a finished product. It is under permanent construction, since the recreational strategy should be included in educational and learning processes in a more comprehensive and systematic way. Psycho-socio-cultural animation in education communities is a permanent catalyser of learning achievement and training in children and adolescents. It is important to promote psychological and educational research around the strategy, with the regions’ universities.
7. The Return to Happiness methodology is solid, and the youth volunteers agree that they are ready for more and manifest that they need more extensive training over time, since they are attracted by the psychosocial themes and would like to contribute a bit more to their direct beneficiaries. On the other hand, the teachers feel the challenge of incorporating new methodologies into their teaching/learning practice, based on recreational practice, and this would be a significant contribution from the universities, as mentioned.
PDF files require Acrobat Reader.