We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2016 Guatemala: C4D Nutrition Evaluation: Evaluación de la Estrategia de Comunicación para el Desarrollo en Seguridad Alimentaria y Nutricional



Author: David Grajeda, Sylvia Campos, Marco Tulio Escobar

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

In the framework of the Zero Hunger Pact Plan implemented by the last government (2012-2015), UNICEF led the implementation of the "Food and Nutritional Security Communication for Development Strategy" project, in coordination with the Food and Nutritional Security Secretariat (SESAN), the Ministry of Health (MoH) and the Ministry of Education (MoE). The main objective was to provide information at the local level to prevent chronic malnutrition, especially among children under the age of 2.  The strategy employs a participatory and culturally acceptable methodology, promoting improved feeding and nutritional practices and empowering local actors as builders of their own destinies, rather than as objects of good interventions.
The Project proposed 3 main outcomes: a) information and basic knowledge for families on good feeding, nutrition, exclusive breastfeeding and pediatric care; b) strengthened communication capacities and structures of the Municipal Food and Nutritional Security Commissions (COMUSAN); (iii) training of authorities, community organizations and adults in charge to identify the causes and effects of deficient nutrition of family members.
During 2 cycles (2010-2012 and 2012-2015), the project concentrated on 8 municipalities in the province of Totonicapán: Momostenango, San Andrés Xecul, San Bartolo Aguas Calientes, San Cristóbal Totonicapán, San Francisco El Alto, Santa Lucía La Reforma, Santa María Chiquimula and Totonicapán (capital). Project implementation was ensured by the COMUSAN Communication Subcommittees in these municipalities, with the support of UNICEF, the  Center for Communication for Development (CECODE), and technical advisors and decision-makers of the FNS sector institutions (SESAN, MoH and MoE).

Purpose/Objective:

The overall purpose of the final evaluation conducted during July–October 2016 was "to evaluate the results obtained and offer recommendations to improve the design, sustainability and scaling up to other coverage areas of the country, bearing in mind four criteria established by the Organization for Economic Cooperation and Development (OECD) for this type of exercise: relevance, efficiency, efficacy and sustainability."  The specific objectives were as follows:

  • To evaluate the strategy's contributions to changes in feeding and nutritional practices , including hygiene, complementary foods, exclusive breastfeeding and pediatric checkups, offering evidence of the progress achieved in comparison with the initial situation shown by the baseline and the effectiveness of its Theory of Change (ToC).
  • To evaluate the strategy's contributions in terms of impact on the living conditions of the target populations from the beginning of its cycle, particularly on reducing chronic malnutrition in boys and girls under the age of 2.
  • To identify the best practices and lessons learned from implementing the strategy that can potentially be replicated and taken to scale in other areas of the country, including those obtained by the actors involved, particularly the members of Communication for Development Subcommittees that report to COMUSANs.
  • To analyze the factors that have contributed to, or have limited, the strategy's outcomes, bearing in mind, among others: a) the level of participation and ownership and the roles of local and institutional players; b) flexibility and adaptability to the beneficiaries' needs; c) the degree of interagency coordination; and d) the social, political, institutional and cultural context.

Methodology:

The evaluation team used the following combination of quantitative and qualitative methods to gather reliable information:

  • Documentary review and analysis: A number of project and references on the Food and Nutrition Security (FNS) sector were reviewed. These were classified into 3 groups: a) documents on design and implementation; b) partners' strategic and program documents; and c) specialized documents such as studies and reports on good Communication for Development and FNS practices.
  • Survey of a sample of families: A survey of families participating in the project's communication activities was designed and administered. The sample of communities was selected through the statistical sampling method, proportional and randomized, and stratified by municipalities. Thus the size of the sample was 10% of the universe, equal to 11 communities distributed in each stratum, made up of the eight municipalities. Eight families were selected from each sample unit for a total of 88.
  • Semi-structured interviews with key actors and partners: Five types of actors were interviewed: a) UNICEF team members; b) central and provincial-level officials and technicians of sector institutions; c) municipal and community-level actors; d) members of COMUSAN Communication Subcommittees; and e) CECODE advisors.   
  • Focus groups with key actors: The technique was designed to obtain information from actors with a homogeneous profile involved in the project, in order to obtain perceptions and opinions regarding the different variables considered.

Findings and Conclusions:

Satisfactory results were achieved in: a) harmonization with the FNS political framework; b) consistency with the needs and rights of target families; c) adaptation to the cultural references /backgrounds of the indigenous communities served; and d) technical and scientific relevance of the theory of change (ToC) to achieving the expected outcomes. The institutions' communication capacities were strengthened, and a reasonable degree of efficiency resource use was achieved, thanks to communication activity management modalities.

Unsatisfactory elements included the absence of an outcome framework that clearly and consistently expresses the cause-effect relationship between interventions and expected changes.

Achievement of planned outcomes was satisfactory level in terms of:

  • Impact: A consistent reduction of chronic malnutrition in boys and girls <5 is seen in most municipalities (though not solely attributable to the project);
  • Effects: changes in specific prioritized practices are evidenced;
  • Products: The project satisfactorily delivered basic information and knowledge on good feeding, nutritional and health care practices for mothers and infants; strengthened COMUSAN communication capacities; and developed the ability to reflect on the causes and effects of the problem of chronic malnutrition.
  • Additional outcomes: a) the communication capacities of sector institutions were developed; b) the existence of traditional practices among public institutions and local governments limited the scope and sustainability of the outcomes achieved; and c) the loss of momentum and initial energy is seen among the main actors, especially at the local level.

Recommendations:

UNICEF: a) To review the existing methodology to integrate with other strategies which are being implemented; b) To develop a technical document that systematizes the best practices and most important lessons learned resulting from the project's experience; c) To incorporate the communication strategy’s Theory of Change into the design of new projects developed in the framework of the National Strategy; d) To design an FNS communication project that supports the efforts of government institutions to implement the National Strategy for Chronic Malnutrition Prevention (2016-2020); and e) To collaborate with the other UN agencies on political advocacy to address the structural causes of chronic malnutrition.

Sector Institutions and Local Governments: a) To develop the Theory of Change or logical framework of the "behavior change" component of the National Strategy; b) To reorganize and continue supporting COMUSAN Communication Subcommittees in the project area; c) To raise awareness and secure the commitment of the new municipal authorities to sustaining the subcommittees; d) To design and implement sustainability strategies to ensure the continuity of ongoing project outcomes in project areas; and e) To resume and strengthen sector coordination efforts to better coordinate the roles and responsibilities of different institutions, maximizing efficiency in the use of resources and efficacy in the achievement of results.

Lessons Learned:

Relevance and Effectiveness of the Theory of Change: The project’s ToC contains several elements that are firmly based on technical knowledge on the subject matter and that explain its operational effectiveness, particularly recognition of the target individuals' prior knowledge and communication habits as a starting point for the design of activities and materials. In addition to inclusion of the audiences, other strategic elements that have also worked well, including the incorporation of culturally-adapted play ingredients and the creation of associative cooperation structures among institutions and between these and the communities.

Results Framework and RBM:  UN corporate standards on the project development cycle and especially the RBM methodology were ignored in the design of the FNS communication strategy. There was no results framework with a logical chain that would make it possible to visualize how the products would contribute in the long-term to attaining higher-level outcomes such as effects and impacts. The project team used community monitoring tools to reduce the negative impact of this shortcoming.

Empowerment of Beneficiaries as an Added Value of the Theory of Change of the Project: The project's approach and communication tools made it possible to develop the autonomy and decision-making capacities of the participating local leaders, including women and children. As noted above, evaluating persons and their cultural references as the starting point for prioritizing their demands and planning communication activities was critical to this outcome.

Development of COMUSAN Communication and Coordination Capacities: The decision to strengthen the communication and coordination capacities of these structures proved effective and succeeded in coordinating joint actions using the same language. This success is evidenced by the fact that the subcommittees' experience was replicated in other municipalities.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2016

Country:
Guatemala

Region:
LACR

Type:
Evaluation

Theme:
Advocacy and Communication

Partners:
SESAN

Language:
Spanish

Sequence #:
2016/002

New enhanced search