We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Brazil: Evaluation of UNICEF Municipal Seal of Approval Initiative



Author: Márcia Paterno Joppert, Clarice Knijnik, Sueli do Couto Rosa, Lenira Machado, Marcia Maria Facchina and Ada Junqueira

Executive summary

Background:

UNICEF‘s Municipal Seal of Approval was launched in 1998 in the State of Ceará. In 2005, the strategy was expanded to the 11 states of the Brazilian semi-arid region and, in 2009, it was extended to the municipalities of the Brazilian Legal Amazon region.

The Seal is a certification process that stimulates positive competition among municipalities and rewards success with visibility for their achievements and efforts to ensure child and adolescent rights and the sustainability of their well-being. It focuses on three types of change: (i) in the lives of children (social impact); (ii) in public policy formulation, decision-making, management, monitoring and evaluation; and (iii) in social participation. A city wins the Seal only if it achieves a minimum predefined score in each of these three areas.

The strategy of the Seal works on the basis of the municipal reality, mobilizing municipal managers and technical experts to develop and implement public policies designed to ensure the full development and citizenship of children and adolescents. For this purpose, UNICEF, with its international experience, provides technical advisory to the municipalities, contributing to mobilize social actors to plan, implement and monitor policies and actions to ensure that the rights of children and adolescents are at the core of the public policy agenda.

At the state and federal levels, UNICEF advocates for political commitment to support the Seal initiative. State government and line ministries actively support the initiative with their own investments for children. UNICEF‘s role in brokering partnerships among the various parties has been noted as significant, and a key support for the implementation of national policies at the local level.

Purpose/Objective:

The evaluation was an independent assessment of the conceptual, technical and practical viability of the Municipal Seal of Approval in Brazil, obtaining lessons on what is working well, and identifying areas to improve and continue dissemination to new municipalities.

The purpose of the evaluation was to support continuing efforts of the UNICEF Brazil Country Office (BCO) to sharpen its focus on equity-focused municipal public policies that guarantee the rights of children and adolescents, especially those most disadvantaged and excluded, through the promotion of the Municipal Seal of Approval as a key strategy. The evaluation aimed to identify the added value of the strategy and its key contributions, and to document good practices and lessons.

Three areas were considered:

a) The Seal as a tool for mobilizing and implementing the equity agenda for children: at the conceptual level, the evaluation analyzed how the Seal evolved throughout the years and if it was substantively, logically and methodologically sound. It analyzed whether the Seal continues to have its own “appeal” and potential to position children’s rights in the agenda of municipal, federal and national governments;

b) Implementation of the Seal at local level and its broader impact on the implementation of international commitments (the CRC and the MDGs), national development priorities and public policies through upstream work and participatory processes: the evaluation looked at how the Seal was introduced, managed and implemented in the different editions; what impact it has had and how it relates to the policy issues;

c) Lessons learnt at the municipal, federal and national levels: the evaluation assessed how the lessons from past experience reflected on today’s Seal and how these lessons could be used to improve the current and future editions.

The results will contribute to the decision-making process and promote greater accountability for performance and achievements.

Methodology:

Overall approach: The evaluation is considered a forward-looking formative exercise, with a summative component.

Process: The evaluation process was be a mix of participatory evaluation, data and documentary analyses.

Data collection methods: The evaluation drew on a variety of data collection methods including, but not limited to:
-Desk-review of UNICEF documents;
-Data collected in official and national databases for selected indicators;
-Data collected in field visits and participatory processes;
-Semi-structured interviews (mainly by telephone/Skype or in virtual meetings/ discussions) based on questionnaires/survey instruments purposefully designed for implementation in the field by the evaluation team;
-Focus groups with groups of municipalities who were awarded the Seal, in order to gather qualitative information on the key factors that made success in the initiative possible.

A reconstructed Theory of Change and an evaluation matrix were prepared during the first stage of the consultancy, as part of the preparation of the inception report.
Data analysis consisted of consultations of databases and sheets prepared by UNICEF Brazil’s M&E team with key indicators used to assess results of previous editions of the Seal. Other resources included a systematization of performance and results prepared by an external consultant. These documents were used as a basis to assess trends, impacts and other major changes in the situation of children and adolescents. In addition, information obtained from interviews and participatory meetings with stakeholders (including beneficiaries) were essential to provide more qualitative inputs to complement the data.

During the evaluation process, the UNICEF Brazil M&E specialists and Evaluation Steering Committee validated methods, tools and findings to ensure that the data and information used and conclusions were consistent. This feedback was done in compliance with UNICEF Evaluation Policy, Procedures and Guidance.

Findings and Conclusions:

This independent evaluation confirms that the UNICEF Municipal Seal of Approval is an important strategy to:
- Support municipalities to implement decentralized public policies and national programmes; 
- Foster collaboration among sectors with key roles in the promotion and guarantee of children’s and adolescents’ rights, such as health, education and social assistance;
- Promote social participation and capacity development of rights-holders and duty-bearers to understand inequalities and vulnerabilities, as well as to claim, protect and guarantee rights for all children and adolescents;
- Increase knowledge and promote better use of social services for children and adolescents.
Most importantly, evidence points out that municipalities with a high level of success in implementing the Seal methodology have seen faster improvements in their social indicators as compared to other regions and to national averages.

The evaluation concluded that the Seal, in isolation, cannot be deemed responsible for promoting better quality of life for children and adolescents – this level of impact is the result of a combination of factors and policies implemented at the local level. On the other hand, the programme is recognized as a catalyst, creating a set of conditions for the development and strengthening of local capacities in the long run. It is these conditions that allow municipalities to improve the quality of public policy management, to empower local societies and to promote equitable practices to achieve equality.

Recommendations:

A large number of recommendations were included in the evaluation report. They are summarized below:

1) UNICEF should continue and expand the Seal initiative, taking specific actions to improve the Seal methodology by: promoting more synergy among participating municipalities; creating customized strategies that cater for specific needs of regions and municipalities, particularly those more vulnerable; enhancing capacity development; fostering collaboration with other local development initiatives; improving communications and participatory monitoring of progress and evaluation throughout the edition; enhancing ownership of the process and intersectoral collaboration at the local level; reducing time gaps throughout the edition.

2) UNICEF should seek ways to improve the management of the Seal, and to maximize the use of financial, human, administrative and technological resources;

3) UNICEF should invest more time, energy and resources into building partnerships at all levels (local, state, national, multilateral agencies), promoting advocacy and developing an exit strategy that takes into consideration future sustainability of results achieved;

4) Federal and state spheres of government should seek to enhance their engagement in the Seal process, as a way to support municipalities in the process, but also to build on the results achieved at local level to advance on national development goals for children and adolescents;

5) Municipalities should improve their participation in the Seal, by investing more time, energy and human resources in participatory processes, documentation, follow-up, monitoring and evaluation, intersectoral work, collaboration with similar municipalities or with vulnerable municipalities that could benefit from the help of others that have advanced further.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Brazil

Region:
LACRO

Theme:
Programme Excellence

Type:
Evaluation

Language:
Portuguese

New enhanced search