We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2011 Tacro Region: An Evaluation of UNICEF Programming at Sub-National Level in Latin America



Executive summary

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:
UNICEF has seen some spectacular success in Latin America, in support of sub-national programmes that work well and have become a template for national programmes. Two important examples are in Brazil, Bolsa Escola; and in Peru, Buen Inicio. The former started in the Federal District of Brasilia and the State of Minas Gerais, to get children out of work and into school. It then became a national programme before combining with other subsidy and cash transfer programmes to evolve into Bolsa Familia. The latter, in Peru, was a nutrition programme that started in the Southern Andes and became the template for the acclaimed Crecer national programme. In both examples UNICEF was involved from the initial stages. For Bolsa Escola this was often in association with other agencies in developing concept papers or assessing results. At different stages from 1995 onwards, the World Bank, ILO and UNESCO were all involved with promoting and supporting the programme and its expansion. This highlights two important points regarding UNICEF cooperation with countries in The Americas and Caribbean Region: firstly the importance of piloting innovations, whether they emerge from individual ideas, local practice or are taken as a learning experience from elsewhere; and secondly, a fixed sub-regional presence by UNICEF is not a prerequisite to do this. In both the above cases the UNICEF country office undertook the major work of supporting the pilot, whether discussing design, partnering in implementation or evaluating progress.

Purpose/Objective:
In order to develop a more coherent approach in decision making on the type of sub-national presence that would best fit needs in different circumstances, this evaluation aims to:
a) Evaluate the effectiveness and relevance of UNICEF programming at sub-national level;
b) Examine the efficiency of programming modalities and operational structures;
c) Recommend criteria for different modalities and structures for sub-national work.
In the Terms of Reference each of the above is then broken down into a series of general and specific questions. These are mainly qualitative and subjective in nature. An attempt was made to add a quantitative and objective dimension by looking at specific transfers of local initiatives, experiences or ideas to a scaled-up or national level, or, indeed, international level. (See Annex 2 for details of the Terms of Reference and the questions)

Methodology:
Each of these objectives had a series of general and specific questions laid out as a basis for review.

There were three phases to the evaluation: a desk review, a series of field visits and interviews, and the preparation and presentation of the final report.

The desk review covered the following documentation: draft Country Programme Recommendations, Country Programme Action Plans, Country Programme Management Plans, Mid-Term Reviews, Annual Reports and statistical annexes, as well as other general documentation. These were available for the eight countries specified: Argentina, Brazil, Colombia, Ecuador, Mexico, Nicaragua, Panama and Peru. Subsequently the consultant also reviewed Bolivia as well as the draft UNICEF Internal Audit report on sub-national offices. This led to a decision to focus on countries with different kinds of modalities for a sub-national presence: Brazil with established Zonal Offices; Argentina operating exclusively from the capital with some previous experience working with an outposted consultant; Peru with both established offices and outposted consultants; Mexico working through implementing partners and with experience of working with a joint UN office; and Nicaragua working through a joint UN office.

In addition, a number of subsequent questions arose from discussions with the Regional Director and his staff as well as with the senior auditor in UNICEF responsible for reviewing the role and function of sub-national offices. These questions included the need to explore the urban question in some depth, the rationale for selecting geographical focus areas, if there is a difference over time between areas of UNICEF focus and other areas, if good indicators have been developed to be able to show what the targets are and the subsequent results.

From the list of countries above, visits were conducted to Brazil, Argentina, Peru, Mexico and Nicaragua. However, in Argentina and Peru the timing of the visit only allowed for discussions with UNICEF staff, and in Argentina with the Resident Coordinator of the UN system. Nevertheless, the visits to the other three countries allowed for sufficient in-depth discussion with counterparts and other non-UNICEF respondents to be able to derive useful conclusions, while the discussions with UNICEF staff in all countries and during the Regional Management Team Meeting in Lima permitted a much clearer vision of expectations from sub-national programming and sub-national presence. The information obtained from the different interviews and discussions was then validated by looking at available data and reviewing articles and publications referring to specific programmes and results.

Findings and Conclusions:
General Findings

a) Evaluate the effectiveness and relevance of UNICEF programming at sub-national level;
b) Examine the efficiency of programming modalities and operational structures;
c) Recommend criteria for different modalities and structures for sub-national work.

General Conclusion

For UNICEF in Latin America sub-national programming through different programme modalities and types of presence has become ever more important as more and more countries decentralize their systems of planning, decision making and resource allocation to different levels of governance. There are ways of working with that at national level that have not been much explored. National or State/Provincial training programmes and annual meetings for mayors or municipal staff serve as potential entry points for introducing ideas and practices of interest to UNICEF. This would help address the issue of frequent changes of officials and counterparts as they could become regular venues for information sharing. Yet country offices are also urged to apply the Equity agenda countrywide. This entails choices and this evaluator believes that these choices of emphasis need to be continuously re-assessed. As pointed out in the Findings above, UNICEF Representatives need to look at overall need in the country first, through their office Situation Analysis or periodic reviews, before referring to specific population groups. At the same time, opportunities emerge which create an improved environment for children (as is the Juvenile Justice programme in Yucatán). As priorities shift it has been difficult historically to not continue with what has gone before. Therefore, few zone offices have been closed. For some the approach has been to expand zone offices in order to address new identified gaps, rather than to prioritize. There has also been little thought given to how long a sub-national presence should stay and what indicators to use to determine success or failure. In the current administrative environment of decentralization it should be mandatory to set not just programme goals but specific outcomes and outputs to be achieved and their time frames for all proposals to open or continue with any sub-national presence. Country offices should be continuously assessing the situation of children and evaluating whether the type of presence or engagement, whether based locally or nationally, is appropriate and achieving the desired results. This should be linked to a clear exit strategy against the time frame set for any type of sub-national engagement. This will allow for a clearer assessment of the need for a presence and the kind of presence needed. The impact of sub-national work on policy at State and National level has always been a feature of the role of sub-offices in Brazil and continues to be so. The potential for such influence at national level was also clearly seen in Mexico. In Nicaragua the need is political; to redress past imbalances in society. In that sense the presence is a sensible response to an identified need but should be within a limited time frame and with objectives that are clearly articulated in terms of access to public goods and services.

There will be a continuing need for a sub-national presence in many places but it needs a clearer programmatic formula for decision making as it can also divert attention from where children are suffering vulnerability and distress in the greatest numbers. The evaluator has attempted to provide such a process chart for decision making (see pages 34 and 35).



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2011

Region:
TACRO

Theme:
Area-based Programmes

Type:
Evaluation

Language:
English

New enhanced search