We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2017 Nicaragua: Evaluation of UNICEF and GRACCS innovation pilot projects.



Author: Análisis y Desarrollo Social Consultores.

Executive summary

 With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report, and the executive feedback summary labelled as ‘Part 4’.

Background:

The South Caribbean Coast Autonomous Region-RACCS is currently an important part of UNICEF cooperation in Nicaragua. Based on the joint work between the South Caribbean Coast Autonomous Regional Government and UNICEF, the process of producing a regional policy for children was established in 2013. This started with the production of a participatory assessment, which identified experiences and opinions about the different issues related to the rights of children through the application of a human-centred design approach.
A broad selection of actors participated, and the process resulted in the identification of five priority areas for children in the RACCS: prenatal health, birth registration, child labour and education, violence and maltreatment, and recreation and socialization.
To address these different problems and based on the outputs provided by the assessment, the Regional Policy and Strategy for Children and Adolescents-PERNA was designed, which includes as operational elements the implementation of six pilot projects for the different priority areas. These are being implemented by the GRACCS with financial and technical support from UNICEF.
The pilot projects are as follows: ToyContigo, TeleSalud, Real-time Monitoring of school management, Building and Plumbing Course with a Climate Change Focus, Injury Surveillance System, and Birth Registration. At the same time, since 2014 UNICEF has been supporting the Bluefields Indian and Caribbean University-BICU in the creation and development of an innovation laboratory.
For UNICEF, the objective of these initiatives or pilot projects is to test out models with a view to scaling them up in the future, with the aim of turning the initial prototype into a public intervention. In this respect, it was considered necessary to conduct an evaluation of the pilot projects to analyse and assess whether scaling them up is feasible and pertinent.

Purpose/Objective:

General objective:
Evaluate the implementation to date of seven pilot projects being implemented by the GRACCS, with UNICEF support, with a view to possibly scaling them up to a higher level.
Specific objectives:
• Evaluate the efficacy and efficiency of the implementation of each pilot project.
• Identify the opportunities and bottlenecks for scaling up each pilot project.
• Assess the degree of sustainability of each pilot project.
• Analyse the ownership of the projects by the communities and local authorities.
• Make recommendations on the kind of support each project requires from UNICEF.

Methodology:

Documentary analysis- From the start until the field work phase, the evaluation involved the documentary analysis of internal sources of information on the projects and secondary information sources, mainly focused on strategic and sectoral areas.
Individual and group interviews- 44 interviews were held during the field work. This social research tool has mainly been used with duty bearers and people involved in the programme implementation and decision-making processes. The information on the background, the local context and their vision of the sustainability of the actions carried out by the programme was highly relevant.                                                                
Focus groups- 31 focus groups were held, mainly with rights holders. These revealed the perspective of the subjects of the process in the field in terms of the relevance of the intervention for responding to the problem addressed by the different projects in each intervention municipality and their complementarity with other previous or current actions in the programme’s intervention area.
Direct observation - A total of 11 field visits were made involving direct observation and interviews and contact with actors.
Surveys- In the case of the Building and Plumbing Course with a Climate Change Focus and the Innovation Lab, surveys were applied as it was possible to access the project participants (25 people in the case of the training course and 28 in the case of the Laboratory). Although the sample was too small to be statistically representative, the results were considered of interest in terms of providing a view of the participants’ opinions on the different elements.
Methodological triangulation- As an analysis strategy, it was opted to employ methodological triangulation to facilitate comparing the information, ensure its reliability and maximize the qualities of the information gathering techniques.

Findings and Conclusions:

The pilot projects have grown out of different identification and formulation processes, but in all cases, it was observed that they are pertinent to the problems identified by the partner entities and by the rights holders themselves, as demonstrated by the high level of ownership observable among the rights holders and implementing technical personnel.
The vast majority of the projects evaluated can be considered to be socially innovative projects, as they influence or attempt to influence outputs, processes, positions and paradigms.
There are various shortcomings in the projects’ designs that make it complicated to objectively measure their effects or impacts as the definition of goals and indicators is barely established and the follow up centres on the implementation of activities rather than on results. The actual design of the projects has not been done from the point of view of a pilot project whose outlook includes exit, sustainability, continuation and/or scalability strategies.
It has been possible to observe the added value UNICEF has been providing in the identification of the actions, where there is an interest in the development of social innovation pilot projects. This interest is either direct, in the form of the identification of projects or indirect, based on the PERNA and support to specific projects. 
Most of the interventions have direct results among the population, which in many cases are highly important and achieved with limited levels of investment, making it possible to consider, indirectly, that the interventions could have positive impacts.
One limiting factor for the scalability of the projects is the need to consider the participation and authorization of the ministries and central authorities as a prior condition for the development of the projects. This situation has caused, or could cause, many projects not to fully develop as they do not have the approval to access information or communication from the central institutions.

Recommendations:

To analyse the regional-level scalability of the TeleSalud project, including ToyContigo in its development, with operational leadership from the Health Secretariat following an implementation and accessibility agreement with MINSA, incorporating broader strategies for strengthening the health brigades and the work with counselling bodies and clubs for pregnant women.
Establish an exit strategy for the Birth Registration project or define a strategy for the Supreme Electoral Council-SEC in the national sphere.
Facilitate the continuation of the Injury Surveillance System project in the aspect of the alliance among entities, with UNICEF playing a minor role, given that it appears complicated to involve all the bodies from the critical path in the use of EpiInfo.
In the case of the Innovation Lab, to strengthen the links with GRACCS by sharing the AWPs of the pilot projects between the Lab and Secretariats, disseminating the bottlenecks and seeking joint solutions. To promote its linkage/twinning with similar initiatives in other countries (UNICEF laboratories in Kosovo, Uganda, Zimbabwe, Copenhagen) or ones that already exist or are being created in this country.
With respect to the Building and Plumbing Courses, to ensure and maintain leadership from the Youth Secretariat in coordination with other government and civil society entities (Youth and Adolescent Network). To adjust the young people’s technical training to the occupational priorities they mention and to use the INATEC catalogue to develop them.
For the Real-time Monitoring, to continue strengthening the pilot project at least throughout 2018, a process that should be transferred from the DRASH to the Regional Education Secretariat during this period. It is not considered advisable to eliminate or add new SMS options; rather those already included need to be consolidated. Neither is it advisable to increase or expand the coverage to other municipalities in the RACCS until connectivity has been improved.

Lessons Learned:

The development of the pilot projects that have incorporated unidirectional information technologies is a complement that has been very well assessed by the health-sector professionals and the final users, with very low maintenance and operating costs and limited investment.
With respect to the projects’ implementation, the analysis of prior conditions is essential, particularly with regard to authorization for accessibility in the informatics sphere, but also with respect to defining sustainability and scalability strategies.
The following up and monitoring of a pilot project must be conducted with greater definition and evidence of the steps to follow in response to the different situations faced with regard to decision making and the formulation of subsequent projects along the same or similar lines.
Complementing the above, the design of pilot projects must take into account right from the start the elements required for following up and evaluating throughout the project management cycle. These must be defined in terms of indicators and those responsible for collecting, analysing and interpreting the information for decision making. In this way, the project will be able to readjust in response to circumstances and factors in the intervention context.
With a view to assuring sustainability, the installed capacities of the managing entities must be taken into account when formulating a pilot project. Based on this analysis, the relevance of including strengthening components that promote adequate institutional foundations for the project’s implementation and continuation should be determined.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2017

Country:
Nicaragua

Region:
LACRO

Type:
Evaluation

Theme:
Managament Excellence

Partners:
South Caribbean Coast Autonomous Regional Government-GRACCS

Language:
Spanish

Sequence:
2017/001

New enhanced search