Author: Bryon Gillespie
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
Aims of the Evaluation
The Terms of Reference for this evaluation required it to:
1. Assess the extent to which the project achieved its objectives.
2. Assess the outcomes and immediate impacts for project beneficiaries.
3. Examine external factors that affected performance of the project.
4. Provide recommendations for continuation.
Data for the review included qualitative information gathered through interviews and focus groups as well as quantitative data gathered through a survey of 326 beneficiaries and non-beneficiaries of the project.
The Youth Empowerment and Security Project is a joint initiative of the World Bank and UNICEF. The World Bank funded the project for 2.1m USD and it was implemented by UNICEF over 2.5 years from November 2008 to April 2011. It aimed to enhance the security of at-risk youth from different backgrounds in 5 republics of the North Caucasus region of the Russian Federation: North Ossetia, Ingushetia, Kabardino-Balkaria, Chechnya and Dagestan. Activities were carried out under four project components:
1. Youth Centres: were established in Ingushetia and Kabardino-Balkaria to provide complementary learning as well as opportunities to engage in safe and inclusive leisure activities.
2. Peace and Tolerance Programming: engaged over 6,000 young people across the region through interactive events and volunteer programmes. The aim of the activities was to increase social cohesion and understanding between people of different nationalities, ethnicities and religious groups.
3. Youth Grants: a competitive grants programme provided NGOs from across the region with funds to deliver youth-focused projects. In addition to providing benefits to young people, the aim of these grants was to build the organisational capacity of the NGOs and strengthen civil society support for young people. The project also ran a competitive small grants programme for aspiring young entrepreneurs.
4. Capacity Building of Ministry Staff: the project provided ministries with basic office equipment, training to increase skills, and a study tour to Syria to exchange ideas and see examples of youth programmes. By working closely with ministry staff on initiatives funded under the grant, UNICEF provided ministry staff with opportunities to "learn by doing" and develop ideas for youth policies and programmes.
The programme has met its objectives and the targets agreed between UNICEF and the World Bank. Following the criteria set by the OECD DAC, the evaluation found the following:
1. Relevance: the aims and approach of the project are highly relevant to the needs of the region. The project clearly benefitted from the 2005 World Bank study mission investigating inter-linkages between youth and security. UNICEF has also been a key factor for the project. The relationships it has built over more than a decade of work in the region allowed the project to remain connected to stakeholders and well-adapted to local realities.
2. Efficiency and Effectiveness: the project has met the targets set for each component identified in the results framework. The UNICEF team is small and has had to focus very clearly on delivering the essentials, which has resulted in a high degree of operational efficiency. A downside however is that the team has not developed a results-based management system. This has limited its ability to clearly articulate programme strategy or generate evidence that would help it influence funding decisions of its government counterparts and other possible funders.
Some elements of the programme have run more efficiently than others. A major cause of inefficiency was the delay in project implementation between 2006 and 2008 (prior to UNICEF’s involvement) which allowed key officials in Ingushetia to change and government commitment to wane. This resulted in a drawn-out effort on UNICEF’s part to re-build support among key individuals. Nevertheless, inefficiencies like these do not appear to have affected the quality of programming as perceived by beneficiaries. All stakeholders report high levels of satisfaction with the quality of the project outputs.
3. Outcomes: the programme is delivering outcomes with each of its main stakeholder groups.
Youth engaged through components 1 and 2: the project is bringing about changes in attitudes, perceptions and behaviours of young people. It is also having an effect on a wider group of indirect beneficiaries as participants influence their friends and family. The kinds of changes that are taking place include increased participation in community activities, increased ethnic tolerance, and the perception that opportunities for young people are expanding. Young women are more enthusiastic about promoting the programme than young men, and while support is extremely strong in Kabardino-Balkaria, it is slightly less strong in Chechnya and Ingushetia.
Ministry staff: trainings in topics like strategic planning have changed the way ministry staff organise and go about their work. Policy papers and the ideas that staff have for programmes highlight that ministry priorities are congruent with the Bank/ UNICEF programme, indicating that the project has influenced policy thinking.
NGOs: in order to qualify for the grants, several youth-focused NGO grantees were required to improve their internal administrative systems. This allowed them to improve their efficiency and better position themselves to receive funding from international donors. The grants also allowed several better-established NGOs to strengthen their focus on young people. Several supported NGOs are beginning to spontaneously collaborate with one another, providing encouraging early signs of a strengthening civil society sector to support youth.
Young Entrepreneurs: although is still very early days for many of the businesses that were started, there are encouraging early outcomes. These include increased personal incomes, self-confidence and respect from others. Grantees are also providing jobs for other young people.
4. Sustainability: much of what the programme has built is intangible, and the departure of the UN and other international organisations at the request of the Russian government poses a number of threats to the sustainability of results achieved. These include:
Declining support for civil society: Russian plans for developing the region focus heavily on bringing investment through tourism. The exit of the UN and others represents the loss of a key source of support for civil society, which also threatens to undermine what has been achieved through the project.
Federal concerns over sovereignty: due to perceptions expressed within Moscow that international assistance is sometimes a covert means for Western countries to influence Russian policy, officials in the region go to great lengths to be seen to be immune from external influence. This makes them reluctant to adopt ideas that are strongly associated with external agencies like the World Bank and the UN, and has made it difficult for UNICEF to gain traction in their attempts to arrange the handover of activities. There is a real risk that project activities strongly associated with the World Bank and UNICEF will not be taken up and continued.
Loss of a convenor: UNICEF has developed good relationships with its government counterparts. Its independence and strong reputation has allowed it to bring stakeholders from different republics together in ways that government agencies would struggle to do on their own. With the departure of the UN and other international organisations, it is not obvious who will be in a position take on this role.
Loss of momentum: UNICEF has initiated several promising streams of work which should be continued, but there is a danger that with its departure, the energy behind them will dissipate. Examples include the volunteer movement, the cooperation that has begun to develop between NGOs in the region, and continued high quality support for young entrepreneurs.
The programme did a number of things very well, which are highlighted as lessons for the future:
1. Maintain momentum with government partners: UNICEF’s work in Kabardino-Balkaria to establish the youth centre has resulted in a very successful partnership between three levels of government. Their experience developing this project should serve as a model for how to co-develop projects with government in the future.
2. Create programmes that fit national frameworks and priorities: the entrepreneurship grants programme was designed within the framework that federal authorities provide to republican governments. Approving the UNICEF programme helped republican governments meet their obligations to Moscow, which facilitated the process of securing the required authorisations.
3. Create synergies with other UN organisations: by working with other UN agencies in the region, UNICEF was able to increase the benefits available to its beneficiaries
4. Adapt the peace and tolerance training method to other locations: UNICEF’s North Caucasus office has been developing its peace and tolerance training since 2005. External stakeholders highly praise the approach, and the training manual should be examined to see how it could be translated and adapted to other locations.
The programme has managed to engage approximately 10,000 young people through its activities. This is an impressive accomplishment but there are 1.75 million young people in the five Republics. In order to bring about population-level changes, future programmes should considering the following.
1. Build stronger mass communication campaigns: communications efforts have done a good job of promoting the activities under the programme but greater focus should be put on using media to influence attitudes at a population level. Stories focused on young people and their accomplishments, outside of the frame of UNICEF and the World Bank, would be more effective for creating a sense of expanding opportunities, and for inspiring other young people to take steps themselves.
2. Create an independent youth-friendly brand: the project was run under the banner of the World Bank and UNICEF. Future programmes should consider creating a separate identity for campaigning activities. Youth would likely connect more strongly to a locally derived brand, and would make it easier for project activities to evolve into an indigenous grassroots peace movement. It would also make handover simpler for government counterparts. Lastly, the image of an independent brand would not be damaged when the project closes.
3. Develop outreach strategies around priority segments of the population: not all people react to outreach in the same way. Future programmes should consider using market research techniques to segment the population and build tailored strategies for reaching priority groups. An approach like this would have allowed UNICEF to better define who youth most at risk are, and to unite partners in efforts to engage people of that profile. It also would have allowed the programme to better understand why uptake was so much stronger among some parts of the population than others. Segmentation also offers the potential to identify and recruit young people who are likely to promote the programme to friends and family, which would greatly accelerate uptake.
4. Use network effects to build momentum: one of the main reasons that young people value the programme is that it offers a window into a wider social world. By making the programme’s ability to help young people connect more obvious, it may be possible to radically scale up youth participation through network effects.
5. Make more strategic use of M&E: one of the shortcomings of the project is that it lacks a system for tracking higher-level results. Future projects should include an RBM system, as this would help them adapt to meet emerging realities. Having evidence about what works and why would also help influence programme and policy decisions of government counterparts and other funding organisations.
PDF files require Acrobat Reader.