We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2018 Kazakhstan: The Evaluation of Adolescents Mental Health and Suicide Prevention Pilot



Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

UNICEF in Kazakhstan has developed, piloted and is bringing to national scale an innovative project which seeks to reduce the risk of adolescent suicide and improve their mental health and well-being. Kazakhstan has been identified by the World Health Organization (WHO) as having some of the world’s highest suicide rates for young adults, adolescents and younger children. The issue is highly sensitive and stigmatized, which has acted as a barrier to raising the profile of the issue and implementing potential solutions.
UNICEF Kazakhstan firstly commissioned a study on prevalence, risk and protective factors of suicide in Kazakhstan in 2012-2013, followed by additional research on the Kyzylorda and East Kazakhstan regions. UNICEF then developed a comprehensive pilot programme (the ‘Adolescent Suicide Prevention Project’) which ran from 2015-2017. This report presents the results from the evaluation of the pilot, but with a view to generating insights that could inform the future development of the programme as it is brought to scale across Kazakhstan, responding to the urgent public health need in Kazakhstan.
The evaluation found that there was good evidence that the ASP had made a strong contribution to a viable model to better identify and refer adolescents at risk of self-directed violence, as well as which improved indicators of mental health and well-being (such as anxiety and depression).
This evaluation is summative in nature: it was conducted once the programme ended and evaluates the programme’s changes at impact, outcome and output levels. Our recommendations are forward-looking, informing the future development of the ASP programme (which is currently being rolled out at national level), and are structured around: (i) capturing learning from the ASP pilot on UNICEF’s role and future programme implementation; (ii) how the components of the programme could be improved; and (iii) issues to consider in the scale-up and replication of the model.

Purpose/Objective:

The evaluation purpose was to (a) assess ‘the pilot’s viability as a cost-effective intervention for adolescents’ suicide prevention and mental health promotion’, and (b) ‘document the Kazakhstani experience in customizing methodologies and approaches with possible use by other countries confronting similar issues in respect to adolescents’ mortality and mental health’.

Specific objectives of the evaluation, as presented in the ToR were to:

  1. To identify the extent to which the ASP programme in Kyzylorda Oblast has achieved its objectives, and the results that have been achieved to date, including any unintended results from the pilot, as well as their particularity at outputs at the individual, household and community levels.
  2. To identify the opportunities and constraints the programme has faced and draw lessons and good practices from them.
  3. To assess the cost benefit of the pilot (final costing methodology depends on availability of data).
  4. To identify the extent to which cross-cutting strategies such as a human-rights-based approach, results-based management and gender equity have been mainstreamed in the design and implementation of the programme.

Methodology:

The overall approach to this evaluation revolves around a theory-based design, which involved reconstructing a Theory of Change (ToC) of the programme, and developing an evaluation framework and Key Evaluation Questions (KEQs). This approach was further complemented by a mix of quantitative and qualitative methods which included: key document reviews, quantitative data analysis, qualitative data collection and analysis through: Key Informant Interviews (KIIs); Focus Group Discussions (FGDs), with 120 respondents interviewed across 15 sites in Kyzylorda Oblast, excluding interviews at the central level; and a cost analysis of the ASP pilot.

Limitations
The main limitation of this evaluation is the lack of consistent and reliable data made available to the evaluation team. The strongest evidence presented in the report mostly relies on the primary qualitative data collected by the evaluation team during a country visit and is complemented by the outcome data which tracks changes in adolescents across the pilot period.

Findings and Conclusions:

There is considerable and robust evidence that the pilot achieved the majority of its aims and has had a positive impact, supported by both qualitative data and the quantitative outcome data.
Five key components were identified: (1) Awareness-raising materials about mental health are disseminated to adolescents, (2) Adolescents are screened for potential mental health problems, (3) Gatekeepers and health providers participate in mental health awareness-raising training, (4) A referral system is in place to provide adequate mental health services, and (5) Evidence and monitoring data is being collected and analysed.
These components were underpinned by an effective multi-sectoral collaboration, which is one of the key strengths and key successes of the programme.
The ASP pilot also contributed to increasing awareness of mental health issues among adolescents (and their parents), decision makers, gatekeepers, and health providers. Furthermore, the programme has also clearly strengthened the skills of adolescents to cope with mental health problems and provide them with knowledge of where and how to seek care.
Other key achievements of the programme include the strengthening of the health and education sector to address mental health needs, and the implementation of a referral mechanism for primary and mental health care. However, both of these achievements have been against a backdrop of challenges.
Interestingly, the ASP pilot has also generated unexpected effects that highlighted the sociocultural and contextual factors contributing to poor mental health in adolescents.
Cost analysis
UNICEF was able to leverage government resources at both federal level, for funding the pilots, and local level, in the implementation of the programme. Despite the local government costs not being included in this cost analysis, the ability of the programme to leverage these funds demonstrates good value for money and bodes well for buy-in and sustainability.

Recommendations:

Our recommendations are focused around three core areas that respond to the evaluation’s objectives: i) capturing learning, ii) improving programme components and iii) scaling and replication

  1. UNICEF should strengthen evidence generation for piloting of innovative approaches, particularly in health systems, and to test what works
  2. The programme’s core aims and objectives should be well articulated and clearly and consistently presented to all stakeholders
  3. Minimise unnecessary referral to psychiatric services through enhanced training for GPs and, to a lesser extent, school psychologists
  4. Improving treatment options for those with complex needs
  5. Improve access to pharmacological treatment
  6. Provide a mechanism for debriefing to carers of those at risk of suicide, so that they can receive professional support
  7. Providers and primary health care practitioners in particular should attend the training in full, to build confidence with working with those at risk
  8. Improve understanding of the gender-based norms which affect incidence of self-directed violence, and further develop a gender differentiated approach
  9. Develop more differentiated screening models which can identify complex needs
  10. Enhance the curriculum for pre- and in-service training for educational psychologists
  11. Strengthen engagement with policy-makers and decision-makers to ensure that the mental and primary health care systems are adequately resourced to meet demand
  12. UNICEF’s role should shift from implementer to lead technical assistant, capturing what is working for adolescent mental health and well-being as the ASP goes to scale in Kazakhstan
  13. Continue to use local organisations as advocates and implementers of the ASP
  14. Continue to improve the use of evidence and data, particularly through a strengthened monitoring and evaluation framework

Lessons Learned:

Itad team worked with UNICEF leads and key stakeholders, such as the Ministry of Education and Science, the National Centre for Mental Health and the NGO “Bilim Foundation” to identify the ‘lessons learned’ for future programme development, building on insights from the pilot.

Lesson 1: Over-coming stigma related to mental health
Lesson 2: The importance of leveraging political will
Lesson 3: Use a multi-disciplinary approach to awareness and identification of those at risk
Lesson 4: Engage with parents
Lesson 5: Expand treatment options for those with high and complex needs
Lesson 6: Improve access to pharmacological treatment
Lessons 7: Adequately resourcing frontline providers
Lesson 8: Using evidence and data 
Lesson 9: Developing sustainable models
Lesson 10: Using a gender and equity focus

You will find the attached labelled as follows:

  • Evaluation Report - Report
  • GEROS Assessment - Part 2
  • GEROS Summary - Part 3
  • Evaluation Management Response - Part 4
  • Marketplace poster - Part 5


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2018

Country/Office:
Kazakhstan

Region:
ECAR

Type:
Evaluation

Theme:
Mental Health

Partners:
ABC Form, University of Molise

Language:
English

Sequence #:
2018/001

New enhanced search