We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 India: Evaluation of Meena Radio Programme



Author: Deloitte Touché Tohmatsu India Private Ltd. (Deloitte)

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report.

Background:

The Meena Radio Programme is an entertainment-education programme that was developed by UNICEF in partnership with the Department of Education, Government of Uttar Pradesh (GoUP) and its Sarva Shiksha Abhiyan (SSA) State Project Office. The programme’s aims at the time of inception included:

  1. To cause a measurable change in knowledge, attitude and intent on key child rights issues, and issues related to child health, sanitation, and hand hygiene amongst students.
  2. To address existing social norms that prevent 100% enrolment, retention and completion of upper primary education for girl children.
  3. To improve skills amongst children reached through the programme, in problem solving, negotiation, and prosocial behaviour.
  4. To model and endorse the concept of child friendly schools, and life skill-based education.

The Meena Radio Programme was designed as a combination of mass media and interpersonal communication. The 15-minute radio programme was developed comprising of three segments: songs, stories and games. It was decided to include the radio programme as a part of the structured school time-table for middle school students. This ensured that all the students could listen to the programme and engage in a post broadcast teacher-facilitated discussion on the key messages of the day’s episode.

The Meena Radio Programme was piloted in 9 districts in Uttar Pradesh (UP) in 2010 and expanded to the rest of the state in 2012. The programme was introduced in Andhra Pradesh (AP) and Madhya Pradesh (MP) in 2012.

UNICEF engaged Deloitte Touché Tohmatsu India Private Ltd. (Deloitte) to carry out an evaluation of the Meena Radio Programme in UP, AP, and MP.

Purpose/Objective:

An evaluation was carried out to objectively and systematically assess the contribution that the Meena Radio Programme has had in enhancing the knowledge and awareness of students and to understand implementation gaps. The evaluation findings will be used to strengthen implementation in the states where the programme is being run, as well as provide evidence for potential replicability or scale up in other states. Lessons learnt and recommendations from the implementation will be examined to ensure improved efficiency and sustainability.

The evaluation sought to examine the Meena Radio Programme across the following criteria:

  • Relevance to ascertain how relevant and pertinent the Meena Radio objectives and activities are to the national flagship programmes of education.
  • Effectiveness to ascertain the extent to which the intervention has addressed knowledge gaps, attitudinal shifts, life skills, equity and influenced behaviour change among children and teachers.
  • Efficiency to ascertain cost effectiveness of the programme and the adequate and optimal utilization of resources - financial, infrastructure, human resources and time.
  • Sustainability to identify the major factors that influenced or hindered the continuity of the programme through SSA and the factors that are crucial to ensure sustainability.

The evaluation also specifically sought to evaluate the intervention’s focus on gender and equity.

Methodology:

The evaluation used a mixed methodology in order to cover each aspect of the intervention. This comprised of primary qualitative data collection and analysis combined with a secondary analysis of the previously conducted studies on the Meena Radio Programme.

The primary qualitative research involved in-depth interviews with stakeholders from across the government system at the state, district and block levels as well as with school teachers and Focus Group Discussions (FGDs) with students and parents. The evaluation used participatory methods through the FGDs to obtain qualitative inputs for the evaluation. Stakeholders at each level were selected on the basis of the sampling design which is described in the Methodology Section.

Non-participant observation by members of the Deloitte Team during the broadcast of the Meena Radio Programme in schools was also carried out. During the field visits, the team also probed for instances where students or teachers had initiated social change in their community on a thematic area that they had learnt about from the Meena Radio Programme. Whenever any such story of change was identified, the Evaluation Team conducted an in-depth discussion with the identified student/ teacher to document it.

Findings and Conclusions:

Brief findings are shared here. For more in-depth findings please refer to the complete report.

The objectives of the programme are well aligned with the educational priorities of the SSA/RTE Act. One of the key thematic areas on which the programme focuses on is education and the importance of enrolling in and attending school. This is one of the main aims of the RTE Act.

The programme has a high recall value among students of class VI-VIII and has been effective in eliciting participation and keeping the students engaged. Primary research revealed that students sing along with the title song, try and participate in the games and are able to recount stories from the programme. Students also reported discussing the programme with their friends and siblings. The programme has been effective in improving knowledge and awareness on topics such as hand hygiene, importance of using toilets, nutrition, child labour, child marriage, and importance of education. Gender roles and stereotypes, however, remain deeply embedded.

Broadcast costs of the programme are borne by the government in each of the three states. Government stakeholders met believed that the broadcast of the programme would continue, but they would require UNICEF support to produce more episodes. Officials also felt that UNICEF support for training and monitoring is required. Its sustainability in schools is dependent to a large extent on perceptions of the importance of the programme to the government. Periodic monitoring by the government in UP and AP ensures that the programme is played diligently in the schools.

UNICEF support has been one of the biggest factors for the success of the programme. UNICEF support would be important in the initial stages of its roll out and implementation in a state. As the programme implementation reaches a stage of maturity, the government can start taking ownership of increasing responsibilities and UNICEF’s involvement in the programme can be reduced.

Recommendations:

Government to continue funding the implementation of the Meena Radio Programme (including the broadcast, training of teachers, availability of radios, IEC materials and ensure the institutionalization of the programme in the school timetable) through the SSA budget and other relevant budgets available. Given its effectiveness and efficiency, government can consider introducing it in other states where broadcast is not happening.

Government to inform private aided schools about the Meena Radio Programme and its implementation modality.

Government to strengthen implementation of Meena Radio Programme in various ways as suggested.

UNICEF to develop new content and new episodes on gender and other relevant topics every two years for the programme to be relevant

There should be clear terms of engagement should be established between UNICEF and the state government from the time of inception of programme.

Annual learning visits could be organized by UNICEF to states where the programme has been successful. Regular fora for discussion of these experiences and systematic documentation will facilitate better knowledge management and consolidation.

Lessons Learned:

Numerous lessons learned emerged from the evaluation. Some are listed here:

The role of UNICEF both as an advocacy and technical support partner has been crucial in ensuring the sustainability of the programme. In UP and AP where there is a dedicated UNICEF consultant to support the programme, it was seen that the programme is implemented well. Additional monitoring support from UNICEF also helps to maintain a sustained interest in the programme.

IEC materials provided by UNICEF also help to keep the schools informed about the Meena schedule. The calendars provide details about when the programme will air. Meena books and Meena CDs that have been provided in UP as a part of the Meena Manch also help to generate and sustain interest in the programme.

Launching the programme with media support and publicity also helps generate interest in the programme. This was done in AP and built interest and curiosity in the programme.

A trained and dedicated teacher in charge of conducting the Meena radio programme ensures that the programme attains its objectives. It ensures that the discussions facilitated after the programme help the students to understand the message conveyed in the programme. In all the three states training workshops for teachers were organized when the programme was launched. However, very few teachers in MP and UP in the schools had received this training. In AP, the state has used video conferencing to train teachers and this has resulted in a larger number of trained teachers who are adept at conducting the programme.

The quality of the radio signal is an important factor that determines whether students and school staff remain interested in listening to and airing the programme respectively. In areas where the quality of the radio signal is poor such as Guna and Sheopur in MP and Aligarh in UP, the students are unable to hear and understand the programme.

 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2015

Country:
India

Region:
ROSA

Theme:
Program Communication; Education

Type:
Evaluation

Language:
English

New enhanced search