2004 ROSA: Evaluation of the Meena Communication Initiative
Author: Chesterton, P.
The Meena Communication Initiative (MCI) is a major human rights intervention campaign in South Asia that began in 1991 with support from UNICEF Offices in Bangladesh, India, Pakistan and Nepal. Its prime intention is to bring about a transformation in the heavily-disadvantaged situation of girls.
The MCI uses a multi-media Entertainment Education approach that involves the use of entertaining stories to convey educational and behavioural development messages to its audiences. In so doing, it seeks to use the drawing power of popular entertainment to influence its audiences' awareness, knowledge, understanding, capacities and practices in relation to the status, rights and treatment of girls. The stories developed for the MCI revolve around the adventures of Meena, a nine year old South Asian girl, and members of her family and village community. The communication materials were developed by the UNICEF Regional Office in collaboration with the participating country offices.
The MCI is designed to support and reinforce programme objectives supported by UNICEF and its partners. As such, it has been linked to a number of education, health and social development programmes undertaken by UNICEF partners in government, NGOs, the media and the private sector.
In recent years, the implementation was decentralized to countries, as part of measures for further expansion, integration and longer-term sustainability. This enabled each country to decide what elements of the MCI it would implement, and how and when this would be done. The country contexts in which these decisions were taken vary widely in terms of prevailing social, economic, political, organizational and religious traditions and practices. This, in turn, has led to different levels and types of implementation at different times across the region, as each country endeavoured to tailor the initiative to meet its own circumstances. The MCI is thus a complex and variable set of intentions, processes, activities and events that have occurred and evolved over varying lengths of time in a range of different settings. The common element is the shared focus on the rights, understandings, life skills and practices of the girl child.
As the funding for MCI was drawing to a close in end-2003, and efforts were under way to ensure its continuation at country level, it was decided that an evaluation be conducted of the MCI to provide a clear identification of its outcomes, implementation processes and costs, and the potential for its expansion and sustainability.
The evaluation, commissioned by UNICEF Regional Office for South Asia (ROSA) in 2003, focuses on the key outcomes and implementation processes of the MCI in the four countries in which the initiative has been implemented most extensively and for the longest time periods - Bangladesh, India, Pakistan and Nepal. Its objectives are to:
- describe and assess the key outcomes of the MCI in terms of achieved reach, awareness and knowledge, life skills practices, and perceptions and attitudes to Meena as an entertainment education medium;
- describe and assess the implementation processes in each of the four countries, identifying the key inputs, activities, implementation strategies, efficiency of activities, enabling factors and constraints;
- assess and document the financial costs of key stages of implementation to enable an understanding of the cost of the MCI; and
- identify potential for expansion and sustainability of the MCI.
The evaluation process was decentralized to allow for wide participation at country level. Separate institutions completed the assessments in the four countries within an overall evaluation design, with adaptations suited for country-specific situations. This report, prepared by the regional evaluation consultant engaged by UNICEF-ROSA, draws from the findings of the country studies, as well as from an independent document review and interviews with a selection of key stakeholders.
In each country, a mix of quantitative and qualitative techniques was used to address the evaluation questions. Quantitative data were gathered from children and adults through household surveys, using structured interview schedules. In Nepal, findings from a survey that had already been conducted in 2002 were used instead. Questionnaires were also used for UNICEF personnel at country and regional levels. Qualitative techniques consisted of document analysis, focus group discussions, interviews, workshops and meetings with people involved in or affected by the MCI.
The findings from the country reports were analyzed to identify commonalities and points of difference, and to draw conclusions and recommendations in the light of insights gained from the documentation review and discussions.
The methodology described above was chosen to provide access to a range of experiences, views and information relating to the implementation processes and short-term outcomes of the MCI in a variety of specific contexts. Judgments about the MCI are made in the report using relevance, effectiveness, efficiency, capacity for expansion, and sustainability as key criteria.
The limitations of the evaluation lie in the extent to which observed outcomes may be attributed to the MCI; the retrospective nature of the study; the use of indirect measures of changes in skills and practices; and the extent to which country-specific findings are generalisable across the region.
Findings and Conclusions:
The evaluation's findings confirm the potential of the MCI, due to the great appeal and attractiveness of the communication materials and the channels, to communicate children's rights, particularly girls' rights, to South Asian audiences and, in so doing, to create awareness, promote acquisition of life skills, and encourage change in life skills practices.
The extent to which this potential has been realized varies, with different levels of awareness, skills and practices reported across and within the four countries. Contextual factors were found to play a major role in influencing the extent of achievement. Findings on the sources of knowledge of the MCI messages among target audiences revealed the significance of parents, villagers/society, teachers, friends and television. Key factors underpinning non-adoption of intended practices were found to include poverty, social norms, the role of adults in attitudinal change among children, local customs and beliefs, and security concerns for girls traveling to school. Together, these findings highlight the importance of further tailoring the means of communication to meet the specific needs and contexts of the target groups, and the need for multidimensional strategic planning that addresses infrastructural and cultural elements as well as those related directly to communication, and the significance of cross-programme planning.
The efficiency of implementation varied across the region, depending on factors such as differential access to resources and expertise, availability of training participants, and accessibility of target audiences. The varying experiences highlight the importance of incorporating MCI as a tool to achieve clearly-articulated results contributing to the outcomes of the respective programmes within a hierarchical results framework. Capacity building was found to have occurred within the MCI, particularly in the areas of research, dissemination and advocacy. This was not as apparent in relation to marketing, management, monitoring and evaluation.
Examples of multiplication and replication within the MCI's implementation were demonstrated, and the value of this was confirmed, subject to an assessment of the degree of fit with the newly intended audience's needs and context. The importance of disseminating details of interventions across the region so that each country and implementing agency is aware of the range of options available, was also noted.
Key factors facilitating implementation have been identified in the evaluation as the dedicated commitment of key players within and outside of UNICEF, and the inherent appeal of the Meena product, due in large part to its underpinning research and development processes. Supporting key players and establishing mechanisms to ensure continuing product appeal will be essential components of future operations.
Constraining factors were identified as gaps in UNICEF organizational awareness and application of the MCI, shortage of funds, limits on availability of materials, non-availability of materials in some local languages, and local security, transport and infrastructure problems. An extension of internal marketing, re-examination of the means available within the regional and country offices to record and disseminate the details and experiences of the initiative, and attention to human and infrastructure resource funding in ongoing strategic planning are recommended as a response to these.
While difficulties were experienced in determining the actual costs of the MCI, materials production emerged as a major component. The relative importance of this aspect might be expected to diminish in future years, in view of the embedding of initial research and development investment expenditure and the adoption of more efficient technologies.
Scope exists for further expansion of the MCI, given its positive outcomes, its unreached or partially-reached audiences, and newly emerging issues relating to girls' rights. The further development of partnerships should play an important role in this, subject to consideration of a number of issues. These include retaining the key Meena messages and intentions, maintaining artistic and production standards, and ensuring that the initiative addresses the specific needs, characteristics and contexts of its various target groups.
Key lessons learnt from the MCI implementation stress the importance of advocates, customising implementation for specific target audiences, simultaneous action on contextual enabling factors, and strategic planning to ensure that Meena activities are part of an overall set of integrated and ongoing strategies and programmes. An organizational model involving the establishment of one or more institutions, connected to but largely operating outside of UNICEF, with NGOs playing a central role, could be an option for the future of MCI.
The report includes a number of recommendations focusing on strategic planning, attention to enabling factors, internal and external marketing, cross-programme implementation, participatory decision making, ongoing evaluation, capacity development, support of key personnel, cost reporting, and adoption of a model for promoting sustainability. Key recommendations include:
- That in the context of evolving results-based management in UNICEF, that future MCI interventions be undertaken within a results framework, to enable clear identification of expected results and their contribution to the programme or project outcomes that MCI supports. This process needs to consider identified target audiences and their specific needs; articulation of intended awareness and practice outcomes, and contributing outputs for each audience; selection of appropriate means of communication in terms of audience access, capacity to use and receptivity; and assessment of possible intervention overlaps, disjunctions and complementarities.
- That implementation planning for the MCI be of a multi-dimensional strategic nature that addresses infrastructural and cultural elements as well as those related directly to communication.
- That internal and external marketing of the MCI be extended and intensified among UNICEF, government department and NGO personnel, to heighten awareness of how Meena materials and messages can be used as an integral and effective tool within and across their various programmes.
- That guidelines on implementation procedures, monitoring of results and costs similar to those that have been established for research and development of the Meena materials, be established, accompanied by specific targets, timelines, and points of responsibility and accountability within a strategic planning framework.
- That ongoing evaluation be built into MCI implementation, to enable continuing assessment of the appeal of the Meena series to its intended audiences, closer monitoring of outcomes, and refinement of processes during implementation where warranted.
- That for future implementation of MCI, consider the adoption of a model where the country offices are supported by two national foundations empowered to use and produce Meena communication materials through research guided by agreed ethical and artistic standards.
Full report in PDF
PDF files require Acrobat Reader.