Evaluation database

Evaluation report

2011 Bangladesh: Report on assessment of the efficacy of the different media being used by the Department of Mass Communication (DMC) in the field and monitoring of ongoing project activities



Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

 

Background:

The third and current phase of the project- Advocacy and Communication for Children and Women commenced in 2006. The project is currently being implemented by nine agencies of MoI. Among these, five agencies are regularly funded on the basis of a mutually agreed annual work plan and four agencies are proposal-based and funded on the basis of merit of the proposals. The five regularly funded agencies are: Ministry of Information (MoI) Planning Cell, Bangladesh Betar, Bangladesh Television (BTV), National Institute of Mass Communication (NIMC) and Department of Mass Communication (DMC).

The DMC is the sole implementing agency within the MoI that is involved with inter-personal communication and reaches grassroots communities through its 68 District Information Offices (DIOs) across 64 districts and four offices in the Chittagong Hills Tracts region. The DMC has focused on heightening the level of mass awareness on various issues related to wellbeing of women and children. And to do that they have chosen educational activities that include film shows including Meena episodes, folk songs and community meetings.

Purpose/Objective:

The objectives of the evaluation were:

• To monitor the activities of DIOs in each division
• To analyze the efficacy of one IPC medium against the other;
• To assess the level of awareness created on the different behaviors promoted through the film shows, folk songs and community meetings;
• To assess the effectiveness of the interventions in order of audience preference and retention capacity;
• To identify any changes in terms of individual behaviors and social norms;
• To assess the degree of urgency among audiences to change behavior based on the messages; and finally;
• To propose recommendations to strengthen and improve the project interventions.

Methodology:

Methodology:
Qualitative and quantitative data were used to monitor the implementation of the activities and to assess the impact and efficacy of films shows, folk songs and community meetings implemented by the Department of Mass Communication (DMC).

Research Methods:
Data for this study was collected through the following methods:

i. Literature review,
ii. Observation,
iii. Focus group discussions,
iv. Exit polls,
v. In-depth interviews, and
vi. Delayed interviews.

Sampling study areas:
Considering the number of areas where activities were being implemented, time constraints and the requirements, RCS proposed an initial sampling plan to UNICEF. Upon consultation with UNICEF, it was decided to cover 12 districts from 6 divisions. The total sample size included target respondents as well as the respective DIO offices from these 12 districts.

Sample Subjects and Size:
For monitoring implementation, 32 film shows, 10 folk song programs and 10 community meetings were selected randomly from selected districts of the 6 divisions and implementation of these activities was observed.

Exit polls were conducted with adult males, adult females, adolescent boys and girls. According to the field plan, 20 exit interviews were conducted for each event of film show, folk song, and community meetings. These 20 exit interviews for each event were again evenly distributed among adolescent boys, girls, adult males and females and five persons from each groupwere interviewed after each event monitored.

The rationale behind conducting FGDs with the audience of film shows, folk song sessions and attendees of community meetings was to explore the degree of understandability, internalization and overall impact of the activities of DMC among the target audiences. According to the field plan, 3 FGDs were conducted in each district. Separate FGDs were conducted for adult males, adult females, adolescent boys and adolescent girls.

In each of the selected districts, 8 in-depth interviews were conducted. The set of 8 interviews comprised interviews of 2 teachers, 2 religious leaders, 2 community leaders (UP Chairman/members) and 2 District Information Officers.

Information was also collected through delayed interviews and household observation methods.
The number of households from each district to be observed was estimated by using a statistical formula and thus the total number of delayed interviews stood at 864.For delayed interviews, households were selected at upazila level in each district. Households were selected randomly.

Research tools development:
Research tools were developed after reviewing the project documents and reports and in line with the objective of the study. RCS followed pre-testing procedures and incorporated feedback from UNICEF. All tools were applied in the field only after getting final approval from UNICEF.

A detailed monitoring and evaluation framework (Appendix D) guided the development of the questionnaires. The evaluators. used a number of data collection instruments which included :

1. In-depth interview questionnaire
2. Checklist and guidelines for activity observation
3. Checklist and guideline for HH observation
4. Questionnaire for exit poll
5. Group discussion guide, and
6. Questionnaire for delayed interview at household level

Implementation:

Recruitment:
Recruitment of field staff for the study was completed in December 2010. Recruitment was done for different levels of personnel including interviewers, FGD moderators, FGD note takers, field supervisors, and quality controllers. Selection was based on educational qualification, presentation and communication skills, courteous and previous experience and good performance in the training program.

Training:
Ten-day training on data collection techniques and styles wase conducted. Training methodologies included briefing, demonstration, role-play and pre-testing. Separate training modules were conducted for exit poll, observation, focus group discussions and in-depth interviews..

Fieldwork:
Fieldwork was completed in two stages. In the first stage, exit poll, observation of activities, focus group discussions and in-depth interviews were conducted. The first phase of fieldwork lasted for 30 days. The second phase of fieldwork included delayed interviews and household observationand started 30 days after completion of observation of the selected film shows, folk songs, and community meetings.

Data Management and Statistical Analysis:
Data entry was done using Microsoft Excel and statistical analysis was carried with SPSS. Separate data management staff was recruited for data management. Activities included training of coders, coding, data entry, data verification and quality control, data processing and generation of analytical outputs.

Findings and Conclusions:

• Film shows, folk songs and community meetings were implemented according to the original plan. All the core messages were disseminated through the IPC media and were aimed at key audiences that included adult males, adult females, adolescent boys and adolescent girls. However, participation of adult females and adolescents was lower than that of adult males.
• Film shows incur the lowest cost per audience. Furthermore, community meetings were more cost effective than folk songs; however, folk songs on average attracted more audience than community meetings.
• Over 90 percent of the audiences from all categories asserted that the IPC activities were understandable and interesting. Over 90 per cent of males reportedly felt the urgency to change behavior according to the IPC messages. Moreover, while close to 90 percent of the adult females and adolescent girls believed that it was urgent to act according to the messages, only two-thirds of the adolescent boys felt the same.
• The timing of the IPC activities were liked more by males and boys and less by females and girls. On the other hand, the venues were liked more by the adolescents than the adults. About half of the adolescent boys, more than half of the adult males and adult females, and two-thirds of the adolescent girls would have liked longer duration of the events. Close to two-thirds of all categories of audience would like to have new messages.
• Almost all the target audience as well as the community leaders (UP chairmen, school teachers and religious leaders) interviewed were aware of the issues on health, education, child protection, sanitation, hygiene, and children and women’s rights. However, the adult males reflected more awareness on the issues of hygiene, sanitation, and children and women’s rights, and the adolescent boys demonstrated more awareness on health, education, sanitation, and hygiene compared to other core messages. On the other hand, the adult female and adolescent girls were found to be evenly aware of the issues relating to all the core messages.
• Film shows and folk songs were found as the most effective IPC mediums followed by community meetings. However, they did not consider these IPC activities as the only effective mechanisms as they considered drama (IPT), school teachers, health workers, Tathaya office (DIO), and television equally effective communication mediums.
• The analysis of how many of the audiences remembered the messages disseminated by different IPC media, revealed that folk songs appeared to be most effective as about 80 per cent of the audiences could recall most of the messages related to the folk song sessions. Film shows were second, with around 70 per cent of the audiences associated the messages recalled with this activity. Finally, around 65 percent of the audiences could relate the messageswith the community meetings.
• Almost all the target audience overwhelmingly displayed a favorable attitude towards changing behavior according to the messages of the IPC mediums. They were found to be convinced that such messages should be followed not only in their individual behaviors but in the community as a whole. Therefore, the IPC mediums could be viewed as successful in influencing favorable attitudes among the target audiences and in helping them differentiate between desirable and undesirable behaviors.
• It was very difficult to assess actual change in behavior as a result of IPC activities as this study had to rely solely on the claims of the audiences interviewed and not on the observation of actual behaviors. Still, while the interviewees in general asserted that there had been changes in their individual behaviors, they also admitted that no change in behaviors targeted by the IPC media could still be observed in their communities.
• In a strict sense, change in social norms was not achieved in terms of enabling or instilling the behaviors promoted by the IPC activities in the communities. However, in a more liberal sense, change in social norms has been achieved with respect to development of appropriate attitudes towards desirable behaviors. For example, although the community did not yet feel compelled to stop child marriage or keeping children away from school, it did not consider such behaviors were right either.
• The role of DMC was found to be effective in implementing these activities according to the original plan. However, though the IPC activities were able to draw all kinds of audiences like adult males, adult females, adolescent boys and girls; the presence of females and girls was lower than the presence of males and boys. Film shows have been found to be most effective in terms of cost per audience while folk songs were found to be most effective in terms of retention of messages. Finally, although the successes in changing behaviors and social norms appeared to be limited, the IPC media were found to be successful in raising awareness level as well as favorable attitudes in the target communities.

Recommendations:

1. It appeared reasonable from the findings that the presence of females and girls could be increased by holding the IPC activities far into the villages instead of organizing them within the premises of haats and bazaars only. Therefore, this study recommends holding the IPC activities in a place convenient for the female audience, for example, in the para courtyards and preferably in late afternoon or evening.
2. One option for increasing retention could be to use reading materials like colorful posters and leaflets during the IPC activities which the audience would take away with them and could read later on or look at these frequently while at home. Another opportunity for increasing retention of messages lies with increasing frequency of the activities, for example, holding film shows or another IPC medium in a particular location once every month throughout the year.
3. Local sensitivity should be addressed through incorporating local dialects and local problems in the dissemination of the messages. Moreover, mixed strategies might also be useful. For example, in areas like Bandarban, more of film shows and folk songs should be held as the locals prefer these most. On the other hand, in areas like Sylhet, community meetings should be held more with involvement of religious leaders as such were preferred more in that locality.
4. It was found that film shows were most cost-effective in terms of costs per audience. Then, while community meetings were more cost-effective than the folk songs, it drew less audience than the latter. Therefore, instead of planning to hold film shows, folk songs and community meetings separately, formal steps should be taken to implement film shows and folk songs along with community meetings only. For example, activities could be planned to hold film show and meeting together as well as folk songs and community meetings combined.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search