AZE 2002/010: Evaluation of Information and Program Communication Components of the Program of Cooperation, Azerbaijan
Author: Aliyeva, A.; van Mourik, J.
The information currently available in the UNICEF Office is rich and diverse: many evaluations and reviews, assessments and surveys, several policies, proposals, programs, reports, leaflets, brochures, magazines, newspapers, etc., some of the highest quality and relevance. This is different and contradicting with the absence of information outside of the UNICEF Office. Information supply and availability are rather critical and not sustainable at the moment, even though UNICEF has managed to break the silence, and managed to facilitate information and communication ways and means.
Purpose / Objective
This Evaluation Report is an effort to better capture the information and communication realities, from an independent external perspective, to provide useful ingredients to make information and communication even better in the future, and to set some clear directions for UNICEF's first Communication Strategy in Azerbaijan, supporting the implementation of the Program of Cooperation for the years 2003 and 2004.
- Assess and analyze the approaches used in the information and program communication components of the program of cooperation in relation to the country program goals and objectives
- Identify innovations and lessons gained in the implementation of information and program communication activities
- Determine appropriateness of current strategies used in information and program communication in relation to changes in country environment and situation of children and women
- Draw out strategy options and future directions of the information and program communication components of the program of co-operation
- Assess the impact of the information and program communication interventions in relation to the situation of children and women
Consultants have used focused questions to evaluate the information and communication components of the Program of Cooperation of the years 2000 and 2001, related to five key evaluation domains: relevance, methodology, impact, budget and cost effectiveness, and management capacity and specialist expertise.
The evaluation findings are based on 6 evaluation activities carried out during the month of August 2002, and are integrally part of this evaluation exercise:
- Content analysis of print media outputs; analysis of media outputs about children's issues
- Focus group discussions with children, parents, and journalists; in-depth discussions on selected issues, eliciting opinions on key issues raised in information and communication, identifying lessons learnt and areas that need improvement
- Key information interviews with representatives at different levels, who are involved in the program of cooperation; educators, health personnel, local authorities, beneficiaries, and policy makers
- Case studies on specific issues in communicating child rights; UNICEF's HOPE Newsletter has been given priority here
- Desk review of an enormous amount of policy papers, reports, surveys, and assessments, to find out how the work to-date has affected UNICEF and its stakeholders in business
- An inventory of communication materials produced, distributed, and currently stocked in the UNICEF Warehouse; assess relevance and impact
Key Findings and Conclusions
Information and communication work has been very relevant to the overall Program of Cooperation. Information and communication is never a goal or objective as such, but is clearly one of the ways 'to get there.' Even though the Program of Cooperation did not go beyond the usual listing of some information and communication activities during the implementation of the program, information and communication have been instrumental in getting across some of UNICEF's priorities. Information and communication have boosted the implementation of all the different programmatic agreements between the Government of Azerbaijan and UNICEF.
Information and communication work have been somewhat fragmented and scattered, which is usually the result of the absence of (and could easily be improved by) an information and communication strategy. The information and communication work has been carried out on an ad hoc basis, and rather needs oriented, though with a good deal of creativity.
Information materials and communication activities have been more or less planned for and these scheduled initiatives have indeed been carried out, but the real relevance remains to be seen and has never been accurately measured. Quality indicators for evaluation have not been present. Baseline data has not been available for this purpose. Behavior of clients, stakeholders, or beneficiaries has not been polled or flagged, nor indicated prior to the production of materials or prior to the organization of events. Therefore, there is a huge window of opportunity for improvement, and for making information and communication truly relevant.
Successes of UNICEF's interventions have been somewhat less effective because of inefficient coordination in the domain of information and communication among international humanitarian organizations and between the international humanitarian organizations and the Government. Everybody seems to be active with information and communication work, but nobody really knows what has been done with information and communication.
Even though some Government or civil society representatives have been involved in the development stages of information and communication activities, this should not be labeled as participation. There have been indications that the Government did not like to be in the driver's seat, and preferred to depend on external assistance. Some beneficiaries and stakeholders supposedly involved and 'participating' in the information and communication work of UNICEF did not know that the work initiated by UNICEF was indeed done by UNICEF, or done by the Government assisted by UNICEF. They oftentimes mixed UNESCO with UNICEF, or UNICEF with UNDP, or UNDP with the United Nations as such, or even USAID with UNICEF, and so on. Consultants like to raise these particular evaluative observations so as to flag the issue of ownership through UNICEF's concrete, visible and relevant information and communication work.
It is unclear how much the Ministry of Health has been involved, and to what extent it has shown active participation in undertaking both aforementioned studies. It is also not very clear to what extent the results of the studies have been fed back into the development of new or revised information and communication policies, or to what extent the results have been shared with partners in business for coordinated follow up.
Over the years though, there have been some very positive signs of increased interest in information and communication, in terms of thinking about communication objectives and how to better integrate information and communication activities into PPAs for example, but they have never been time-bound, measurable, specific, or manageable.
Methodology for information and communication work has been so much 'methodology' as such. Objectives of the information and communication work were often very clear. As far as methodology is concerned, on the other hand, it has been more of 'common sense within the country context'. The individual Program Officer usually decides about the information and communication work required for his or her activity to be implemented. Sometimes, it was a poster, leaflet, t-shirt, sticker or brochure, and sometimes it was a meeting, a workshop, a seminar or other event, produced and organized, without many objective indications whether other outputs could have been useful.
It is, oftentimes, much of the old-fashioned way of top-down information or propaganda being orchestrated in Azerbaijan. Consultants find it impressive to notice how strongly UNICEF has been encouraging partners to be more pro-actively involved, given the difficult country context. Target groups are being drawn into some of the phases of information and communication project conceptualization and implementation and follow up. Children, youth, parents, teachers, doctors, nurses, consumers, et cetera have not yet been fully involved in the information and communication project design. Consultants' impressions are that this has changed into a positive direction during the evaluated period of time.
Some very positive examples of targeting are related to breastfeeding (targeting mothers and healthcare providers), mine risk education (targeting adolescents), and immunization (targeting parents), even though the information and communication materials are not known to have significantly contributed to the implementation of policy objectives. Consultants believe that, to some extent, these information and communication materials have indeed contributed to the successful implementation of the objectives referred to. Due to the lack of concrete baseline data, effective evaluation of information and communication work has not really been possible here.
The whole point here is that Consultants have the impression that UNICEF Azerbaijan has tried very hard to bring about useful information and communication work, but that it has not looked into alternative ways of getting the best. It is believed that UNICEF did not avail of a well-planned and integrated methodology to produce and organize information and communication posters, leaflets, events, et cetera.
Encourage a shared understanding of UNICEF information and communication terminology, approaches and ways of information and communication work, ensure that UNICEF annual plans for 2003 and 2004 get integrated strategic directions and activities for information and communication work, preferably through the participatory communication planning approach, involving both stakeholders in business and beneficiaries, as well as UNICEF staff. Explain the steps of the information and communication planning process, and apply the planning process utilizing relevant case studies (e.g. education for girls at the secondary education levels, HIV/AIDS care and prevention among adolescent boys and girls).
Consider alternative ways of putting the Government in the driver's seat in the domain of information and communication work, in order to contribute to the increase of people's trust and confidence in their Leaders. Assist the Government by raising awareness of the relevance of some of the Government's information and communication work related to issues UNICEF likes to deal with, through offering on-the-job training of relevant Government Staff.
Use the opportunity of the MTR to further integrate information and communication in the Program of Cooperation using the CRC, GMC and other successful initiatives as unifying elements, which add value to sector-wide efforts. Consider using the branding toolkit to renew UNICEF's image and enhance UNICEF's visibility among target groups, both among partners in business and beneficiaries.
Full report in PDF
PDF files require Acrobat Reader.