Evaluation database

Evaluation report

2001 Global: An Evaluation of UNICEF's Policy Response to HIV/AIDS in the Nineties: The Use of Carrots, Sticks and Sermons



Author: Kruse, S.; Forss, K.

Executive summary

Background

In 1992 the Executive Board recommended that UNICEF intensify and expand its support to HIV/AIDS prevention activities (E/ICEF/1992/L.11) by: (a) focusing on programmes to reduce HIV transmission among young people; (b) providing information and communication to reach youth to promote informed and responsible sexual behaviour and (c) promoting improved reproductive health of women and youth. The Board's recommendations can be taken as the most significant and pregnant expression of UNICEF's policies regarding HIV/AIDS.

Purpose / Objective

Has UNICEF responded appropriately and effectively to the worsening AIDS epidemic?
This evaluation report goes a long way towards answering this important question.
This evaluation is an assessment of experiences from the last decade. It summarises major lessons learned and provides a platform for improved programming. It is primarily a policy evaluation of UNICEF's response to HIV/AIDS. It is not a programme or project evaluation -- it does not assess the implementation and results of specific activities. The evaluation focuses on three questions:

Policy intent: What policies and goals have guided the UNICEF response? Are these 'good' policies, that is, are they clear, comprehensive, focused, and flexible? Are they relevant in response to the size and severity of the epidemic? Are they co-ordinated with other policies in the organisation?

Policy implementation: How were policies introduced, made operational and implemented in UNICEF at global, regional and country levels? What policy instruments were used? (Did UNICEF have a strategic and effective mix of instruments?) ; What were the main tools to ensure organisational learning? for dissemination of new knowledge and rapid replication of best practices?

Policy results: What is the evidence of results of an intensified and accelerated response?- in respect of programme resources, new and innovative approaches to the epidemic and new partnerships, integration with national programmes, etc.

Methodology

The evaluation was conducted between May and September 2001 and covers the period from 1992 to 2001. It builds on analysis of policy intent at the global level, followed by a review of how policy instruments have been applied at different organisational levels, from headquarters in New York, via Regional Offices, and to Country Offices. The evaluation methods consist of: (1) interviews with key informants, (2) review and analysis of documentation, and (3), case studies of selected country programmes (Myanmar, Thailand, Uganda and Zimbabwe).

Key Findings and Conclusions

A. Policy Intent

UNICEF's policy has evolved. There is not one policy intent, but several. It must be seen as emerging, and over the decade it was expressed in many documents. It is relevant to speak of a policy system, where the Board's recommendation of 1992 is followed by other communications from the Board, from the Executive Director, from Regional Directors, Programme Divisions, and other sources that have a major say in what the organisation does.

Policy intent has many dimensions, foremost of which are: how much to do (volume, quantity); priority areas; modes of operation and collaborating partners. The Executive Board was very clear in recommending that UNICEF intensify and expand its activities in HIV/AIDS prevention and control. However, later policy statements have not repeated this instruction, and in practice the organisation is uncertain about what this means and how the recommendation should be applied.

UNICEF's early policy focused on youth, and emphasised 'communication' as a means of promoting responsible sexual behaviour. The policy was formulated from tried and tested approaches. It has continued to develop on the basis of experiences in a few countries that are at the forefront of developing HIV/AIDS projects. But towards the end of the decade, new project opportunities and modes of intervention were introduced: foremost of which are activities around mother to child transmission, care for aids orphans, and care for people living with AIDS.

The initial policy statements were clear in the overall approach, even though they did contain some internal contradictions. As the policy evolved during the 1990s, it incorporated new practices and became more comprehensive. The fact that new subjects were included indicates a flexible approach. But at the same time, the response became less focused.

There is actually no document that is entitled UNICEF's HIV/AIDS policy.

The main tenets of policy are well-known among UNICEF staff and it is also understood among co-operating partners - but few can refer directly to the key documents. There are also misconceptions about policy. Confusion is understandable, as policy is found under titles such as 'programme approach', guidelines, or under specific subject titles.

The approach has been ambivalent in respect of monitoring and evaluation, the role of research, and the advocacy function. HIV/AIDS policies do not appear to be co-ordinated with other policies. Policy statements in the areas of health and education have few cross-references to HIV/AIDS work. Emerging areas of organisational practice, such as rights-based programming, have not been connected to the issue of HIV/AIDS.


Implications for Policy Intent:

What are the lessons brought out by the study?
How can the expression of policy be improved?

First, UNICEF's approach to let field experience play a major role in policy formulation needs to be preserved. This is a guarantee that policies are relevant and can be implemented. It makes sure that the experience of people is used, and that the organisation learns.

Second, people inside as well outside the organisation would be well served by a document, which is actually entitled an HIV/AIDS policy. Such a document should be easily retrievable, and also explicitly updated when the policy changes.

Third, a policy document should be precise in its categories of analysis and action, and must not be contradictory. It is necessary to pay a great deal of attention to the dimensions of policy, and to subject it to a real test of how it is understood, what is included and what is excluded.

Fourth, the process of policy formulation should include a screening of other relevant UNICEF policies, and should make connections to these.

Fifth, a policy statement needs to be revised, and there should be a mechanism to visibly and explicitly and regularly update the policy statement.

In sum, the lessons learned here point to the need to make the process of policy formulation more transparent and better co-ordinated. Yet another lesson is that the rhetoric of policy must receive more attention, to make sure that policy statements are clear, precise and comprehensive.


B. Policy Implementation

In the analysis of policy implementation, the evaluation works with three categories of policy instruments:
Regulations - rules, directives, and the like, which force staff to act in accordance with what is ordered. Such measures are popularly called 'sticks'.
Resources - instruments that involve paying - either the handing out or taking away of human and financial resources. These measures can be called 'carrots'.
Information - or advocacy, motivation, exhortation, covers all attempts at influencing/convincing country programmes, governments and staff through the transfer of knowledge, the communication of reasoned arguments (negotiation) and persuasion. These measures are called 'sermons'.

When a new policy is designed, headquarters can in principle instruct (use directives), pay (use subsidies, provide resources) or persuade (use information) their country representatives, to make them comply with policy. The evaluation talks about these policy instruments in the broad categories of sticks, carrots and sermons.

It is difficult to trace the use of the stick as a policy instrument. The few orders and directives that can be documented concern administrative formalities. They do not apply to the core dimensions of HIV/AIDS policies; such as the volume of HIV/AIDS project activities, priority areas, or mode of operation. However, we cannot exclude that there are subtle ways of instruction that cannot easily be refused. UNICEF has a strong organisational culture, which is what holds the decentralised operations together. In practice, it could be difficult to distinguish between a strong organisational culture that enforces policy, and orders that serve the same purpose. Towards the end of the decade, the evaluation found an increasing use of instructions and requests for feedback, in the communication between some Regional Offices and Country Offices.

There are examples of how carrots have been used to make the Country Offices respond to the HIV/AIDS policies, but these examples are few and far in between. It was not until 1997 that the Executive Board decided to set aside specific funds to accelerate programme priorities, and it was not until two years later that the first allocations were made of these. In the beginning of the decade, the technical support provided by the TSG was important, but lasted only between 1993 and 1995. It is only recently that headquarters has responded positively to Country Office's call for funding of HIV/AIDS focal points. And recruitments to the regional offices are also recent. The most significant aspect of the carrot as a policy instrument lies in the organisational support to obtain supplementary funding. This may have been crucial to the intensified and expanded programme in some countries.

The evaluation found that the most frequently used policy instrument was sermons; neither sticks nor carrots were used nearly as intensively as sermons. Still, it is worth remembering that all policy instruments were applied at all times, even if in different degrees. Policy implementation is a complex field and touches on all aspects of governance, management and leadership in the organisation. This is not a full-fledged study of management, and hence we only convey a partial and incomplete picture of policy implementation in respect of HIV/AIDS.

The evaluation counted more than 6.000 pages of publications and close to a hundred titles. Among these were some really significant, path breaking studies that are widely respected and quoted. Furthermore, the organisation uses other publications than those produced internally, hence the sermons can be based on UNAIDS, WHO, World Bank, or other publications. Conferences, workshops and internal seminars are common means to motivate staff members to take action in respect of HIV/AIDS (as well as on other issues).

It is also quite clear that the use of policy instruments changed over time. In total, the organisation was mostly subjected to policy instruments that pushed HIV/AIDS issues in two periods; the beginning and mid 1990s, and later in the final years of the decade. In between, there were distinctly fewer efforts to apply policy instruments. Even if the policies remained the same, there were fewer efforts to bring the intents to practice. The evaluation can thus provide an explanation of why UNICEF's response to HIV/AIDS issues grew weaker in the period between 1996 and 1998. It was not because policies changed, but because the use of instruments to make the intentions happen were not applied to the same extent as before and after.

Implications for the Use of Policy Instruments:

What are then the lessons learned about the use of policy instruments?
We have not been able to document any particular processes that explicitly deal with the concept of policy instruments. Once the policies were formulated, different parts of the organisation coped with the policy intent as best as they could. Nobody framed the issue as a question of how to use a specific set of instruments to realise policy intents. We believe that a more explicit recognition of the selection and application of policy instruments could have reinforced policy intent. In order to increase the impact of policies, the following broad categories of study and organisational reform can be suggested:

First, much as it is easy to call for a coherent approach to policy instruments, practice is more difficult. It is generally believed that sticks and carrots are more effective policy instruments. But sticks and carrots are more centralised management tools, and the cost of centralisation may be high. Our study is one of the first policy evaluations in UNICEF. There is a need to learn more about the comparative advantages of policy instruments.

Second, it would be desirable to develop a coherent management approach to the choice of policy instruments, to make sure that sticks, carrots and sermons are used in a balanced way, to supplement each other, and reinforce the organisation's response to the policy intent. One suggestion would be to supplement policy statements with a note on implementation. Such a note could suggest which instruments to be used in support of the policy. It may even be possible to suggest budget allocations for specific policy instruments.

Third, sermons were the favoured policy instruments, and maybe there are reasons to believe they are the most appropriate instruments for UNICEF. Is it possible to improve the efficiency of sermons as policy instruments, given that they seem to be preferred, and better in harmony with decentralised modes of operation?

Fourth, there is a connection between policy issues and the level of change required. When there is a need to address large global issues and make fundamental changes in what the organisation does, it is necessary to use the more effective policy instruments, that is; directives and financial allocations (sticks and carrots).

In sum, it may seem as if the lesson learned is that there is a lot more to learn. The study shows that the organisational response to policy has been ad hoc. There has not been a systematic, comprehensive approach to realise policy. We have not been able to detect and document any application of managerial tools to explicitly realise policy intents. This is not to suggest that there are no such tools, but the daily, practical line management of the organisation does not always have a close and vibrant connection to the processes of policy formulation.


C. Policy Results

Leadership Commitment
Since 1992 UNICEF has aimed at expanding its approach to the epidemic. The leadership commitment behind the process was uneven during the nineties particularly in the latter half of the decade. While HIV/AIDS has become a priority for the organisation, it has not yet been matched with human and financial resources.

Globally, the TSG process contributed to raise awareness, commitment and knowledge about HIV/AIDS between 1993 and 1996 in a group of countries. With the turn over among key technical staff, internal reorganisation of the Health Section and change in top management of UNICEF, the focus on HIV/AIDS especially on youth lost some of its momentum around 1996/1997.

With the highly visible and dramatic impact of the epidemic in UNICEF programme countries and the increasing global political awareness and commitment, HIV/AIDS has become a UNICEF programme priority. The Executive Director speaks at major international meetings and events, advocating for increased attention and activities to combat HIV/AIDS. Regional Offices are taking a more active role and there are Country Programmes that have a stronger HIV/AIDS profile.

Human and Organisational Resources
Few staff worked full time on HIV/AIDS during the nineties and it is recently (1999) that a separate HIV/AIDS Unit was established at global level. UNICEF did not invest much in building technical and organisational capacity in the area of HIV/AIDS.

Financial Expenditure
The critical constraint is the lack of systematic data on expenditures. Strictly speaking, UNICEF is not in a position to assess to what extent the organisation has intensified and expanded its response to HIV/AIDS in terms of funding. Policies and priorities are not properly linked to the budgetary process. Without a baseline and data on intended and actual expenditure, UNICEF has a weak basis for setting and changing priorities. Policies tend to become sermons.

Estimated figures, however, indicate an increase in expenditure on HIV/AIDS, but not a rapid and dramatic increase (30% between 1996 and 2000). Expenditure on HIV/AIDS as compared to total UNICEF spending is relatively low. In 1999, UNICEF expenditure on education was $ 120 million or 12.4% of global expenditure and 14.7% of programme expenditure. Comparison with expenditure on health would be even larger. The estimated average annual expenditure on HIV/AIDS between 1996 and 2000 was approximately US$ 32 million.

Though UNICEF increased its global allocations to HIV/AIDS programmes during the nineties, the increase remained relatively modest. HIV/AIDS did not become a programme priority for the organisation in terms of funding from regular resources. Our case studies show an increase in expenditure in the four countries, but a decrease in Thailand and Zimbabwe at the end of the decade.

Geographical Expansion and Coverage
The expansion of the response has been geographically uneven and concentrated in a few countries and regions - mainly in Eastern and Southern Africa and a few hard hit countries in Asia and Latin America. Tanzania, Uganda and Zimbabwe were the largest single recipients of funding.

Major Policy Achievements
UNICEF introduced and supported at an early stage a broad societal, multi-sectoral and behavioural approach to the prevention of HIV/AIDS. HIV/AIDS was taken out of the medical field and defined as a health and development issue.

UNICEF prepared the ground for a stronger focus on HIV/AIDS in the context of adolescent health and youth. Since the epidemic hit young people stronger than small children, a new target group (with their own concerns and problems) were introduced to the organisation.

UNICEF's broad societal and inter-sectoral approach to HIV/AIDS prepared and supported the ideological platform which led to the formation of UNAIDS in 1997.

Programme Achievements
There are examples of innovative programmes that were replicated and taken to scale such as:
life-skills education in schools, Information, Education and Communication (IEC) initiatives and to a lesser extent youth friendly health services.

School Based Interventions - Life Skills Programmes absorbed the largest share of UNICEF's resources in the nineties. There are few in-depth evaluations of the performance of such programmes. They have made the environment for HIV/AIDS awareness and prevention programmes much more open, increasing the potential for positive behaviour development and change, but the actual change in attitudes and behaviour of young people is uncertain and unknown.

Out-of-school peer-to-peer education used young people to disseminate the message. Youth are educated in an effective context, one that includes teachers from their age range and culture. Peer education has a great potential and is presented as effective in Uganda and Zimbabwe, but peer education programmes have been even less rigorously evaluated than school-based programmes, so there is no solid basis for comparing results.

Information, Communication and Education programmes (IEC) was the second most important area of interventions. Several pilot projects were started and tested out in a broad range of countries - mainly in collaboration with NGOs and churches.

Youth Friendly Health Services. was the third programme area. A key concern was to demonstrate the operational feasibility of going to scale with interventions that promoted and maintained sexual and reproductive health. We found examples of youth friendly health services approach in Uganda and Zimbabwe, but much less significant in volume and impact than life skills education and IEC programmes.

Each of the strategies are useful and relevant in their own right, but could have been stronger as part of a multi-strategy approach. A school-based programme is more likely to be effective if reinforced by other supporting strategies, such as policy, media, access to good services and supplies, etc. Single strategies are often too weak to make inroads into the change of social norms, attitudes and behaviour. In this area of multi-level strategies, there is scope for improvement in UNICEF.

Replication and Scaling up of Programmes
A key concern for UNICEF was to identify new programme ideas, test them out as pilot projects, summarise lessons learnt from country implementation, and scale up small projects to national coverage, and replicate successful projects in other countries.

There are several country examples where programmes were replicated and taken to scale. Myanmar, Zimbabwe and Uganda are countries where life-skills programmes were planned and implemented with national coverage. In Thailand, an early PMTCT initiative has been integrated in the national health system. Achievements in terms of coverage have also been impressive. Questions are more related to lack of information about outcomes and impact in classrooms and among children and youth.

There is a tension between innovation and scaling up. Innovations require a culture of research and experimentation, while scaling up calls for the capacity to implement. Within a scaling up strategy, UNICEF needs other types of expertise, another time perspective and more level of funding than when pursuing an innovative strategy. It is not obvious that the same organisation can do both - and do both well. In our view, UNICEF may have placed too much emphasis on scaling up a few key programmes - and less on searching for new ideas and approaches. It lost some of its innovative potential. UNICEF is not a research organisation, but not merely an implementer either. The concept of scaling up has often been too one-dimensional. Rather than scaling up one programme in one country, it would have been possible to scale up through different models, e.g. life skills education in schools in some districts, peer led programmes in others, etc. The point would be to test out different alternatives and aim at national coverage through a range of models.

Strengthening New Partnerships
UNICEF broadened its range of partners during the nineties. In the early nineties it opened up a much more active collaboration with external partners and to included new and non-traditional partners.

WHO was the main partner at a global level and interaction centred on UNICEF and WHO during the TSG process. Other UN partners were much less involved. In the latter half of the nineties, WHO's role was taken over by UNAIDS.

UNICEF HQ expanded its partnerships with technical institutions and NGOs during the TSG process to implement an integrated and multi-disciplinary approach to HIV/AIDS. UNICEF HQ was not able to maintain those partnerships.

Partnerships between UNICEF and NGOs have been better maintained and have also expanded rapidly within country programmes. Private sector and technical institutions have played minor roles.

UNICEF's programmes and mode of operation were not significantly changed as a result of UNAIDS at the country level. In the early period of UNAIDS, UNICEF could hardly be called an enthusiastic cosponsor. Attitudes and practices have changed slowly and UNICEF has emerged as one of the most active cosponsors (in the four case countries).

Improvements in Co-ordination and Communication
UNICEF HQ provided weak global support and co-ordination to HIV/AIDS in the latter part of the nineties. A new central HIV/AIDS Unit has been recently established.

Regional Offices from the beginning played a marginal role in supporting HIV/AIDS programmes, but the mandate of ROs has changed. The Regional Management Team in the Eastern and Southern Africa Region meets, for instance, two to three times a year to determine how best global policies and priorities can be introduced and applied within country programmes, and to monitor progress in the implementation of the regional HIV/AIDS work plan.

UNICEF during the nineties was actively involved and supportive in efforts to consolidate various global co-ordinating mechanisms, including:

(a) The UN System Strategic Plan for HIV/AIDS 2001-2005. The first plan presents, to a large extent, a summary of what the individual parts are doing.

(b) UNAIDS Unified Budget and Work plan (UBW 2002-2003) has mainly included global and regional HIV/AIDS activities - not UNICEF's and other cosponsors regular country budgets. This means in practice that a major part of multilateral funding for HIV/AIDS at country level is not included and reflected in the UBW.

(c) The UN System Integrated Work plans (IWPs) on HIV/AIDS, prepared under the auspices of the UN Theme Groups, are joint prioritisation and planning exercises at the country level. Developed within the overall system of UN Development Assistance Frameworks (UNDAF), the IWPs are intended to include all of the HIV/AIDS-related actions of the UN system in support of national HIV/AIDS plan. In our case countries, UNICEF have played an active role in the preparation of Common Country Assessments (CCA) and UNDAFs advocating children's and women's rights, but it was difficult to track the importance of such plans and processes.

(d) HIV/AIDS Theme Groups have become fora for communication and co-ordination among UN agencies. In an increasing number of countries, there are Expanded Theme Groups with participation of bilateral agencies and NGOs. UNICEF has gradually increased its involvement in national theme groups and more inter- agency collaboration. In January 2001, UNICEF chaired 19 Theme Groups compared to 16 in 1998. An opinion seems to prevail that the Theme Groups are wanting in strategic direction and analytical depth.

(e) Country Programme Advisers (CPAs) are facilitators of more active collaboration and joint initiatives within the framework of national strategic plans. CPAs have played and continue to play important co-ordinating roles, but the role of UNAIDS and CPAs are not clear to cosponsors at the country level - including UNICEF.

Capacity Building and Integration with National Plans
UNICEF has been criticised for seeking visibility and being reticent about collaboration. UNICEF has had a tendency to take on a "prime moving role" - not sufficiently ensuring ownership and leadership of national governments.

The country case studies present a more multifaceted picture in which UNICEF works more consistently with capacity building, policy development, processes of consultations and integration of programmes in local structures.

Evaluations and Evidence of Results
UNICEF is increasingly pressured to measure and document outcomes and impact in order to demonstrate for donors that the organisation provides an effective and efficient response to HIV/AIDS. But existing M&E systems have their limitations:

UNICEF initiates an increasing number of evaluations, but there is no systematic approach to evaluation in the organisation where evaluations of a cross-section of projects are carried out at regular intervals.
Most evaluations are undertaken of individual projects and programmes. Data and information are not available for broader thematic or geographic areas.
Assessments of impact mainly use qualitative methods. Results are consequently based on impressions and less on quantitative data and analysis. There is not much information on how much and how well.

Changing Country Trends on Outcomes and Prevalence
Official figures on numbers of HIV positive and reported AIDS cases in Myanmar indicate that the rate of transmission is growing. The same is true for Zimbabwe. The most recent data from Thailand suggests that the numbers of new HIV infections peaked in the early 1990's and since then has declined by more than 80%. There has also been a positive reduction in Uganda.

The varied experience raises some key questions around evaluation in UNICEF. The Thai national programme for HIV/AIDS prevention has been considered very successful. The same is true in Uganda. The transmission of HIV has been significantly reduced, and there are clear changes in sexual behaviour.

It would be misleading to assess UNICEF's performance by using prevalence as an indicator. There is no direct causal link between increase or decrease in prevalence rates and UNICEF performance. The role of the international organisations in general and UNICEF in particular, is hard to pin down. UNICEF has no doubt contributed to many of the activities that took place. It has had an important role as an advocate for change, and for pioneering efforts in communication with youth. But we cannot know for sure what the relative importance has been. In particular, the weakness of the monitoring and evaluation system, the lack of critical examination of project and programme results, make any such estimate mere guesswork.

Recommendations

Implications for Policy Results

The review shows that the results are considerable, but what are the lessons in terms of policy formulation and implementation? Can the results be attributed to policy and the choice of policy instruments, and if so, to what degree? There can hardly be any doubt that the results to some extent depend on other factors than policies.

In some countries, results were achieved even before policies were formulated, let alone implemented. In other countries, it can be assumed that there are no results at all, even if policies have been around for some time. Yet in other countries, there is a close causal connection between the formulation of policies, application of policy instruments, organisational response, and results. Myanmar appears to be a prime example.

Is the lesson that policies are neither necessary nor sufficient to produce results in respect of practical action to combat HIV/AIDS? No, the situation when policies were not necessary belongs to history (Thailand and Uganda 10 to 15 years ago). In later stages, policies have encouraged people to take action, has legitimated initiatives to develop HIV/AIDS projects, while at the same time allowing different responses.

However, policies alone are not sufficient. In particular, the process of policy formulation must be followed up with the application of policy instruments. Policies do not make a difference in their own right, particularly not in a policy rich environment. When many issues claim the attention of decision makers it is even more important to provide directives on the most significant areas of work. In the end, if policies are to provide results, there are financial implications. Policy formulation can be more or less closely linked to budget allocations. The closer the link is, the larger will the impact of policy formulation be.

To make policy happen with the help of sermons can be quite costly (there are both direct costs involved in preparing and delivering the message, as well as opportunity costs). Could the same results be produced by using a more cost-effective policy instrument? More cost-effective policy instruments are centralised, standardised and less flexible. However, it must be remembered that the costs involved are not only monetary, they also come in terms of organisational values and preferences.

 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2001

Region:
Global

Country:
Inter-regional

Type:
Evaluation

Theme:
HIV/AIDS - Other

Partners:

PIDB:

Follow Up:

Language:
English

Sequence Number:

New enhanced search