2000 MLW: Going to Scale: Sustained Risk Reduction Behaviour for Youth Evaluation Report
Author: Reijer, P.; Chalimba, M.
The UNICEF programme sponsored by the Netherlands Government aims at providing in and out of school youth with the necessary tools and skills to avoid high-risk sexual behaviour in order to reduce the HIV-transmission rate. The project had a budget of US$1,200,000, of which $1,032,227 was received due to exchange rate change. It commenced in January 1997 and extended to December 2000, but was evaluated in July 2000.
Purpose / Objective
The objectives of the evaluation were:
1. Examine the programme's relevance in the context of Malawi;
2. Determine the status of the implementation of all programme components;
3. Assess the effectiveness of the prevention activities; and
4. Identify a possible way forward.
The project proposal contains unrealistic or not measurable objectives, unclear budget lines and discrepancies between the logical framework and the text. Therefore, the evaluation could not focus on the logical framework and the budget. However, the programme components are well described and the evaluators decided to base the evaluation on the narrative information in the project proposal and not so much on the objectives and specific objectives and the corresponding budget headings. The six programme components are: in-school curriculum; in-school extra-curricular; clubs for out-of-school youth; enabling environmental support; reproductive health services for youth; and research, monitoring and evaluation.
The evaluation was done by a review of documents, focus group discussions with stakeholders and field visits. Participants in focus group discussions included 5 girls and 5 boys from a primary school Anti-AIDS club; and the chief, his deputy and 9 male members of a village youth group.
Key Findings and Conclusions
In-school curriculum - Life Skills Education:
The achievements are:
- A scope and sequence chart for standards 1-8 has been produced
- Materials for standard 4 have been developed, field tested, adjusted and are being printed
- Drafts of the materials for standard 3 and 5 have been produced
- The Ministry of Education (MOE) has decided that Life Skills Education is a subject on its own in Primary Schools
The original project aim was to introduce the curriculum in standard 1-5 during the project period. This objective was impossible to achieve. The curriculum developers needed to adjust themselves to the completely new approach and the materials had to be field-tested and this is time consuming. Realistically seen, the achievements made for this component are considerable.
UNFPA is developing a curriculum for standards 5-8 on Sexuality and Reproductive Health. This curriculum will be integrated into other subjects. The curriculum will use Life Skills as an integral part of the approach. There has not been any cooperation between UNICEF and UNFPA concerning this development. The evaluators strongly recommend that UNICEF, UNFPA, MOE and the Malawi Institute for Education (MIE) will meet and combine their efforts in order to prevent duplication and avoid the risk of inconsistent or conflicting messages, and to save resources.
In-school extra curricular - Anti-AIDS Clubs:
Approximately 3,200 Anti-AIDS clubs (out of the 3,500 planned) were established. The Youth Technical Sub-Committees (YTSC) and NGOs like InterAide implement this component. Not all clubs have a trained matron or patron. Over 1,500 head teachers were orientated and matrons/patrons were trained during the project period. Some were trained before the project's start. The quality of the training could not be established during the evaluation, but some of the YTSCs, who are responsible for the actual training, have little knowledge of Life Skills Education and participatory teaching methods. The clubs have little to no materials (e.g. handbook for the matron/patron, IEC materials for the pupils or sports equipment). Most schools have one club with a membership of around 30 pupils. Many more pupils would like to become member, but clubs are advised not to exceed 30 members and the capacity to start a second club is not always available.
The members who participated in Focus Group Discussions (FGD) during the evaluation, indicated that they had adopted safer sexual practices, either by remaining inactive, abstain (second virginity) or by making use of condoms. The members of the Anti-AIDS (locally known as Edzi Toto Clubs) are supposed to act as peer educators for their age-mates. Specific research will be needed to explore if this has been effective, but in areas with active Edzi Toto Clubs the number of girls who dropout of school due to teenage pregnancy was reported to be declining.
Most clubs have more boy- than girl-members, but the participation of girls is increasing. All those interviewed during the evaluation, including policy makers, teachers, YTSC members and members of the clubs found the clubs an important and highly relevant component of the project.
Out-of-school youth - Skills Clubs:
This component includes a variety of Anti-AIDS Youth Clubs, more general Youth Clubs and Youth NGOs. The activities carried out by these clubs include HIV/AIDS information and education for the members, some outreach activities (e.g. through drama, festivals, etc.) and peer education.
Membership varies from 10 to well over 150. More boys than girls are active, although not in all clubs. Most clubs and NGOs lack materials and means of transport to perform outreach activities. Some have started Income Generating Activities (IGA) to support the members or to improve the performance of the club. The training and follow-up of club leaders and peer educators, to be organised by the YTSC or implementing NGO, has been insufficient in some cases.
The total number of clubs established is small in relation to the number of clubs for the in-school youth. Out-of-school youth in most parts of the country will not have access to one of the clubs, due to geographical distance. Clubs in urban areas might therefore be more effective than those in rural parts of the country.
The members of the clubs claim to have adopted reduced risk behaviour (abstinence, faithfulness or safe sex practices). Some clubs, but certainly not all, have developed very high, almost saint-like role models and provide messages to the public likewise, which might not be very effective and is sometimes judgemental.
Enabling environment support:
This component aims at breaking the silence and to create understanding in the population about the situation of the youth and to offer youth alternatives to spend their free time. Three aspects of the component have been carried out in full, i.e. the production and broadcast of the weekly 'Straight Talk' radio show, the radio soap 'Tinkanena' and the jingles (or commercials). Both 'Straight Talk' and to a lesser extent 'Tinkanena' are very popular in Malawi. Other activities have not been carried out, such as creating a platform for discussions between parents and teachers on youth and reproductive health, production of a video and the purchase of video equipment. Partly done are the purchase and distribution of sports equipment (for which there is a great demand) and the purchase of materials as incentives, such as caps, T-shirts, etc.
Youth Reproductive Health Services (Youth Friendly Health Services):
This component aimed at improving accessibility to formal and informal health services for youth (youth to youth sexual health services). The project was to be carried out in close co-operation with the Ministry of Health and Population (MOHP) and the National Family Welfare Council. A committee was formed, but no progress towards the actual implementation was made. The plan lies with the National Youth Council of Malawi currently for further exploration. Several peer educators and youth counsellors from youth NGOs have been trained as part of the component, however the main objective of the component has not been achieved.
Research, monitoring and evaluation:
An effective monitoring system of the activities in the districts has been installed through the YTSCs. To boost the capacity of these committees, UNICEF involved the Regional AIDS Teams (RAT). These teams bring in more expertise and capacity, which has improved the monitoring greatly. The introduction of monitoring boards in all districts, from which most of the quantitative information concerning the implementation of the project at district level can be obtained, is an innovative initiative.
The financial monitoring is confusing. The expenditure data, as provided by the programme staff, is different from the financial data provided in the annual reports. The income and expenditure situation as provided by the finance section of UNICEF Malawi is different from the information obtained from the programme staff. A clear picture of the actual financial situation could not be obtained during the evaluation.
Explore possible savings in time and resources through adapting materials used in other countries. Coordinate with other donors working in this area, e.g., UNFPA. Implement curriculum in all grade levels. Be realistic about objectives and time lines.
Strengthen and support Anti-AIDS Clubs in schools by providing adequate resources and training for matrons/patrons. Encourage more participation by girls. Discourage overly rigid rules about membership. Promote the peer educator approach so that club members can reach out to non-members.
Programmes that sponsor out-of-school youth clubs should set up clear criteria and objectives for these clubs. Clubs whose purpose is changes in sexual behaviour are not necessarily ideal for skills training; other funding mechanisms should be found for these activities.
Mass media approaches need specialised research, monitoring and evaluation to measure and improve their impact. Objectives for each programme component should be realistic, focused and measurable.
Do not include ambitious programme components that depend on partners who are incapable of supporting them. Explore other approaches such as quality improvement techniques to improve health services for youth.
Partner NGOs and CBOs need capacity building in monitoring, evaluating and reporting. UNICEF offices need to ensure that there is good coordination between programme officers and financial officers and donors, especially for tracking project expenditures.
UNICEF should take steps to improve the quality of project proposals, including realistic and specific objectives, approach, activities and budget. Donors should be responsible in assessing proposals and monitoring projects.
Full report in PDF
PDF files require Acrobat Reader.