Evaluation database

Evaluation report

2005 BTW: A Report of the Evaluation of "RINGING THE BELL: A RE TSOGENG” Project

Executive summary

Background:

In 2002, Botswana Network of People Living with HIV/AIDS (BONEPWA) entered into a partnership with UNICEF and the Ministry of Education to engage people living with HIV/AIDS (PLWHAs) as role models and ambassadors of hope in Botswana’s fight against the escalating HIV/AIDS epidemic. This initiative known as Ringing the bell (A re tsogeng) project empowered the PLWHAs to bring “behaviour change messages to schools and in the process making the schools become youth-friendly information centres on adolescent reproductive health, sexuality, and human rights.” It was anticipated that the PLWHAs as the credible voice would help fight stigma and discrimination.

Purpose/Objective:

The purpose of the evaluation was to identify lessons that can be used to inform the scaling of the BONEPWA/MOE/UNICEF project to national level. The evaluation is therefore intended to identify strengths and challenges of the project for continuous improvement and quality enhancement during the up-scaling of the project with particular focus on replicability; identifying its relevance, challenges and opportunities in reducing the stigma associated with HIV/AIDS by engaging PLWHAs as effective role models for how to live healthy, productive lives with HIV and reinforce information about the need to adopt and maintain safe behaviours.

The evaluation was expected to assess the following aspects of the project:

• Effectiveness of BONEPWA, MOE and UNICEF activities in general and UNICEF support in particular
• Efficiency of the BONEPWA/MOE/UNICEF project activities
• Relevance of the project as a strategy to establish HIV/AIDS information out-reach resource centres in the schools and engage PLWHAs to promote greater involvement of PLWHAs
• Sustainability of project and programme outcomes and their replicability at national level
• Specificity of objectives and activities and their relevance and appropriateness in relationship to the Human Rights Approach and the MOE HIV/AIDS programme

Methodology:

The evaluation used both qualitative and quantitative methods. The quantitative methods were to provide data to quantify the outputs and the outcomes of the project. The qualitative methods on the other hand were to get in depth reflections on the issues in the project and facilitate the assessment of impact factors that are not easily captured in quantitative data. Fourteen primary schools were visited.
The following specific methods were used:

• Desk review of relevant literature
• Interviews and focus group discussions with pupils
• Pupils’ stories and statements
• Pupils’ drawings
• Interviews with head teachers, guidance and counseling teachers, field officers, Principal Education Officers and other key informants
• Focus group discussions with pupils, parents, field officers and Principal Education Officers

A stakeholders’ forum will be organized to receive the evaluation report. Field officers, teachers, parents, community leaders, DAMSCs, PLWHAs, BONEPWA, Ministry of Education officials, UNICEF and other partners will participate in this forum and will discuss the future of the project. The forum will review the recommendations of the evaluation and the experiences of the partners to chart the way forward.

Findings and Conclusions:

Strengths of the Project:

• The project is a partnership in which each partner has “ brought something to the table.” BONEPWA brought the PLWHAs who are the credible voices, UNICEF brought the funding, MOE provided the schools and teachers and the community brought children and parents.
• The project is in keeping with the goals of Vision 2016 to ensure that there will be no more infections by 2016. This Vision recognizes the potential contribution and support of PLWHAs in its realization of Botswana’s goals.
• Involvement of PLWHAs in the response to HIV/AIDS pandemic is recognized and supported internationally.
• PLWHAs as the credible voice, is a unique example that Botswana and its partners have operationalised their commitment.
• Participation of PLWHAs illustrates that AIDS does not seclude an infected person from productive work and this also promotes their sense of self-dignity. 
• The project is appreciated and accepted by pupils, parents, teachers and people working at various levels in government and civil society as all recommend that the project should be revived and replicated throughout the country.

Issues and Challenges:

Partnership and Collaboration
The response to HIV/AIDS requires a holistic and multi-sectoral approach that demands synergies and linkages at implementation level. This project was a partnership between UNICEF, Ministry of Education and BONEPWA with Ministry of Education joining the partnership later. The need to broaden the partnerships was recognized. There were, however, no linkages with other parties like DAMSC, which operates at community level and has formed HIV/AIDS committees at the District and village levels. There were overtures in area like Kasane for PLWHAs’ support groups to join those committees. While there were those support groups both at the district and village levels, these networks do not seem to have been coordinated and school based activities linked to them. The nature of the collaboration should be defined and the coordination structures be strengthened.

Project Management and Coordination
The Ministry of Education school staff undertook the day-to-day supervision. BONEPWA Programme Officer was not able to visit schools on a regular basis because of financial and transport constraints. They did not have a direct link with the supervisors at the school level. BONEPWA was responsible for the day-to-day implementation of the project. There is need to establish closer links between the Regional and Principal Education Officers, the supervising teachers and the BONEPWA Programme Officer. Adequate transport should be provided to enable the officer to visit schools. In addition, meetings should be organized so that the participating PLWHAs can meet on a regular basis to exchange experiences as well as give and receive feedback.

Project Monitoring
Monitoring is a critical management tool. NGOs often cite lack of monitoring of their programmes as a key challenge. Evaluators noted that BONEPWA needs to sharpen the monitoring aspects in the programme. Though the general objectives were specified and the field officers had a Facilitators’ Manual, there were many important guidelines that were missing. It is therefore important that when the project is redesigned, a clear framework should be developed containing specific objectives of the different activities. The framework should spell out the topics for various targets and age groups, strategies and methods to be used, roles of different players, integration of activities into existing programmes, linkages with other related initiatives, resources available as well as monitoring and evaluation systems. In addition, precise indicators of efficiency and effectiveness should be established. The teachers, parents and participating PLWHAs should be involved in the development of the indicators and the monitoring framework. A clear reporting system should be established. BONEPWA and Ministry of Education officials should give feedback to the schools after the schools have submitted the reports.

Integration of the Project into the Curriculum
The project did not have clear guidelines on how it would be integrated into the curriculum and the school timetable. Hence there was mention of integration but no evidence was available.

Content and Methodology
Field officers did not have adequate skills and information and no practice was provided on how to facilitate children’s learning using child-friendly and child-centred methods. In addition, the field officers had to cope with the constraints of teaching large numbers within a limited period of time with, most times, insufficient and appropriate materials for the pupils. There were general complaints from parents, teachers and key informants that some topics covered in lower primary were not appropriate for the age group. The Facilitators’ Manual that the field officers used handled the topics in a general way. It did not specify topics appropriate for different ages and developmental levels. It does not appear that the preparation of this manual made reference to the current guidance and counseling, and science curricula.

Relevance
In general, the project is very relevant given the extent of HIV/AIDS infection rates in Botswana. Its focus on pupils is a strategic move as childhood is the best time to influence behaviour formation and change. Parents and key informants expressed concern about the appropriate age to discuss sexuality and HIV/AIDS with children. In particular, they were concerned about the imagery used in the materials used for pupils as well as in the media. Finally, the project should take cognizance of people’s cultures

Parent and Community Involvement
There was concern from the parents that they had not been well informed about the project and consequently not many were involved in the project. They felt that it was not enough to expect all the information about the project to be relayed by the pupils.

Impact, Behavior Formation and Changes
Though pupils and teachers reported improvement in knowledge and certain changes in behaviour and life skills, pupils had some glaring misconceptions or were uncertain of the facts. A considerable number of the lower primary pupils have many gaps in knowledge on vital issues such as how HIV is transmitted, how HIV is not transmitted and also in life skills.

Sustainability
It appears that the partners had not thought through and mobilized adequate human and financial resources to maintain the pilot project. The duration of the project was too short to allow for adequate lessons to inform an expanded and sustainable programme. In particular, BONEPWA should consider and identify other ways of mobilizing resources from diversified sources.

Recommendations:

Project Design:

• The project design should

• Include an advocacy and sensitization programme so that the community is aware of the programme. This can lead to increased demand and mobilization of resources.
• Indicate how the inputs of PLWHAs fit within the curriculum and school programmes with the involvement of MOE.
• Include a sensitization programme for principal education officers, head teachers on the objectives of the project, how it fits within the school programmes, their roles and those of the school governing bodies in the implementation of the project.
• Provide a joint orientation and planning programme for guidance and counseling teachers and PLWHAs who will be involved in the school activities.
• Establish criteria for selection of PLWHAs who will be involved in the school-based project and indicate the competencies they should have.
• Post an adequate number of trained PLWHAs in one district for a specific period to allow for full coverage of schools in that district before they are moved to another district.
• Introduce a programme for orienting and involving all the teachers in the school, in particular there should be an input into the TCB programme so that parents can participate in the conversations with teachers.
• Developmentally and culturally relevant materials should be made available in sufficient quantities to ensure quality learning. Imagery portrayed in the materials should be appropriate and relevant.
• The community and grassroot professionals should be involved in deciding on the most relevant and appropriate messages and effective delivery approaches. In this regard they should have been utilized.
• Relevant and appropriate local and indigenous knowledge, skills and methods should be researched on and integrated into the educational programmes on sexuality and HIV/AIDS.

Partnership and Collaboration:

• The Reference Group should be strengthened by incorporating other agencies such as the AIDS Coordinating Unit (ACU) and the Behaviour Change Intervention and Communication Board, and UN agencies, which are key actors in the fight against HIV/AIDS.
• The partnership should define clearly the roles and its mode of operation as well as the expectations of the different partners,
• MOE should play a more central role in the project. Teachers should teach most aspects of the project and involve the PLWHAs as resource people in areas where they give added value such as the fight against discrimination and stigma.
• The mechanism and structure for facilitating collaboration of the partners should be established with clear procedures guiding frequency of meetings, communication and monitoring progress.
• UNICEF should facilitate mobilization of more funds from funding and development agencies.
• The government should support the expansion and ensure sustainability of the project.
• MOE train guidance and counseling teachers and be more involved in training the PLWHAs, supervision in the design of the program to ensure proper fit of the project in the school curriculum and programmes.
• As MOE has a responsibility to the learners, parents, and community regarding the quality and management of education it should ensure that professionally trained educators/teachers teach, impart knowledge, and facilitate learning and acquisition of life skills in learners. Its guidance and counseling teachers who supervise the PLWHAs should be trained, be more involved in training the PLWHAs, and supervision in the design of the programme to ensure proper fit of the project in the school curriculum and programmes.

Coordination:

• The Project Coordination Office in BONEPWA should have qualified persons who are supported with the necessary infrastructure to be able to coordinate the inputs of the partners, and monitor the project effectively.
• The BONEPWA officer should work closely with the MOE officials at the headquarters and in the field.
• The linkages should be established between the support groups of PLWHAs, the project and DAMSCS and Village AIDS Management Committees (VAMSCS).
• The vertical and horizontal communication should be strengthened to ensure efficient flow of information between the different stakeholders and partners.

Monitoring, Supervision and Support:

The following are needed to improve monitoring:

• Specific objectives and indicators for assessing the efficiency, effectiveness and impact
• Monitoring, reporting and feedback system with specification of roles and responsibilities for the monitoring of various aspects.In addition there should be indication of how the monitoring results will be utilized
• Documentation and regular reporting of the activities
• MOE officers should be more involved in the monitoring process working hand in hand with the BONEPWA project coordinator.

Full report in PDF

PDF files require Acrobat Reader.

• Report