2010 Zimbabwe: Evaluation of Programme of Support for National Action Plan for Orphans and Vulnerable Children Impact/ Outcome Assessment
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
It is estimated that more than one in four children in Zimbabwe has been orphaned in Zimbabwe. Many children classified as orphans still have a surviving relative or parent and there is therefore the need scale up support through families and communities where possible in line with the internationally endorsed Framework of Protection, Care and Support for Children living in a world with HIV and AIDS and more recent communiqué from the Global Partners Forum on Children and HIV and AIDS. As well as children who have lost one or both parents, there are many highly vulnerable children living on the streets, those who have engaged in hazardous child labour, children with disabilities and those living with HIV and AIDS who require special attention.
Child vulnerability has been driven by a combination of factors in Zimbabwe including HIV/AIDS and the economic downturn which has led to the collapse of many basic services. The National Action Plan for Orphans and Vulnerable Children (NAP for OVC) is the vehicle through which the national response to the plight of OVC is being implemented. The programme was developed in a collaborative manner involving government line ministries and development partners. Following the costing of the NAP for OVC programme the Programme of Support (PoS) was developed to provide a multi-year – multi-donor and predictable source of funding for the implementation of the NAP for OVC programme. The design of the PoS is in line with DAC/OECD principles for working in fragile states.
DFID NZAid, SIDA, Germany, AusAid, Netherlands and the EC have since contributed to the PoS creating a pooled fund of around US$ 75 Million to date. DFID is the largest contributor to the pooled fund with a commitment of £23 million over 4 years. Whilst funding continues to be channeled through UN and NGOs, Government of Zimbabwe has played a constructive and lead role in this programme.
The PoS was launched in 2006 following Cabinet approval for the Guidelines for the Operationalisation of the NAP for OVC. Immediately after, a national call for concept papers was launched, inviting civil society organizations to submit their proposals in line with the seven objectives of the National Action Plan for OVC. More than 160 concept papers were submitted and after a thorough and transparent process, in November 2006 21 project proposals were approved by a Technical Review Committee composed of 5 government ministries, National AIDS Council (NAC), donors and UNICEF. In February 2007, tripartite agreements were signed between the 21 organizations and the MoLSS and UNICEF for a duration of 3 years, starting implementation thereafter.
Disbursements through the programme have been slower than expected partly due to the challenging operating environment over the last few years. Most partners were unable to operate for more than 6 months in 2008 due to the elections and suspension of NGO activities. Moreover the unstable economic environment caused many implementation challenges.
This study is an impact assessment of the Programme of Support, in the build up to the third Annual Review that will be conducted early 2010. This study will feed into the Annual Review and aims to outline the efficiency, effectiveness and relevance of the programme, assess the impact the PoS has had on OVC the last three years and examine the value and cost effectiveness of the programme
The purpose of the study was to assess the extent to which the Programme of Support has reduced the overall vulnerability of OVC in Zimbabwe,2 by analysing the Programme’s ability to meet its stated objectives. Linked to this, there were four subobjectives to:
• assess the outcomes of interventions as well as the number of children being reached by the programme’s NGO partners;
• analyse the cost effectiveness and value for money of the different approaches adopted by the Ministry of Labour and Social Services (MoLSS) and UNICEF (e.g. NGO-led Vs BEAM) to reduce children’s vulnerability and compare them with alternative approaches in Zimbabwe and the region;
• identify the benefits of different approaches to reduce the vulnerability of OVC in Zimbabwe and assess their strengths and weaknesses;
• highlight opportunities, challenges and lessons learned for the design of a future programme.
The research involved the following activities:
• a desk review of reports and other documents provided by UNICEF and DFID, and of information on similar programmes in Zimbabwe and in the region;
• extensive interviews with key stakeholders in Harare, including MoLSS, UNICEF, OECD/OVC group and NGOs, and a two hour session with over 40 NGOs during the UNICEF-NGOs quarterly meeting;
• field research involving interviews with 1,326 children (926 ‘treatment’ and 400 ‘control’) and household heads in 15 wards in 10 districts (see Map in Figure 1), using a structured questionnaire, focus group discussions and interviews with key informants (from the community and government officials). The districts, wards and households were randomly selected. All OVC in the households were interviewed; and Data analysis using SPSS (Statistical Package for the Social Sciences) and discourse analysis of the qualitative information.
Findings and Conclusions:
The overall rating scores proposed by the consultants, against OECD evaluation criteria are as follows:
Overall programme B
6.2 The OVC approach was relevant in the context of a complex emergency situation. Elements of the approach remain valid and will continue to be so in future. When the PoS was launched in 2005/06, government had limited funds, there was an urgent need to extend the basic services to large numbers of unserved OVC and to protect children from abuse. These conditions still exist today and, at least for the next few years, the PoS will continue to be needed.
6.3. UNICEF, MoLSS and partners have achieved a great deal in very difficult circumstances. Outputs have been delivered and this has been done efficiently. Almost 400,000 OVC have received some benefit from the programme and implementation rates are high. UNICEF has striven to be as efficient as possible (e.g., reducing overheads, implementing the SG model and using centralised bulk procurement) in order to direct as many of the funds as possible to NGOs for OVC, which is commendable.
6.4. However, in the Mission’s view, the programme could have achieved better outcomes for OVC and higher overall value-for money (VfM). UNICEF may have cut its overheads and operating budget too far. By spending a little more (e.g., to procure vehicles more quickly, to employ a Grant Manager with necessary staff, to monitor quality and learn lessons in a timely way) it could have delivered better for OVC. The Mission did not spend long enough with NGOs to come to a view on whether their overheads and operating budgets are adequate.
Recently, UNICEF has made important changes to the programme (e.g., capacity building emphasising mentoring and horizontal learning; involvement of MoLSS staff in trainings; and operational research on innovative approaches to capture and disseminate best practice). However, further improvements are needed to improve the ways in which UNICEF and its NGO partners deliver for OVC. The mission recommends that consideration is given to:
(a) Strengthening the monitoring and evaluation system so that UNICEF can:
• monitor the quality of services provided by NGOs and SGs, against NAP standards which would enable UNICEF and MoLSS to assess the relative efficiency and effectiveness of different organisations and improve outcomes for OVC. The DSS at provincial and district levels has a key role to play in quality assurance and ways should be found to finance this; check the quantitative self-monitoring data provided by NGOs and SGs (e.g., though unannounced visits by UNICEF or consultants appointed by UNICEF) and devise a system for self-reporting of the quality of goods and services delivered;
• undertake rapid operational research assessments of key aspects of the programme in order to understand the reasons for successes and failures and disseminate these lessons across the programme;
• better assess the outcomes and impacts of the PoS, by undertaking tracer studies of a random sample of OVC who benefitted from the main programme interventions to assess outcomes (behaviour changes) and impacts (sustainable improvement in the quality of OVC’s lives). These studies would require detailed interviews with children, their carers and the service providers.
(b) Strengthening the grants management unit in UNICEF to minimise the time UNICEF technical advisors spend on administration thus freeing up time for them to (a) visit the field more frequently and (b) ensure that learning from the field is used to improve the programme (i.e., are fed into organisational learning).
(c) Clarifying and revising grant agreements and performance indicators for NGOs and SGs to encourage them to (a) maintain contact with OVC over time (e.g., though followup visits) and (b) take a more holistic and comprehensive approach to the needs of individual OVC, which takes account of their households’ needs.
Producing operational guidelines on the NGO-SG partnership. These should include: (a) specifying how funds will be shared between NGOs and SGs; (b) ensuring that SGs are provided with capacity building, which addresses their needs; and (c) specifying performance standards for the NGO and SG in respect of monitoring, flow of funds and supervision; and (d) recommending regular workshops among SGs to discuss any problems they face and identify solutions. NGOs and SGs should have recourse to UNICEF and/or district DSS staff, if the guidelines are breached.
6.6 The PoS has benefitted 397,000 OVC31 and 517,000 more children from the revitalisation of BEAM. The PoS has already reached a significant proportion of the 1.35 million orphans and other vulnerable children it defined as its target group.
6.7 However, while some interventions have been effective (e.g. education, psychosocial care), the fragmented approach (with many OVC receiving only one or two benefits, and not necessarily their key priorities) means that outcomes have not been as good as they could have been.
Before the end of the PoS in December 2010, it is recommended that MoLSS, the OECD donor group and UNICEF should undertake the following next steps:
1. Design a 3-year a follow-on Phase II Programme of Support (2011-13). The new programme would aim to:
• build on the success of the PoS extending its reach to cover all children in ‘Category D’ households in Zimbabwe with comprehensive high quality packages of support;
• develop and test new approaches to social protection (e.g., cash transfers, removal of user fees in water and sanitation, health, education, etc) whilst at the same time strengthening the Family and Child Welfare system to ensure quality protection services to the most vulnerable, with special attention to girls and children with disabilities. This would complement work in other social sectors to improve access to and quality of health, education, birth registration, nutrition services, etc) and support the scaling up of these so that they become building blocks of the Government’s new National Social Protection Framework;
• enhance the capacity of Government to take over full management of the National Social Protection Framework that is child and HIV sensitive and that duly recognizes the special vulnerability of orphans and HIV-affected children, once in place.
2. Further strengthen and consolidate the PoS ‘model’ so that it can be expanded rapidly and effectively in Phase II. This would include:
• ensuring that each intervention is guided by minimum standards, including reviewing the current MoLSS standards, revising these where necessary, training partner NGOs, SGs and district DSS staff in their use, and mainstreaming these into on-going NGO projects;
• experimenting with different ways for NGOs (probably initially in a district or group of districts) to work together comprehensively to address the needs of OVC, combining by different skills (e.g., in education, health, food and nutrition, psycho-social support, economic strengthening and child protection) using an alternative approach to sub-granting;
• assessing the capacity and effectiveness of UNICEF’s current NGO partners, with a view to reducing their number, while developing their capacity more effectively to work together and with sub-grantees (including producing operational guidelines on the NGO-SG partnership and providing hands-on support such as tailored training and mentoring during implementation); defining clear criteria for partnerships with NGOs based on financial and technical competencies;
• strengthening the capacity of DSS staff coordination structures at both national and sub-national levels including providing them with the needed financial and logistical resources to enable them to play a fuller and more effective role in both coordination and quality assurance of NGO service delivery (reach, quality, follow-up and cost-effectiveness);
• identifying and documenting best practice for all main PoS interventions, promoting wider application by primary NGOs and SGs and, where feasible, finding ways to introduce these into government service delivery systems;
• strengthening programme management and monitoring by putting in place systems to: (a) check the accuracy of quantitative self-monitoring data provided by NGOs and SGs; (b) assess the quality of support being provided by NGOs and SGs; (c) undertake rapid operational research assessments of key aspects of the programme in order to understand the reasons for successes and failures; and (d) assess the outcomes of the PoS for OVC through ‘tracer’ studies; and
• UNICEF undertaking a review of its PoS management arrangements to assess whether there is a need to strengthen staffing and/or systems to more effectively deliver for OVC and to ensure value for money.
The aim of all these actions would be to (a) build the capacity of the PoS to deliver more effectively in Phase II and (b) put these improvements in place before the start of Phase II.
3. Generate up-to-date and reliable statistics on OVC to inform programming by commissioning a ‘National OVC Census and Status Survey’ to generate up-to-date information on demographic characteristics of OVC, their spatial distribution, and the main pull and push factors determining their vulnerability. Special attention should be paid to the situation of children separated from their families and living in residential care. This information would form the basis for formulation of a more comprehensive social protection programme for Zimbabwe. The Central Statistical Office, the Department of Social Services, the National AIDS Commission, Ministry of Health and Child Welfare and the Ministry of Education, Sports, Arts and Culture should be actively involved in the survey.
4. Research on Orphaning to improve programme design (e.g., on the links between high rates of maternal mortality and orphaning; on what happens to orphans taken into care by external families (e.g., possible discrimination); on the extent of differences between OVC and non-OVC in access to services), including on HIV-affected orphans in families and communities. Assess how the delivery of critical complementary OVC services by Government can be strengthened, including how to bolster birth registration capacity and services of the Registrar General’s Office, strengthening the role of the DSS in the family reunification of abandoned children, and role of the public health sector in continuing with pediatric ART and other critical complementary health services for the OVC. To effectively tackle this, the new PoS should have a strong component on policy advocacy.
6. Evaluation of BEAM to improve the reach and quality of support provided to OVC in the period of transition (2011-13) to a National Social Protection Framework, which would include the abolition of fees in health and education. The evaluation should include a school-level research to discover whether BEAM is costeffectively resulting in quality education and to make recommendations on how to improve the system, including on the role of NGOs.
7. Coordinate more effectively with other programmes (e.g., PRP, ETF, SAT, Children First) to ensure a variety of social transfers to OVC and other extremely poor households and to benefit from possible synergies (e.g., sharing beneficiary lists, mapping of activities in each district, joint monitoring, avoiding duplication). This District Social Services Teams should play a key role in this process.
Full report in PDF
PDF files require Acrobat Reader.