Evaluation database

Evaluation report

2011 ESARO Regional: Children and AIDS Regional Initiative Programme Completion Report



Author: Sue Enfield and Siân Long

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

The Children and AIDS Regional Initiative (CARI) is a five year programme implemented by UNICEF across nine countries in Eastern and Southern African to strengthen the coverage and quality of the national response to children affected by HIV. The goal of the programme is to improve the human development outcomes amongst orphans and children made vulnerable by HIV. The purpose is to increase the proportion of children affected by HIV receiving family, community and government support. The programme has four outputs that contribute to the achievement of the purpose:

  1. strengthened family and community based responses;
  2. increased access to essential services;
  3. strengthened national policy, planning, monitoring and coordination; and
  4. effective management and technical support provided by UNICEF Eastern and Southern Africa Children and AIDS team at regional level (ESARO).

The initial focus of the programme was supporting national governments to implement their National Plans of Action on Orphans and Vulnerable Children (NPAs). As the programme was implemented UNICEF sought to work with national level OVC coordination committees to build a comprehensive strategy at national and regional levels, developing policy and legislation, strengthening coordination and working with government and non-state (civil society) actors to deliver activities envisaged within the NPA (where this existed) or within sectoral plans. Over time the programme focus has shifted toward supporting the development and implementation of national frameworks to protect highly vulnerable children, recognising diverse forms of vulnerability. The programme has focused on enhancing coordination across relevant sectors and strengthening capacity at community level linking and referring children through to national systems.

The programme, at its inception phase in 2005, was conceived of as a number of linked but largely discrete country-level activities. Over the duration of the programme, the two donors aligned for greater impact across the region. The national-level work is complemented by a regional component; the log frame was revised to a combined log frame; AusAID and DFID aligned their reporting mechanisms and in fourth and final years conducted joint reviews.

Scope of the Review

The project completion review was undertaken jointly, with shared objectives by a seven member team that included AusAid and DFID staff, USAID Senior Regional Technical Advisor (Orphans and Vulnerable Children) and two independent consultants. The purpose of the joint project completion review was to assess the overall performance of the programme and its impact, and the extent to which it has achieved its purpose. The evaluation process was designed in order to:

  1. Assessprogress of the programme against the purpose and outputs of the logframe;
  2. Score the project’s progress against the logframe;
  3. Assess the impacts, both positive and negative, of this programme; and
  4. Make recommendations and identify any immediate action points before programme closure.

Evaluation Findings

The findings of this project completion review are analysed and organised in relation to standard OECD criteria of relevance, effectiveness, efficiency, impact, sustainability, gender equality, analysis and learning (M&E).

Relevance

The programme was highly relevant. The consequences of HIV are immense and continue to be felt across Southern Africa. Prevalence rates continue to be higher amongst women, especially women under 24 years, and the number of people living with HIV continues to rise. Sheer numbers of orphans are of concern. The CARI programme analysis recognises other, non-HIV specific, inequalities. From inception, the CARI programme encouraged a national appraisal of and response to the multi-faceted challenges facing HIV-affected and vulnerable children. This strategic policy engagement continues as national sectoral plans continue to evolve and there is emerging focus on social protection as a poverty alleviation mechanism in its own right. The State in all target countries is scaling up resources to identify, register and provide essential services to vulnerable children and their families. More sophisticated poverty analysis and understanding of the vulnerabilities of many children (developmental, psychological and practical), which has occurred within the CARI initiative has led the discussion to move from a somewhat finite response to the problems of children orphaned by HIV to a greater understanding of the varied reasons that may render a child vulnerable.

 

Effectiveness

Achievements at output level

A review of achievements under Outputs 1 through 3 and the regional elements of Output 4 show clear linkages between the four outputs and coherence within the logframe from output to purpose level. Key results from each output are summarised as follows:

Output 1: Strengthened family and community based responses

The programme has undertaken a number of approaches responding to the need to provide support to families and communities who take on most of the responsibility of caring for those affected by HIV. In all countries the key areas of focus are identification of vulnerable children, appropriate care and support within their communities, and referrals for essential services. Work on identification of vulnerable children has had a positive impact in several ways. Investment in data collection has led to more information about numbers of vulnerable children and their needs. It has drawn attention to the issue of childhood vulnerability and this has led to increased responses.

There has been considerable investment in the types of care provided at community level. There are improved linkages and referral mechanisms among civil society implementing partners and between CBOs and public sector service delivery sites; there are increased referrals of vulnerable children through these community-based bodies, in particular of children entitled to cash grants. Community based OVC care groups support referrals by assisting eligible households through the bureaucracy.

In summary, there have been significant achievements in terms of capacity and scale, most notably where UNICEF has identified an existing strong partner and worked with them to strengthen capacity holistically. There is more robust data collection across the region about which vulnerable children areliving where. This forms the bedrock of future community-based responses. It is hoped that a greater focus on the particular niche of non-state actors, and continued focus on how to ensure effective referrals, will further strengthen this area of work and continue to leverage state funds, directly and indirectly, into community-driven responses.

Output 2: Increased access to essential services

The focus of this set of interventions was to increase access to a comprehensive set of services for vulnerable children that broadly fall into basic services such as health and education, child protection and social protection. Approaches were similar across the nine countries, even bearing in mind the different policy and resource context. Key achievements are found in progress against the following indicators: effective and decentralised birth registration systems; piloting of innovative models for government adoption; and child protection services. Birth registration is for most children the ‘key to the door’ for access to many other essential services, such as health or school. Across every CARI country, birth registration rates were low in 2006 and substantially increased by the end.

Another core area of access to essential services is the growing focus on social protection, and the contribution made by UNICEF to child sensitive social protection responses. Interventions have included support to pilot cash transfer schemes that have provided a robust evidence base about what can be done to reach poor and labour-constrained households. Well-timed research interventions on existing government-funded social protection schemes have highlighted how to reachthe more marginalised and excluded children and increase primary school attendance as well as access to other services. The approach in countries that already have government schemes has been to provide a critical equity and efficiency lens.

Activities under the indicator on child protection have yielded sustainable and substantial results in investment in strengthening the social welfare sector, in terms of numbers of personnel, cadre and technical skills. UNICEF has partnered with USAID/PEPFAR to support capacity strengthening of the social welfare workforce, there is a growing range of child friendly courts, safe spaces within police stations and awareness of gender based violence.

Output 3: Strengthened national policy, planning, monitoring and coordination

This output recognised that delivery of essential services and support of children within their own communities is only possible at scale and in a sustainable way if support through government and non-government means exist within a regulatory framework. It is in this area that it is possible to see a substantial level of achievement. UNICEF has actively used CARI (and other) funding to remain at the centre of ongoing policy debates, contribute to ongoing discussion, as well as respond promptly to external changes in the environment in several fields.

The results are evident. A particular role appreciated by other stakeholders in the field visits has been the convening role played by UNICEF between different government ministries, between government and civil society, and a range of development partners. In 2005/6, there was in general a weak policy environment for vulnerable children. By the end of the programme, all countries have evaluated or strengthened the national plan for vulnerable children but have also advanced substantially on the development of broader, more comprehensive plans of action for children, taking a range of vulnerabilities into account. Whilst it is not possible to ascribe growth in these comprehensive Children’s Acts exclusively to CARI, it is clear to the review team that the opportunity to provide longterm support in terms of advocacy for policy development; capacity building of lead ministries, facilitating key government and non-government partners to work together at times; and strategic research and evidence building across the region, has built a compelling case for comprehensive legislation and helped shepherd these policy processes through. By the end of the programme, seven of the nine programmes in the CARI country have substantially advanced in this area and the other two are actively advancing.

Output 4: Regional policy interventions

This section comments primarily on the regional ‘added value’ that this area of work contributes. One of the most impressive findings is the regional development of a new recognised training system for community workers, both government and non-govenrment staff and volunteers. CARI funds have been used to support the Supported Open Distance Learning (SODL) certificate course for youth and community workers leading to completion of a successful pilot course with an over 80 per cent pass rate, 1,000 new students enrolled and five countries now incorporating this training into their national systems. The initiative is supporting communities of practice across the region.

An area of focus to create regional added value is the engagement with regional economic bodies, SADC and the EAC. Although some work on a SADC level OVYC strategy was achieved in practice it was unclear to the review team what the current level of UNICEF engagement with these two bodies was. Overall, the regional elements of the programme have filled in some of the important pieces of work that complement country-level work. The three commissioned pieces of work that have not been completed by the end of the extension year should be pursued as they will be useful contributions to the policy debate.

Efficiency

Management arrangements have evolved constructively after a slightly difficult start. Governance and management arrangements (between UNICEF, AusAID and DFID; and between ESARO and Country Offices) appear to have been appropriate in terms of reporting and shared learning, and effective in delivering an evolving response to an ongoing need, with some important spin offs in the social protection policy arena. The relationship between the ESARO Office and Country level delegations is one of technical support and oversight but not management and overall responsibility for programme delivery. The role of the regional office has been important in facilitating regional learning; sharing practice; organising study tours and learning visits; and in managing the relationship and reporting to donors. Synergy of work between sections within UNICEF country teams (for example Child Protection and Social protection staff; and between Health and Social Protection units) has grown from a shared focus on vulnerable children. This has brought a measure of operational efficiency and a way of working suggested by the CARI programme framework and objectives.

Impact

At purpose level this programme was expected to deliver an increased proportion of orphans and vulnerable children receiving family, community and government support as measured in quantitative numerical terms. National data has been sourced to provide evidence for progress against these indicators. UNICEF through the CARI programme has made a clear contribution to impact at this level. The overall consensus within the review team is that there has been a strongly positive impact for vulnerable children across all the participating countries as well as some strengthened regional capacity and coordination. This is both in terms of increased numbers of children receiving services and also, more importantly, in terms of an enhanced understanding of what works in terms of delivering quality and equitable services.

Sustainability

Overall, UNICEF’s strong and supportive role of a range of government partners leading to increasing government leadership and ownership is a sustainable approach. Technical assistance and limited yet flexible funding sourced via the CARI programme has helped ‘to start the ball rolling’ or rather to add momentum and evidence, to the national debate on social protection and the development of national structures and systems. There is evidence of greater ‘systems thinking’ with actors (state, civil society and other development partners) considering the needs of vulnerable, poor populations and the duty to deliver services to all citizens, including the highly vulnerable. The programme resources have allowed UNICEF “to take a place at the table and partner in a genuine way around building a social welfare workforce” in target countries.

There has been investment into staff (community childcare and social workers); the knowledge base (concerning knowledge of rights in general and succession planning in particular); and mechanisms to engage with children in communities (the strengthening of community based child protection/OVC groups and committees brought tangible impacts for children). Taken together these have strengthened national capacity in both state and non state sectors and this is an important element of sustainability. The programme has impacted on the institutional and regulatory environment; these policy and legal gains are clearly sustainable.

Gender Equality

UNICEF has a unique role in advocating for implementation of child rights approaches, which would require gender analysis. It was assumed that participant countries would generate sex disaggregated data that would contribute to quality of information with national data systems strengthening. There has been progress over the course of the programme but this area remains weak. The review team noted that in the final year of the programme UNICEF ‘met its target’ of undertaking the planned gender review by ICRW, and that this review found some promising examples of UNICEF gender programming at country level. However, countries and the regional office acknowledge that they have been somewhat gender blind although there is substantial potential for a greater gender focus. This review observes that there is a lack of consistent gender analysis clearly informing the design of programmes and this could be strengthened. The team does note that the strong emphasis on equity within gender over the past year offers substantial opportunity to actively reflect on genderdisaggregated analysis within the broader equity approach.

Monitoring and Evaluation

UNICEF Country Offices reported annually against logframe indicators, accompanied by narrative reports. Detailed financial reports were submitted. Summary reviews took place annually and one further in depth multi-country review on gender in Years 4 and 5 complemented this. Over time the CARI programme has been judged to progress steadily towards a greater, sustained level of support for most vulnerable children across all target countries.

Analysis and Learning

The CARI programme when designed was rooted in analysis of the situation of children affected by HIV and AIDS. Over the lifetime of the programme there has been a major shift in emphasis from downwards service delivery towards more comprehensive and linked efforts to build systems, frameworks and national capacity to deliver a sustained response. The cornerstones of policy dialogue have shifted from orphan status and HIV exclusivity to child poverty and most vulnerable children (MVC).

A solid outcome of this five year process has been that UNICEF, and its partners in country, are now able to articulate a clear evidence-based ‘position’ for child rights and priorities for programming within OVC programmes; and the work is clearly linked to a continuum of care throughout childhood which needs to extend into adolescence and assuming adult community roles. The roles of family, community and state are clearly identified; the changing needs of children and young people are considered; and the whole is linked to child protection interventions. There is strong consensus across the country programmes that upstream work is critical to ensure that strategies for sustained implementation of effective interventions are put in place and take vulnerable children into consideration. As services are rolled out there is a clear desire for standards of services to be maintained. The passage of various Children’s Acts has formalised legislation and adoption of national standards in many countries. UNICEF has been able to bring learning and analysis from investigations of pilot activities, some key staff capacity audits, tracking of disbursements and impact of several child grant or other cash transfer payments systems, to the dialogue at National level.

Conclusion and Recommendations

The multi-year multi-country nature of the CARI programme has allowed for sustained work on some critical areas. The programme allowed UNICEF and her partners to ‘reach for some of the high fruits’ in terms of difficult policy challenges and this has seen better national systems for birth registration and child sensitive M&E data established. Legal frameworks have established child rights and sector policy set standards for services delivery. Important skills audits, focused on gaps in state provision, and linked to appropriate and accessible training has helped to upscale human capacity. Community based child protection groups and committees remain at the frontline of work to identify and respond to children’s needs in communities. The multi-functional support that is necessary at this level is exemplified in the role played by the community childcare workers within the South African NGO Isibindi – but across all countries there are motivated volunteers and staff engaging with children who are highly vulnerable and helping them to access rights and services. The response across Eastern and Southern Africa has been scaled up, in part as a direct result of CARI programme efforts. There is an enlarged community of practice now engaging with issues of highly vulnerable children, and the reach of the policy debate, that is now seeking to target highly vulnerable children within broader social protection systems and mainstream delivery of key services (education and health), is extremely positive.

In furthering these gains, the review team recommends that:

1. UNICEF continue to inform MVC policy discussions and practice drawing on concrete experience with partners, complemented by the research, analysis and learning that has been a tangible output

from the CARI programme. UNICEF should seek to make this learning more accessible in terms of formats and amplify promising practice towards clearly targeted audiences. The review team applauds the broader, vulnerability focus and recommends that UNICEF continues to use political and technical opportunities to deliver health, education and protection to all children through mainstream Ministry Policy and resourcing of this.

2. The opportunity for social protection approaches is used to secure greater budgetary allocation for children sensitive programming and the ESARO team are encouraged to provide additional support to all countries to facilitate cross-learning in this area. Country Representatives within UNICEF and staff at ESARO, are encouraged to promote a greater niche for UNICEF within the macro-economic and poverty reduction strategy discussions.

3. Work on community level care and referrals, delivered by families and other non-state actors, should continue to merit attention since whilst state systems strive to scale up, the day to day survival of children is underpinned by voluntary commitment at community level.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2011

Region:
ESARO

Theme:
HIV/AIDS - OVC

Type:
Evaluation

Language:
English

New enhanced search