2004 BHG: Evaluation of UNICEF's Capacity Development Strategy in Bosnia & Herzegovina 2002-2004
Author: Nyroos, V.
The 2002-2004 Country Program goal of UNICEF Bosnia & Herzegovina was "to improve the capacity of the Government of Bosnia and Herzegovina and civil society to fulfill and protect the rights of children and women". Implementation operates on the three levels of policy, capacity building of professionals at municipal levels and support to community-based responses.
Programming is divided into three sections: Child Survival and Development (with projects on Early Childhood Development and Child-Friendly Basic Services), Child Protection and Participation (with projects on Social Protection, Transformation of Institutional Care, and Mine Risk Education and Mine Victim Assistance), and Policy Development and Advocacy (with projects on Policy Development on CRC and CEDAW and Social Mobilization and Advocacy). The evaluation focused on three interventions:
"Increased knowledge and skills of health professionals on prevention and adequate treatment of acute respiratory infections, diarrhea, dehydration and nutritive anemia"
"A multi-disciplinary approach to combating domestic and other forms of violence against women and children" and
"Capacity building for the monitoring and implementation of children’s rights in Bosnia & Herzegovina".
UNICEF BiH has undertaken several research studies and evaluations of projects during the span of the CP. In anticipation of the new country program 2005-2008, it was felt that a comprehensive review was necessary to assess progress made, and reflect on challenges to inform planning for the future. The evaluation report served as a basis for internal discussions within the UNICEF BiH office in order to inform the next Country Program for 2005-2008. In addition, this summary is shared for discussion at the Annual Review in December 2004 with partners.
The objectives of the evaluation were to assess the effectiveness of capacity development activities within three selected interventions, as well as the relevance of the activities carried out given the wider arena of possible interventions. The evaluation provides an assessment of the capacity development strategy within the Country Program design.
1. Are capacity development efforts effective according to stated goals at the output level?
- Are goals clearly formulated?
- Did capacity development interventions target the appropriate actors?
- Were the methods appropriate for the capacity development objectives?
- Did the capacity development interventions lead to the specified change?
2. Are capacity development efforts in the three projects relevant?
- Did UNICEF make use of its comparative advantage = bringing together government, the international community and NGOs, enabling NGOs to provide leadership - UNICEF MTR Conclusion?
- Did the intervention lead to a change in individual and/or organizational performance?
- Was the intervention the appropriate choice related to other capacity gaps (gaps in responsibility, authority, human, economic and organizational resources)?
- Was the intervention appropriate given the extent of the challenge versus the resources and time frame for action?
Project processes were analyzed through the Triple A cycle (1- analysis of a situation and decisions for action, 2– action, 3- assessment of the results, feeding into a new analysis). The method for data collection was mainly qualitative, since such processes are not readily measured in numbers or figures.
Data sources were UNICEF project implementing partners as well as participants in project activities. For the health project, doctors and nurses were interviewed in 8 focus groups in various locations across the country. For the violence project, observation was conducted and in all, 29 police officers, social workers, health workers, judges and prosecutors were interviewed across the country. For the child rights program, observation was conducted and 4 focus groups with children and youth, 2 focus groups with teachers and 2 with journalism students were carried out in various locations and according to involvement in activities organized by different NGOs. Within the scope and strategy of this evaluation, data collection from beneficiaries such as e.g. children and parents recipients of health care services or victims of intra-family violence, was not possible.
Findings and conclusions:
The Country Program is effective in terms of 'getting things done'. UNICEF BiH results are strong in the area of human resource development, focused on awareness, knowledge and skills of individuals. Goals are clearly formulated, appropriate sectors are targeted, appropriate methods are used and, overall, interventions have led to the targeted change at the output level. Efforts are practical and make use of UNICEF’s comparative advantage – UNICEF plays a critical role in bringing together government and NGOs, and mobilizing and ensuring participation by NGO counterparts. It is a strength that the CP works at multiple levels, targeting all three levels of policy, service provision and community for action.
The Country Program does not ascertain that 'the right things are getting done'. The challenge of sustainable capacity development is greater than current efforts recognize. The justification for capacity development choices is not clear enough, due to inadequate assessment of partners' capacities and inconsistencies in planning documents. Generally, process documentation is lacking, which means that important opportunities are missed for building on successes and gaining a better understanding of the capacity development challenges.
De-motivated professionals can prevent equitable resource and service allocation due to indifference or unwillingness. Therefore, motivational CD needs to be developed to enable such individuals to take on responsibility. As demonstrated by the Medica Zenica project, it is doable, and it needs to be incorporated into other capacity development interventions as well.
Interventions have been pushed in the direction of short-term, tangible results, whereas the largest capacity gaps might sometimes be of a more systemic nature, requiring more long-term, less visible efforts. The effect on organizational performance is relatively small, since specific efforts are not directed at improving performance of institutions or organizations as such. There are some effects, but these changes are not systematized and recorded, and can therefore not be built on. For long-term, sustainable results, more focus is needed on partners' organizational resources, especially strategic planning, process documentation and institutional learning, as these are weak links in all three interventions analyzed. Learning processes mostly exist on a formal document production level, and do not contribute to greater understanding of challenges.
1) Review usage of RBM. RBM places so much focus on reaching the targets that other positive developments might not be documented. This is especially so when projects aim to develop capacity, which is a highly unpredictable process with many possible side effects. For a critical discussion of the use of indicators, see Whitehouse, Chris: the Ants and the Cockroach – a Challenge to the Use of Indicators. UNICEF BiH should discuss the best possible usage of the RBM system in the future, so that the system does not become either a paper exercise at great distance from real life project implementation, or an all-encompassing document that impedes attention to important issues that cannot be captured in a logframe. Possibly through adoption of a context-based RBM system can it become more of a consistent monitoring tool. For guidance, see e.g. Qualman & Morgan: Applying Results-based Management to Capacity Development.
2) Develop strategic and comprehensive CD efforts.
- Together with partners, identify the most crucial capacity gaps (as they might indicate a different focus for UNICEF's support), within a systemic approach to capacity development for change (within social protection, health and education systems).
- Address motivation issues both in the planning stage – a strategy of incentives to achieve project participation by the right people, and also in e.g. training design to instigate commitment and leadership.
- Develop strategies for addressing capacities in the field of authority and responsibility through addressing policies, also at a lower level of e.g. standards of service delivery, and accountability issues.
- Ensure that CD efforts go beyond providing inputs to facilitate change processes - the key often lies less in providing resources than in improving organizational procedures, policies and structures.
- Adapt CD efforts to the needs and circumstances of the organizations they support, not vice versa (pace and direction of needed organizational changes do not automatically fit with the project targets), possibly through inclusion of specific budget lines, so that the individual organization/institution can concentrate some efforts on what they are in most need of for each project.
3) Develop new methods for process documentation
- Seek to develop new tools for process documentation that enable documenting e.g. obstacles that need to be addressed and positive developments not captured by indicators, possibly through more extensive field monitoring reports allowing for reflection on processes.
- Enable partners to take care of more of the monitoring themselves.
4) Focus on institutional learning. Develop a process with partners where planning and assessment systems are integrated to promote individual and organizational learning, and improvement strategies. A learning environment provides a space for impetus for systemic change, creation of crucial incentives and ensures ownership of the development process. All of the above recommendations can, if implemented, contribute to the development of such processes. As a consequence, UNICEF BiH's role would switch to more that of a mentor than that of a manager in the process of capacity development.
Full report in PDF
PDF files require Acrobat Reader.