Evaluation database

Evaluation report

2007 BH: Joint Country-Led Evaluation of Child-focused Policies within the Social Protection Sector in Bosnia and Herzegovina



Author: Pennarz J., Hilicek RA, Rasidagic KE, Rogers D. ITAD Ltd.

Executive summary

Background
UNICEF in cooperation with the DEP/EPPU Directorate for Economic Planning (former Economic Policy and Planning Unit) has commissioned a comprehensive evaluation of the ongoing process of strengthening child-focused and evidence-based policy making within the Medium-term Development Strategy (MTDS). This evaluation is meant to inform both the ongoing MTDS process and the strategic support provided by UNICEF. The evaluation has been carried by a team of consultants assembled by ITAD between March and June 2007

Purpose
The terms of reference provided by UNICEF have defined the purpose and intended use of this evaluation as an “ex-ante evaluation of the MTDS process to inform and structure the production of the strategic social sector documents in 2007, and to inform UNICEF’s Mid Term Review and UNDAF Evaluation, assessing UNICEF's contribution to the Bosnia and Herzegovina (BiH) Social Protection sector.” However, during the course of the evaluation the purpose has been modified to some extent and it became focused on the review of evidence-based policy making in relation to Children’s Allowances in BiH and UNICEF’s contribution to development of evidence-based, child focused methodologies in the social protection sector. There was agreement between stakeholders that this is not an evaluation of the Directorate of Economic Planning (DEP) who is in charge of coordinating the MTDS process.

Methodology

UNICEF has taken the lead in strengthening the DEPs capacity in the area of country-led evaluations (CLE) under the Paris Declaration commitments. UNICEF has worked closely with DEP during 2006 during the preparation of this country-led evaluation which is seen as a major contribution to DEPs capacities building on M&E. This evaluation has been an attempt to assemble a wider range of stakeholders in a joint review of a specific sector. The process has been conducted in a way that it ensures a maximum of stakeholder participation and country ownership. Time was, however, a main constraint and has restricted the extent to which issues could have been reviewed through a participatory process.

Findings and Conclusions:
Findings on MTDS process
The MTDS process is seen as relevant to support policy change towards more evidence-based and child-focused policies. The process of policy formulation has drawn from a wide range of studies and recommendations which highlighted how the children and in particular families with more than three children are affected by poverty. The measures that have been outlined in the strategy are aimed at structural changes within the child protection system and relevant legislation in general; they are not specifically targeted to the identified needs of poor households with children. However, MTDS monitoring shows that implementation of these measures is lagging behind due to a number of governance issues.

The effectiveness and efficiency of the social protection sector is constrained by the decentralized structure. Legislation is made at entity and cantonal levels (respective district level). This has resulted in a highly fragmented approach to policy development and implementation, leading to significant differences of outcomes in the development of policies on child allowances and service delivery at local level. The MTDS therefore correctly states that standards on social protection are needed, but can only be achieved through adequate legislation and institutional reform. A more efficient social protection n system must build on co-ordination and partnership at all levels, with a particular emphasis on strengthening the capacities for bottom-up policy making.

The BIH government has committed itself to ensuring human rights and social protection for its citizens. However, as a result of the difficult socio-economic situation and weak governance there are gaps in the protection of these rights. The MTDS identifies the poorest and most vulnerable groups and states the need for better targeting. It is obvious that in the present situation the state does not have the capacities and financial means to address the needs for social protection and to ensure equal access to welfare programs throughout its territory. But, the evaluation also found that the priorities applied to the social protection sector often do not reflect the actual needs of the poorest and most vulnerable groups. It is therefore encouraging that the MTDS includes indicators on poverty and social security together with targets for building a monitoring system which are meant to strengthen accountability within the social protection sector.

Findings on MTDS monitoring
EPPU has been established as central monitoring institution for monitoring progress under the MTDS. Furthermore institutional responsibilities have been clarified at the state level which assigns roles to specific government bodies. But at the moment there is no role for government levels yet. Experiences reviewed during this evaluation show that monitoring at local levels will make a contribution to improved governance and that municipal level authorities therefore should have a role in monitoring MTDS targets. Civil Society has been assigned a role in monitoring the MTDS; but their reports need yet to cover key indicators in relation to child protection.

To address lack of statistical data on poverty a number of surveys has been funded by international donors as part of the MTDS process; withdrawal of support now leads to a shortage of data on poverty and social indicators. Availability of data at state level could be improved through coordination of data collection approaches and sharing of information across institutions. Donors have provided or supported virtually all of the data gathering exercises that have informed policy formulation; they still need to step back from a gap filling role and support the government to lead in policy analysis, data collection and monitoring, even if in the short term this is at the expense of quality.

Poor coordination of support has distorted the demand for socio-economic data on the national side. Independent studies that were meant to influence policy making did not have the expected impact because they did not address policy makers’ concern. Policy makers question the quality of evidence provided by independent surveys so far, but they think that NGOs have an important role to play in exposing critical issues within the MTDS process. Research and studies need to be tailored to meet the priorities of policy makers. At the same time outcomes from research programs and studies can only be influential if they are effectively communicated to policy makers.
Findings on UNICEF-supported activities in the child protection sector:
UNICEF has identified an important “policy window” for shifting the agenda towards evidence-based policy making through monitoring and survey initiatives. It has been successful in gaining high-level support by engaging government partners at the state level. UNICEF uses a strategic approach for building links with government partners which is very effective in strengthening the policy dialogue on priority issues such as social protection.

The evaluation has reviewed three of UNICEF supported activities: the human rights based approach to programming (HRBAP) in child protection has promoted a human rights based approach to address gaps in service provision at local levels; the project to address iodine deficiency disease (IDD) has introduced data collection methods that are relevant for informing policy processes; the Participatory Action Research (PAR) project has developed an approach to identify and support vulnerable groups at community levels.

The projects present different entry points in supporting a shift towards evidence-based policy making. HRBAP presents an example where a participatory approach has been used to identify priorities for improved child-protection services and provide evidence for local government decision-making. IDD is an example where evidence has been collected which directly addressed policy makers’ needs for information. PAR presents an example where good practices have been used to address wider issues of exclusion.

The review of project experiences shows good practices for successfully influencing policy processes. The case of IDD that proper identification of policy windows and policy maker’s demand is a precondition for creating relevant evidence. HRBAP shows that a participatory approach creates helps to create legitimacy and wider support. PAR shows that communication of good practices changes helps to influence attitudes and mindsets. However, the evaluation found it difficult to establish evidence on the extent to which data have been used in policy making processes.

UNICEF uses a strategic approach for identifying strategic partners which is however constrained by the limited number of NGOs; there is a risk that the capacities of partners are overstretched in individual projects. NGOs still need to build capacities and links to influence government decision-making.

The review of project experiences shows a number of good practices on how to address governance issues. HRBAP is an example where cooperative mechanisms have been effective at local levels; links with central levels however have not been strong enough to influence central level policies. HRBP used advisory committees at the central level which are meant to strengthen communication links; but they need to be formally acknowledged to become effective in policy processes.

Government partners value the direct benefits derived from projects which are directly relevant for their routine work, such as IDD. The evaluation found that ownership at local levels is strong; Municipal Management Boards (MMB) in HRBAP pilot municipalities see their roles in policy reform processes as substantial, structured, and effective. It has therefore been noted that as a result of UNICEF’s partnership approach, national partners are changing their perceptions on ownership; but, they need to see their role and contribution to the process

Findings on donor co-ordination
The Paris Declaration sets out the global aid effectiveness agenda. All major donors in Bosnia and Herzegovina are signatories to the Declaration, but currently the BiH State Government is not. Donors have established a forum for donor harmonization and coordination. The government of BiH is not included in this forum nor is it treated as an equal partner. It is apparent that the situation in BiH falls well short of the aspirations of the Paris Declaration.

In 2007 a Donor Mapping exercise has been conducted which did not yet lead to a workable approach for improved coordination. On the other hand, the Bosnian Government is reluctant to take over a role in donor coordination. Recently, the Council of Ministers prepared a strategy for setting up a new coordination body; however, this process is still donor driven.

Donors have difficulties in addressing weak capacities and governance issues within their partnership approaches and they tend to take over a dominant role. As a result, national stakeholders have only a limited sense of ownership of donor-funded programs and the resulting policy changes.

Donors appear to be taking a short term perspective, avoiding the more politically sensitive areas such as education and social protection. But, withdrawal of support and funding threatens to undermine institution building and governance in the social protection sector. At the same time, this situation creates opportunities to engage on priority issues for organizations with an ongoing commitment such as UNICEF.

Conclusions
Policy Context: The MTDS provides a platform for national and international stakeholders to engage in policy processes around priority issues such as social protection. For UNICEF and its partners, the MTDS is an important vehicle for addressing (and advocating) issues of child protection across different ministries, agencies and donors. The MTDS as document reflects relevant strategic priorities to implement and monitor measures to improve social protection.

Efficiency and effectiveness of the social protection system are affected by a number of governance issues. It is a major constraint for the social protection sector that policies are defined at entity, district or even municipality levels. The lack of central-level coordination makes it difficult to identify effective entry-points for influencing sector-wide policies. This has, for example, prevented the implementation of uniform standards on child allowances throughout BiH. It remains a challenge to link policy processes at different levels. But, the dynamics and positive experiences created at local levels can clearly create a momentum for driving policy change at higher levels. Therefore, in a fragmented and multi-layered system like BiH it appears to be critical that more support is given to aspects of vertical integration. UNICEF together with a few other donors has piloted some useful approaches to strengthen feedback of local experiences to central-level policy makers.

Evidence: Availability of good quality data on key social and economic indicators is critical for informing policy processes. Experiences from the past show that where good information has been made available, the quality of policy documents has been improved in terms of identifying the neediest groups within society and designing targeted support to address their needs. However, it remains a challenge to design, implement and monitor specific measures to target these groups.

Among the donors involved, UNICEF has made significant efforts to define information needs in partnership with state institutions which as a result have contributed (or are expected to contribute) to the availability of socio-economic data and in particular child-related information in BiH and influenced subsequent policy processes. An important lesson from these experiences is that outcomes from research programs and studies need to be communicated to policy makers if they want to become influential. UNICEF’s partnership approach has laid a good ground for shifting emphasis to more demand-led approaches in M&E. The partnership with DEP is seen as very important in that aspect.

Links: The provision and use of evidence for policy making is constrained by a number of institutional issues. The link between data collection and analysis on the one side and use of data for decision making on the other side is missing or weak. The capacities of the statistical bureaus are weak and their data collection exercises are not coordinated with the demands of policy decision makers. Also, the vertical integration of the system is weak; lower levels are usually not involved in the monitoring system. UNICEF in cooperation with DEP has piloted approaches (MICS, DevInfo) which are likely to overcome these issues. To support evidence-based policy making it will be critical to work at entity and municipality levels on monitoring key social development indicators.

UNICEF has assumed a high-profile role in advocating child-focused policies in BiH. It has done so through working in close partnership with both government and NGO partners. It seems now that partnerships with government institutions and especially with central level institutions are becoming more influential. UNICEF recognizes the need to strengthen capacities of state-level institutions, like DEP. Partnership with DEP is likely to gain further strategic importance in the future. Not only that DEP is one of the key institutions driving the MTDS process; it is also one of the institutions that may strengthen its coordinating role at all levels if it continues to build its skills and capacities for communication and facilitation.

In addition, UNICEF has built solid partnerships with government institutions at all levels through implementation of a number of activities which are of direct relevance to its partners. The Ministry of Health has become a key partner as a result of implementing projects such as IDD. These partners clearly recognize the profile which UNICEF has on child-focused issues and policies. These partnerships, and notably the partnership with DEP, will also provide a good starting point for strengthening the FBiH government’s role within the process of donor coordination.

Recommendations:
Recommendations to DEP
The MTDS process will be important to build capacities for evidence-based and child focused policies at state levels. For this, it should be a priority for support to strengthen coordination at the state level and to establish minimum standards for child allowances throughout BIH. For DEP, this will be an opportunity to further define and subsequently strengthen its role as convener of different groupings, such as the Reference Group, and around identified themes, such as social protection.

In order to influence social protection effectively it is critical to identify approaches which would help to take policy processes like the MTDS to entity and municipality levels. DEP/EPPU in cooperation with donors, including UNICEF has gone a long way to initiate approaches which create space for entities and municipalities to engage in state-level policy processes, like the Reference Group for this CLE, the OSCE supported initiative and the MTDS working groups.

DEP’s partnership with international and national partners including UNICEF is crucial for strengthening M&E capacities within the country. DEP has taken some fresh approaches to strengthen municipalities’ capacities through DevInfo but also as a result of work through the LSE/DFID project in monitoring key social development indicators and improving vertical and horizontal communication which an essential element of a well-functioning M&E system. Theses approaches should be strengthened and further developed:
• DevInfo is an important project which may provide important lessons on how to improve data collection and analysis at municipality levels and how to strengthen vertical integration into national monitoring systems.
• MICS will be important to show how generic social development indicators can be adjusted to the specific context of BiH. Survey outcomes should be well communicated to decision-makers to present them with some ideas on how poverty impacts of policy measures could be monitored.
• Since DEP has already taken initiatives for better coordination of actors at different levels, it should also find ways to communicate the outcomes of its various M&E projects to stakeholders and decision makers in these forums. The presentation of evidence collected through these projects will be important for shaping the discourse and process on social policies at all levels and throughout different institutions.
• This CLE has been important to establish a platform and explore new approaches for joint review of key social protection policies by representatives from different ministries and localities. The momentum created by the process should be continued through further engagement of the reference group in follow-up activities.
A major challenge for DEP and UNICEF will be to link analysis and presentation of monitoring data to the right level and institution where policies are made and implemented. A top-down dissemination approach of study outcomes from state to lower levels alone will not be effective; decision-makers need to see the direct links between policies and outcomes at the level at which they are operating. For social protection policies, this would be the entity, district and municipality levels which are particularly weak on M&E at the moment.

To strengthen its role and approach to coordination it is recommended that DEP should work out a work plan which integrates the various initiatives and projects into a strategic perspective. The strategy for coordination should identify groupings of institutions that would be convened by DEP around certain themes which need to be identified through a consultation process. Elements for a strategy have been suggested in this report.

Recommendations to UNICEF
UNICEF BiH will undergo its MTR later this year and, on the base of this review, prepare a new CPAP in 2008. The action plan will continue to focus on issues of social inclusion, with a special focus on children and youth. It is expected that within this process UNICEF will redefine its strategy to influence child-focused policy making.
The following recommendations from this CLE may assist UNICEF’s strategic development
• Positioning: UNICEF is trying to strike a balance between being an organisation with a specific mandate for children issues and the role (together with the experiences) it has acquired in promoting wider issues which in particular include partnership strategies and capacity building for M&E related tasks. It appears that the perceptions other organisations, including national partners, have of UNICEF’s mandate are not identical with what UNICEF’s aspirations and potentials are if they want to achieve greater influence in the social protection sector in BiH. Greater clarify could be achieved through a joint exercise with key partner mapping out UNICEF’s future role and contribution and position the organisation within the setting of donors and institutions in BiH.
• Donor coordination: Similarly, it is important that UNICEF tries to strengthen its influence within the donor community to push for compliance with Paris Declaration Targets. UNICEF’s partnership with state-level agencies including DEP and its commitment to build capacities and ownership at central levels has placed the organisation well in comparison to other donors. This evaluation is another step to prove that the BiH government is willing to take over greater responsibility in the process. UNICEF should feel encouraged to take a stronger role in advocating coordination within the donor community together with like-minded donors.
• Partnership: UNICEF’s partnership with DEP provides a sound base to continue strengthening M&E capacities within the BiH government. It is important that the present activities to support capacity building within DEP which have more of a pilot character and are focused on technical aspects at the moment are placed into a strategic perspective. A better understanding of M&E systems and the associated institutional dimensions will help to ensure that these activities can be institutionalised and scaled-up in the longer term. There is scope to strengthen its influence on governance issues through support DEP together with other agencies involved, namely the statistical bureau, to adopt a more systemic view in building roles and relations on M&E at all government levels which are likely to strengthen accountability and feedback for improved social services.
• Thematic focus: UNICEF is supporting a number of activities within a broad perspective of influencing selected issues. Links between policy issues that are of key concern for decision-makers and issues that are promoted by UNICEF can be improved through a process-oriented approach for strategy and project development which builds on close consultation with key partners. This point is actually about striking the balance between being led by a strategic issues and being responsive about issues raised by partners. The mapping exercise to identify partners and priority areas is as a good starting point and should be elaborated in that respect.
• Networking: UNICEF’s involvment in promoting networks between partners around shared issues could be strengthened. However, it is important that experiences from individual projects and activities are scaled-up through networked relationships in order to become influential. It should be a priority to strengthen the facilitating role of institutions which have a genuine interest in building partnerships and coordination like DEP in the first instance. Also, the recent initiative supported by OSCE  to coordinate activities at various government levels is likely to provide a window to networking.
• Communication: Any study or project experiences that are meant to influence policy processes must be communicated to decision makers. A communication strategy has been foreseen in the present CPAP. It is important that the new CPAP will similarly develop a communication strategy which is tuned into the revised strategy to influence policy processes on key issues identified in the plan. Previous experiences on communication and influencing within the government system should be reviewed in greater detail during the preparation.

These approaches should underpin the strategy for influencing policy processes. Elements for this strategy have been identified in this report.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2007

Region:
CEE/CIS

Country:
Bosnia and Herzegovina

Type:
Evaluation

Theme:
Ecomomic Issues

Language:
English

Sequence Number:
2007/002

New enhanced search