“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
In July 2011, UNICEF engaged Universalia Management Group to evaluate its Education and Child Protection Programmes in Bangladesh. The evaluation examined the programmes’ performance between 2006 and 2011 at the outcome level and provided recommendations for future programming. The evaluation intended to inform policy advocacy and resource mobilization work as well as programme management and decision making for the country programme cycle 2012-2016.
The evaluation intended both leveraging purpose (e.g., providing evidence to inform policy advocacy and resource mobilization work) as well as a programme management and decision-making purpose. In this respect, the findings, recommendations and lessons learned from the evaluation will be used to inform UNICEF’s strategies, programme approach, management structures, and the monitoring frameworks of the two programmes for the next programme cycle due to begin in 2012.
The evaluation was carried out between July and December 2011 in conformity with the evaluation Terms of Reference (August 2011) and international evaluation standards. The Evaluation Reference Group provided guidance and feedback and approved the final evaluation methodology and matrix that served as the basis for data collection and analysis.
The Evaluation Team conducted:
• Document review of key UNICEF guiding documents and strategies, as well as Bangladesh Country Office (BCO) country and sector plans, strategies, reports and evaluations
• Interviews and focus groups with approximately 85 individuals, including representatives of the Government of Bangladesh, non-governmental organizations, UN agencies, bilateral development agencies, UNICEF in Bangladesh, and the UNICEF Asia Pacific Shared Services Centre in Thailand • Observation/site visits to UNICEF BCO and to Jessore and Barisal in July 2011 to meet with UNICEF BCO partners and beneficiaries in the districts, observe UNICEF programming and activities, and interview UNICEF BCO staff based in and/or working in the districts.
Findings and Conclusions:
BCO Programme Context In 2011, UNICEF‟s Education and Child Protection programmes in Bangladesh are at different stages of evolution. They vary significantly in terms of their length of existence in Bangladesh, the maturity of their programmes, their accomplishments, their size and their value. UNICEF support for the educational needs of children dates back to the early 1970s; this contributed to its strong and positive reputation in the country today. UNICEF has been able to leverage its credibility with GoB and the development community into increasingly strategic interventions over the past few years with acknowledged success, respect and financial support.
UNICEF‟s Child Protection Programme is in much earlier stages of development and is less known and understood in Bangladesh. Until 2008, it was largely issue-driven and project-focused. Since then, BCO has revised its programme strategy to make it more systems-focused, in keeping with UNICEF‟s 2008 corporate child protection strategy. In comparison to its educational counterpart, the Child Protection Programme has relatively limited access to staff and financial resources, and is generally less flexible in terms of support that it is able to generate.
Over the past six years UNICEF education and child protection programming objectives in Bangladesh have both been highly relevant to the Bangladeshi context and to the needs of children, particularly the most vulnerable. The programming objectives have been congruent with the priorities of GoB, UNICEF and UNDAF, and have complemented the priorities of bilateral donors and civil society.
The Education Programme has been effective in realizing most planned outcomes. It is held in high regard and has a positive reputation among consulted stakeholders. UNICEF has been able to leverage its results and resources in the education sector to obtain broader impact at the national level. Its internal strengths, programming choices, and approaches have contributed to its effectiveness in the Education Sector. Most of the constraints to effectiveness in education can be attributed to contextual causes.
The Child Protection Programme realized several planned CPD 2006-11 outputs over the period but had relatively modest success in realizing planned outcomes. This reflects a combination of factors including some overly ambitious outcomes given the internal and external contexts, as well as several challenges associated with developing and implementing a radically different strategy in the face of staff shortages, relatively few BCO staff members with experience in this new approach, and limited flexible financial resources (due to projects that were conceived and developed in an earlier era). There is a significant disconnect between what UNICEF has been implicitly trying to accomplish in the BCO Child Protection Programme since 2009 in its annual workplans and what it reports as its outcomes.
UNICEF BCO strategies contribute to the sustainability of programming at the individual level in both programmes. Sustainability of results at the institutional level is mixed in the Education Programme and modest in the Child Protection Programme. This reflects the approaches used to design and support capacity development at the institutional level as well as mixed attention in the programmes to the creation and use of sustainability strategies. Internal BCO reports and some external reports suggest that there is insufficient attention to tracking and reporting on sustainability of results on an ongoing basis. For example, while UNICEF supports improved teacher training, no data is available to show to what extent teachers are practicing improved teaching methods in the classrooms. Analogously, while the Child Protection Programme has provided significant support for several aspects of the birth registration system in Bangladesh, there is no specific attention paid in reviewed BCO reports to the likely or actual sustainability of this system over time.
UNICEF programme planning in Bangladesh has been adversely affected by the absence of clearly articulated theories of change, in particular in the Child Protection sector. The Education Programme design is logical and results oriented, although it lacks an explicit theory of change. One of the key benefits of having a theory of change is that all stakeholders can see and understand the logic of the entire programme and where and how their efforts will contribute to the higher level expected results. Another benefit is that a theory of change can help programme managers test and modify interventions as they evolve, and identify possible alternatives.
Over the programme review period, both the Child Protection and Education Programmes have initiated several measures to leverage UNICEF resources for the benefit of children. While the Education programme has been very successful, the Child Protection Programme would benefit from a more deliberate focus on leveraging. Both programmes fostered some valuable south-south exchanges in the past few years, which complement domestic leveraging efforts in UNICEF BCO.
While CPD 2006-11 documents emphasize convergence as a programming approach, the concept is not commonly understood among interviewed UNICEF BCO staff.
Over the programme period, there has been limited synergy between the UNICEF BCO education and child protection programmes.
Finally, UNICEF BCO‟s existing results planning, monitoring and reporting systems have a couple of shortcomings. One is that the system to formalize changes to results established in the CPAP needs clarification. A second is that existing monitoring and reporting systems pay insufficient attention to tracking the cumulative performance of programmes in realizing planned outputs. Given that planned UNICEF BCO outputs were generally quite significant in terms of expected achievements (e.g., decentralized primary education and management practices at school, upazila and district levels; local government and other relevant institutions operate functional birth registration system) it would seem very important to report on their cumulative achievements over time. At present, annual programme reports report on annual achievements only.
UNICEF BCO should articulate clear theories of change for the Education and Child Protection Programmes in the CPD 2012-16.
The evaluation flagged several limitations with the design of the Education Programme and particularly the Child Protection Programme. Both programmes would benefit from more explicit theories of change. This would require UNICEF BCO stakeholders to:
• Clarify the overall developmental objectives of each programme, ideally in each of the identified strategic areas;
• Identify key operational objectives for the programme and clarify how they are linked to developmental objectives;
• Clarify how individual investments, projects, and other activities (existing, new, or future) will contribute to the overarching objectives; and
• Identify the core assumptions underlying the programme strategy.
Based on an agreed theory of change, the BCO could develop programme results frameworks that would provide a basis for tracking progress and how individual projects contribute to overall objectives.
UNICEF BCO should continue to identify ways and incentives to support cross-programme synergies.
The evaluation noted the modest level of synergy that exists among BCO Education and Child Protection Programmes at the moment. There are several immediate opportunities for the two programmes to work more synergistically. One is for both programmes to explore how UNICEF‟s expertise in child protection can be dovetailed with UNICEF‟s respected relationship with the Ministry of Education to influence changes in the education sector that would support child protection in Bangladesh. One suggestion would be for both programmes to explore with the Government ways to insert BSST/PSST training in teacher education programmes.
Other concrete ways for UNICEF BCO to increase the synergy between the two programmes include: 1) to physically position an education officer in the Child Protection Programme and vice versa, with specific mandated responsibilities to identify and support cross-programme synergies; 2) to hold regular cross-programme consultations; and 3) to develop a deliberate cross-programme strategy that is built into the workplans for both sections and monitored and evaluated regularly.
If UNICEF BCO is truly interested in inter-programme synergies, it will need to re-examine existing systems and the extent to which they encourage or discourage such behaviours by individuals. This will require mechanisms (e.g., regular monitoring and reporting on cross-programme synergies) and incentives (e.g., recognition in annual performance appraisal processes and public recognition among UNICEF BCO staff) to support and encourage an integrated approach. Otherwise, despite best intentions, synergy will remain an afterthought.
UNICEF BCO would benefit from further study of this matter, with particular attention to institutional barriers and constraints.
In future, UNICEF BCO programme designs should encompass articulated strategies to support sustainable institutional development. These should include results that are clearly defined in terms of institutional development, appropriately defined exit strategies, identification of risks, and mitigation strategies to achieve sustainable results.
The evaluation found varying performance within and between the Education and Child Protection Programmes in terms of the realization of sustainable results, particularly at the institutional and policy levels in the Child Protection Programme. While these limitations are not unique to UNICEF BCO, they have had a negative effect on its performance.
Sustainability should be considered in the design of new programmes (and individual projects or investments) and monitored throughout the life of the programme. This will require UNICEF to define what sustainability will look like in institutional terms, ensure that there is strong likelihood that certain necessary requirements (such as ownership) are met, ensure that results are defined in terms of institutions (rather than only or mainly individuals), ensure that programmes include capacity development strategies to support institutional development as well as strategies to guide and determine exit. It will also require section heads to monitor and report regularly on progress and challenges encountered and to identify mitigating strategies to support sustainability of results. Finally, as part of its annual programme performance reporting, BCO programmes should report on results sustainability.
In conjunction with UNICEF HQ, the BCO should clarify, develop, and foster a common understanding, approach, and strategy to capacity development that fosters sustainable institutional development. Future capacity development support should consider how such initiatives will benefit institutions, and where longer term benefits can be realized.
UNICEF BCO should clarify the meaning and practical implications of convergence for its programmes in Bangladesh.
UNICEF BCO is rich with new ideas and strategies for enhancing its programming effectiveness. More recently, this has included a renewed emphasis on convergence. However, as noted in the evaluation, this concept is not yet fully or commonly understood by staff, leaving the possibility for mixed approaches and mixed effectiveness in implementation (as has been the case with capacity development in UNICEF). UNICEF BCO is therefore encouraged to continue to clarify this concept and strategies with staff and pay particular attention to how it is implemented.
UNICEF BCO should adapt existing results planning systems so that they provide a clear and formalized basis to assess a programme’s performance over time, and should enhance and adapt existing monitoring and reporting systems to capture regular (annual) information on the cumulative performance of its programmes at both output and outcome levels.
It is commendable that UNICEF BCO has initiated measures to adapt its results framework over time to respond to contextual realities. However, it needs to have an effective system in place to formalize these changes, and ensure that they are used as the basis for future results tracking and reporting.
BCO currently tracks and reports on outputs on an annual basis, but not on a cumulative basis. The absence of such information makes it difficult to know how programmes are performing at the output level over time, which in turn inhibits trouble shooting at the programme level. The establishment and timely use of such a system could help BCO managers flag and resolve problems in timely ways. It would also provide a very valuable basis for programme evaluations, which could draw on this information as an important basis for review and validation.
Lessons Learned (Optional):
Articulating a theory of change for a programme helps all stakeholders understand the logic of the entire programme and how each of their efforts are intended to contribute to the higher level results. It can also help programme managers modify interventions as they evolve.
Organizational change requires strong leadership, commitment, as well as the necessary incentives, support mechanisms, and checks and balances.
The results of a capacity development initiative are more likely to be sustained if the initiative is mainstreamed within the targeted entity, if it supports one-time rather than ongoing costs, if exit strategies are defined and used, if sustainability issues are identified and addressed in timely ways, and if individual capacities are supported in tandem with institutional capacity development rather than in isolation.
An inception phase, if planned and managed appropriately, can enhance the potential utility of an evaluation.
PDF files require Acrobat Reader.