Evaluation database

Evaluation report

2012 Tanzania: Evaluation of Government of Tanzania/UNICEF 7 Learning Districts Strategy (2007-2011)



Author: Jimat Consult and Ifakara Health Institute

Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."

Background:

UNICEF has been supporting development cooperation interventions at sub-national level in Tanzania for over 15 years. From 2007-2010, UNICEF implemented a country program (CP) that had a dual purpose of influencing national policies, regulations and standards upstream while at the same time sharpening the testing of promising child-sensitive programming models and intensifying investment in a reduced number of “learning districts” (LDs) (7 in total) to improve service delivery.

Within the 7 LDs (Bagamoyo, Hai, Magu, Makete, Mtwara, Siha, Temeke), UNICEF adopted a comprehensive package designed to achieve results in just under 30 programme components spread over four thematic areas: Young Child Survival and Development (YCSD), Child Protection and Participation (CPP), Basic Education and Life-skills (BELLS) and Policy Advocacy and Planning (PAAP). [...more in report]

Purpose/Objective:

The overall objective of the evaluation was ‘to assess effectiveness of the area-based programming approach, the “theoretical model” of the 7LDs, and draw lessons learned for future programming internally and in other countries’.

The Evaluation focused on answering a set of questions under each of the following OECD Evaluation Criteria: Relevance; Effectiveness; Efficiency; Sustainability; Impact and the extent to which Cross-cutting issues (such as prioritization of the most vulnerable, enhancement of gender equity, extent to which district planning and budgeting are results based, and extent to which community participation was promoted) were addressed.

To conduct the evaluation, two institutions (Jimat Development Consultants and Ifakara Health Institute (IHI)) were competitively selected and twinned to provide the services with the former focusing on the national and district stakeholder interviews and Drafting the Main Evaluation Report, while the latter were tasked with the administration of sub-district data collection and drafting of the End-line Survey report. This report presents the results of the evaluation jointly carried out by these two institutions based on data collection at national, regional, district and community level over a 4-month period. It has 11 Chapters, the first two providing background information and the methodology. Chapter 3 presents the status of facilities and services in the Learning Districts and 6 comparison districts selected as controls at the time of the Baseline Survey in 2009. Chapters 4, 5, 6, 7, 8, 9, present findings in relation to relevance, effectiveness, efficiency, impact, sustainability and treatment of cross-cutting issues, respectively. Chapter 10 presents the lessons learnt while the last Chapter is on Conclusions and recommendations.

Methodology:

Taking into account the completion status of each of the key result areas funded by UNICEF, a quantitative End-line sample survey was designed and carried out targeting households, and relevant stakeholders at institutions, facilities and within communities who were reached by the UNICEF/GoT interventions in the 7 LDs. A matching sample survey was carried out in 6 NLDs for comparison. The learning districts included: Temeke, Bagamoyo, Hai/Siha, Magu, Makete and Mtwara rural. The non-learning districts were Kinondoni, Mkuranga, Moshi rural, Misungwi, Njombe and Tandahimba. These districts had more or less similar characteristics with the learning districts.

A “difference in difference statistical test” was used to ascertain whether changes in LDs were significantly higher than in NLDs. For purposes of this executive summary we present results by key result area, then the conclusions, main lessons and main recommendations. 

Findings and Conclusions:

Overall the 7 LDs interventions were found to be highly relevant and had positive results. The results achieved varied by key result area and below are some of the highlights on major changes that have occurred in Learning Districts versus non-learning districts by programme component. [...more in report]

The Evaluation confirms that the 7 LDs Strategy was not only highly relevant but also produced many positive results. It has built capacities of Learning Districts to plan and budget for children’s priorities and the capacity will be retained in the government system. The main challenge reducing the actual amount of financial resources made available by districts to children’s priorities is now the absence of an explicit budget line for social welfare under which children’s priorities, especially of child protection nature, can have a dedicated vote and cost centre from which they can be funded. While nutrition is now a permanent budget line in the MTEF as it is budgeted for first in the Comprehensive Council Health Plan, and then reflected in the MTEF with funding from the health basket fund, child protection and socio-economic support for MVCs do not as yet have a budget line under which they can be funded. The District Social Welfare Officer activities are funded either from a small resource envelop under the mandate of the “Community Development Officer” (and this is not enough nor the proper pipeline to finance such interventions) or by way of small “donations” from sectoral budgets as part of the multi-sectoral HIV and AIDS response. The latter practice was not found in LDs but among NLDs, Moshi Rural, which was innovative.

UNICEF has through the LDs experience piloted a number of successful models that need to be well studied, documented and then scaled –up. Hai has good experience on how WDSP has increased pass rates in primary schools, it also has promising experience with the reduction of the number of children’s homes that did not provide proper care for children. Makete is an example of good practice on male involvement in PMTCT, in out-scaling the Whole School Development Plan approach, in financially supporting the activities of the ward education coordinators and in preparing high quality MTEFs. Makete also offers good experience in water treatment and other hygiene behaviours. Magu, Makete, and Temeke offer good models for institutionalising WASH through village by-laws and therefore promoting sustainability of WASH promotion activities. UNICEF missed an opportunity to promote replication of some of these best practices in NLDs by not effectively engaging Regional Administrative Secretariats to achieve this objective.

It is also very clear from the evidence that some of UNICEF-supported interventions lacked impact on some very critical indicators contributing to the well-being of children, and these included (a) access to improved sanitation, (b) hand washing at four critical moments, and (c) prevalence of diarrhoea in under-fives. Though access to sanitation is inherently a slow change indicator, these findings suggest that UNICEF may need to revisit its strategies under the 7 LDs Strategy as it carries on with WASH in the new Country Programme phase; this includes revising both approaches for community WASH and for BCC. Both the BCC component implemented through a PCA agreement with PSI and the Community WASH approach implemented through districts need to be carefully analysed for effectiveness and value for money, and redesigned as necessary.

In some cases, the Evaluation reveals that results achieved by UNICEF could not be analysed sufficiently because the M&E system did not track them. UNICEF’s M&E system for the 7 LDs Strategy had a number of gaps that need to be addressed, going forwards into the next phase of sub-national engagement.

We further conclude that by spreading too thinly, through coverage of several districts with a wide spatial distribution, UNICEF could not sustain the planned quarterly joint monitoring visits, as some sections with a shortage of staff could not be represented consistently by their own staff member. UNICEF may also have spread its resources too thinly in two other ways, (a) by working both at the national and sub-national level, and (b) by working in more than 30 key result areas, 26 of which were primarily district-oriented. Participation of senior members of staff was needed both at national and sub-national level but proximity to the national level, and frequency of policy dialogues and meetings at the centre limited UNICEF’s engagement of districts using its most senior staff.

Recommendations:

1. UNICEF’s new strategy for sub-national engagement should be elaborated at the onset of such engagement, with a clear set of objectives, targets and entry and exit criteria for each major intervention area.

2. UNICEF together with PMO-RALG should develop explicit entry and exit criteria for sub-national engagement which are based on development results (e.g., child poverty, child vulnerability and critical MDG indicators) in order to improve transparency and objectivity of regional targeting, progress assessment and decisions on phasing out of UNICEF support.

3. Exiting a district or region should not be on the basis of expiry of a Country Programme phase, but rather on exit criteria that are linked to the achievement of results in the form of either output or outcome targets, development impact or learning outcomes. All 7 LDs strongly encouraged UNICEF to consider as exit criteria, “progress made towards achievement of MDGs” and the likelihood that results achieved will be sustainable.

4. Within the selected focus regions, and for interventions that cannot achieve full district coverage, UNICEF should assist districts in defining explicit criteria for selecting first priority communities (wards, villages) in order to guide the channelling of the limited resources to the most vulnerable and deprived children.

5. The number of key result areas pursued by UNICEF (thematic scope) and the geographic spread both in terms of the number and spatial distribution of the districts should be informed by child deprivation and/or poverty indicators and matched with the human and financial resources available within UNICEF to provide for adequate district and thematic level engagement by UNICEF Senior Staff for purposes of (i) more adequately engaging and (where possible) jointly programming with other actors supporting districts/sectors for similar results, (ii) deepening coverage and convergence of UNICEF supported interventions at facility and community levels, (iii) strengthening supportive supervision and quality assurance, and (iv) ensuring systematic evidence generation and learning, effectively feeding into regional replication and national policy influencing discourses.

6. UNICEF should in the future deepen its engagement with development partners with active programmes at the district level. This should be done by senior staff within UNICEF and balancing it with national level policy engagement. This may require strengthening PMO-RALG’s coordination capacity and role to ensure better coordination, joint programming and experience sharing.

7. Capacity building support should remain one of the priority strategies for strengthening the protective environment for children, planning and budgeting for children’s priorities and improving the quality of basic services that reduce child poverty and vulnerability and improve the well-being of children. Creation of new structures should be avoided as much as possible, but rather more investments are needed to re-engineer the current structures to provide missing but priority services for children, imparting critical knowledge and skills into the current structures, providing them with working tools and resources, strengthening linkages between them through an explicit strategy for promoting programme convergence, and developing and testing models for their financial sustainability. This principle should cut across all key result areas tackled by UNICEF’s sub-national engagement.

8. UNICEF should reinforce programme integration within and across its programme components through an explicitly written strategy for maximizing programme convergence and synergy. Priority should be given to mainstreaming of child protection and participation in the entire country programme portfolio, including by packaging CPP into training packages and BCC messages delivered by the various programmes of UNICEF. Furthermore, UNICEF should identify and make use of a number of opportunities that exist for strengthening convergence of UNICEF supported interventions at health worker, CoRPS, WASH artisan, school teacher, Tuseme Club, MVC Committee, and Multi-sectoral HIV and AIDS Coordination Committee levels. At the minimum UNICEF should explore the use of harmonized training packages and multi-skilling of facility and community based service providers to ensure that UNICEF supported services converge at the facility, community and household levels.

9. Through greater programme integration, multiple channels for addressing sanitation and hygiene promotion need to be found to address diarrhoea disease incidence in under-fives, which is one of the critical indicators on the quality of life of children which was not impacted upon by the 7 LDs Strategy.

10. In the specific area of child protection, both family- and community-centred approaches to address the underlying causes and prevent the perpetration of violence and physical, sexual, emotional and economic abuse against children and their caregivers should be given more attention than previously, alongside the on-going strengthening of current initiatives to respond to and mitigate impacts of the abuse. The latter should also be broadened to ensure that the child protection model currently being tested in pilot districts addresses all needs of children in relation to child protection services.

11. In relation to training, specifically, UNICEF should invest more into institutionalising the on-going trainings by strengthening the curricula and delivery capacities of existing competent national or regional institutions to play a greater role in providing those courses that need to be delivered at scale and should reach new entrants (e.g., new generation of health service providers, teachers, planners and budget officers) whilst they are still being trained for entry into the service.

12. Due to the high level of unmet need among those already in service delivery, in terms of new skills necessary for child-friendly schooling, delivery of more effective child protection services and higher quality health services the Government of Tanzania and UNICEF should not abandon or suspend in-service training of service providers, but explore new models for delivering the trainings in a manner that does not remove trainees for long periods of time from the core business of service delivery. Alternatives to off-station/residential courses (such as innovative approaches for providing on-the-the job coaching and mentoring) should thus be identified and tested to establish their comparative advantages vis-à-vis current approaches.

13. Whilst the training of trainer cascading approach works and is important for creating capacity at both national and district levels for continuation of training, resource allocation between courses offered to ToTs, and those offered to service providers at facility and community levels should be reconfigured, by exploring less resource intensive methods of providing ToT courses, and availing adequate resources for training of frontline service providers (e.g., teachers, health service providers, CoRPS, VMACs, WMACs, village MVCCs, ward MVCCs, sub-district CPTs) and, in where necessary (e.g., in the case of school WASH), final beneficiaries (school pupils).

14. UNICEF should strengthen its M&E System for tracking results of sub-national engagement and provide adequate support to intervention districts especially for tracking results from community based services, and identifying, documenting and sharing learning. Planning for key outcome results and output targets with the learning districts/regions should ideally follow a four year time horizon harmonised with the UNICEF’s Country Programme Action Plan in order to have synchrony in objectives and targets (but operationalized through annual plans for financial management). The M&E System should facilitate aggregation of results from all LDs to more fully see the contribution of UNICEF and inform exit in a transparent manner.

15. UNICEF should prioritize investments in upgrading M&E capacities of intervention districts for all interventions, but with special focus being given to enabling the LGAs and UNICEF (with support from the Regional Advisory Secretariats) to track critical core indicators of performance at output and intermediate outcome levels for all interventions including those of a software nature and for innovations that will be piloted at health facility, school or community level. Such an M&E system should build on (and make use of) existing national M&E systems, with the ultimate objective of further enriching the national systems through institutionalisation of the new elements of the M&E. Hence it is recommended that UNICEF works closely with relevant sector ministries (MoHSW, MoEVT, MoW, and MoG) and agencies such as the national statistical office, RITA, sector ministries, PMO-RALG and the Planning Commission to develop appropriate M&E tools for monitoring intermediate outcomes at community level.

16. Monitoring of the work of community volunteers should be strengthened (e.g., the 3 levels of supervision for health services, for example, – national, regional and district health management teams - should monitor the work done by trained CoRPS.

17. PMO-RALG should complement UNICEF efforts at district level through: (1) working with relevant sector ministries to put in place a well-functioning system for dissemination of national laws, policies, strategies, guidelines, and standards; (2) influencing GoT and DPs to increase resource allocation to basic education – school per capita grants, teacher housing, text books and stationery; (3) creating a sub-vote and cost centre for social welfare and child protection services under the mandate of the DSWO; (4) allocating resources to sustain motivation and activities of WECs, CoRPS and Community Justice Facilitators; and (5) transforming the DMETs into more autonomous teams with a separate budget line, providing technical oversight and a mechanism to hold districts to account for performance against budget targets. The excellent work done in registering MVC should be complemented by Ministry of Health and Social Welfare in mapping service providers and strengthening coordination of their support to MVC, together with mobilising more resources from the development partners and Treasury to provide resources to reach these registered children, alongside support for economic strengthening of their families. A child and family centred approach in addressing child poverty and vulnerability is needed for holistic care in future and UNICEF should in collaboration with others champion policy advocacy in this direction.

18. UNICEF Tanzania Country Programme needs to strengthen gender mainstreaming through an explicit gender mainstreaming strategy which is linked to both corporate policy and the National Strategy for Gender Development, to inform choice of interventions and whose implementation and achievements are also systematically tracked through the UNICEF and LGA M&E systems.

19. UNICEF, PMO-RALG, sector ministries and LGAs should document all models of good practice developed in the 7 LDs, test them in new districts for suitability, and then promote replication in these new districts2. Expertise required for this documenting good practices would include thematic specialization, communications, and journalism. The central role Regional Advisory Secretariats can play in promoting replication should be recognised and formally mobilised and supported technically and financially by UNICEF and PMO-RALG.

20. UNICEF and PMO-RALG should together with LGAs take stock of the completion status of all interventions in the previous LDs and come up with a strategy, time-line and resourcing mechanism for completing all unfinished business. In cases where a decision has already been made by UNICEF and PMO-RALG to discontinue some of the support, in order to concentrate on new regions, PMO-RALG should assist LGAs to find alternative funding for completion of outstanding work. In the same vein, the Evaluation found special need to continue capacitating Siha District with almost the full package of support, as this district is relatively new and still lagged behind all other LDs on some critical capacity indicators and MDGs. Phasing out support in Siha District would be pre-mature at this stage. Opportunities to continue capacitating Siha District should be found by PMO-RALG.

21. Given the gap between Strategic Plans and MTEFs in incorporating children’s issues, with children’s issues featuring more in MTEFs than SPs, UNICEF should take advantage of the present opportunity provided by the new round of strategic planning to lobby PMO-RALG to support LGAs in developing new strategic plans that incorporate children’s issues as part of exit strategy.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search