Evaluation database

Evaluation report

2011 Tanzania: Strengthening National Disaster Preparedness and Response Capacity -End of Programme Evaluation



Author: Martin Fisher, Dr Ngila Mwase

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Purpose/Objective:

The purpose of this evaluation is to evaluate the achievement of results (outputs and outcomes of Joint Programme 6.2 since its commencement in 2007. The evaluation will cover both Tanzania mainland and Zanzibar. The target audience of the Evaluation will include the UNCT and the Government of Tanzania. Secondary audiences will include Implementing Partners and Development Partners.

Methodology:

Evaluation Methodology
2.1 Introduction
The Joint Programme 6.2 aimed to closely address the current and expected disaster profile of Tanzania with a focus at the level at which disasters occur – at district level and below, amongst communities and families – with strong co-ordination and facilitation provided by institutions located at the national and the sub-national levels. The JP intended to ensure that disaster risk management was seen as essentially a multi-sector and multi-stakeholder activity, cutting across departmental boundaries where all staff understand their roles and responsibilities, and to build on their complementarities.

As such, this work was mostly working at an institutional and policy-making level, with very limited interaction with the traditional ‘beneficiary’ focus at community level – indeed the question of “who is the target group mentioned in the ToRs?” was discussed in the inception meeting.

The following sub-sections explore the methodologies used during this study in more detail.

2.2 Evaluation WorkPlan
The work was designed to fall into three phases over a four-week period, namely the inception phase, the data-gathering and analysis phase and then the writing and presentation phase. A work plan (Annex Five) was prepared at the outset as a planning tool for arranging the meetings and travel plans. This was used as the guideline during the inception phase, but was presented and amended slightly after discussion with partners at the Inception meeting, principally due to the time limitations (described below under point 2.8).

Phase One : Inception
The initial phase of the work incorporated a review of provided programme documentation (hard and soft copies) and the annual work plans and reports. Initial meetings with the focal points (FPs) in the PUNs and Government Ministries to explain the process and gauge preliminary feedback on expectations were held, and to request other relevant documentation. Questionnaire formats for each of the key informant groups, to help build a comprehensive and validated picture during later meetings, were developed, along with a detailed evaluation matrix.

The Inception Report was presented to the PUNs, the GoT and other partners on Monday 20 June 2011, for approval and acceptance.

Phase Two : Data Gathering
The data-gathering phase commenced with a short visit to Zanzibar, where the key UN Agency and Ministry informants were interviewed. In addition, several visits were made to facilities that had been assisted via the JP 6.2, including the emergency warehouse in Zanzibar town, and to IPs’ offices and activity venues.

Further meetings were held in Dar es Salaam with IPs and UN Agencies. A subsequent visit was undertaken to Dodoma to meet with key Government officials, visit the emergency warehouse and to meet with several district and regional level officials.

Phase Three : Data Analysis and Report Preparation
This phase included follow-up interviews, analysis of the data and information gathered, addressing outstanding questions, the preparation of the draft report and its verification and validation. Preliminary findings were presented for discussion at a final meeting with the partners held in Dar es Salaam on Thursday 30 June. The draft report was submitted to the Evaluation Supervisor on Saturday 09 July 2011. Feedback from the various partners was encouraged and requested by Thursday 14 July, with its integration as appropriate in the following days.

This Final Report was submitted to the Evaluation Supervisor on Monday 18 July 2011.

2.3 Review of Documents
The consultants received a number of the historical programme work plans and reports and other documents at the outset, provided by the Evaluation Supervisor, with additional documentation being requested from all programme partners if available. The main documents consulted were: the original programme outline document, the annual JP 6.2 work plans, annual and mid-term reports, minutes of quarterly coordination meetings, certain email communications, workshop minutes and programme reports, the National Disaster Management Policy and the National Operational Guidelines for Disaster Management. Other associated documents, such as the Standard Operating Procedures of several IPs developed as part of JP 6.2, were also provided, and others such as the food security assessment reports in Zanzibar were consulted in situ. Additional documentation regarding the ‘DaO’ agenda, the UNDAP planning outline and other related paperwork was also made available.

A bibliography of the main documents consulted is given as Annex Three. No previous review of JP 6.2 had been undertaken.

2.4 Development of Data Collection Tools
During the Inception Phase, an evaluation matrix was developed to address the primary questions being asked by the Terms of Reference, identifying certain indicators and sources of data collection, ensuring the evaluation team was prepared to gather sufficient and appropriate information to adequately answer the questions asked.

In addition, a series of questionnaire guides was developed (Annex Two) to structure the interview processes with the key informants. The evaluation matrix and the question formats were presented to, and discussed at, the Inception Meeting and were included in the Inception Report.

2.5 Interviews with Key Informants
The Evaluation Supervisor, on behalf of the evaluation team, set up initial meetings with focal points of all Implementing and UN partners, during the first week. Follow-up meetings, where considered necessary, were confirmed directly at the time of presentation of the Inception Report or subsequently by telephone.

Separate interviews were held with all key informants, who were advised that any comments made would not be attributed. Several individuals who had worked on the JP in the earlier phases were contacted by telephone or email.

2.6 Meetings with other Stakeholders
This JP 6.2 was mainly focused at the level of national and sub-national Administration officials and staff of UN Agencies, and was not a ‘normal’ programme intervention working at community level or with direct beneficiaries. However, as several of the interventions, notably in Zanzibar, had been made through community groups the evaluation did team conduct four focus group discussions with representatives of separate communities at Shehia level. These meetings, arranged by the IPs, included community members with a wide age range (teenagers, teachers, older people etc) and were gender balanced.

The evaluation team requested to meet with one or more of the key donors to the ‘Delivering as One’ process in Tanzania. The meeting held was with the technical representative of the current Chair of the ‘Friends of the One UN’ donor forum, and as such presented the general views of the forum’s membership rather than individual donors.

Due to time constraints and the long distances involved the only field visit made on the mainland was to Dodoma, where one evaluator was able to discuss the programme with the Disaster Management Teams (Regional and District levels), including the Regional Focal Point for Disasters, and to get information on capacity building at sub-national levels.

2.7 Field Visits to Activity Sites
Several visits were made to sites where programme inputs had been made. In Zanzibar, the evaluation team visited the emergency warehouse stocking emergency relief supplies provided as part of the programme. Further visits were made on Zanzibar to other facilities that had been assisted under JP 6.2 – notably these were the seed production facility of the Ministry of Agriculture & Natural Resources (provided with laboratory materials), and the Plant Protection Services Pathology Laboratory (rehabilitation of buildings and provision of laboratory materials). During the short visit to Dodoma, the evaluator also visited the zonal warehouse.

2.8 Limitations of the Evaluation Methodology
The evaluation work was originally designed to cover 42 days, but due to a delayed recruitment process and start date, and the proximity to the end of the programme period, this was shortened to 30 days in total. The consultants were constrained, therefore, concerning time to travel to sub-national level (except in Zanzibar and briefly to Dodoma) and direct informant feedback at these lower levels was unfortunately not widely gathered. Some Government officials were also not available due to official commitments outside Dar es Salaam during the evaluation period.

In a multi-year programme such as this, it is inevitable that staff of Government departments and UN Agencies move on during the period, and the majority of the interviewees for this study had become involved in the Programme some time after its commencement, thus first-hand knowledge and experience of the planning and early stages of the intervention was difficult to obtain. Several of the Ministry meetings included former staff that were brought back in for the sessions with the evaluation team, which was greatly appreciated.

A wide range of documentation was also studied as part of this study. The evaluation team found the use of the programme document formats and terminologies to be very inconsistent from document to document with, for example, the various annual work plans being presented in widely different ways. This is discussed in more detail on pages 26 & 27.

Findings and Conclusions:

Evaluation Findings
Introduction
The World Conference on Disaster Reduction, held in Hyogo, Japan, in January 2005 (at which Tanzania participated) adopted five priorities for action (subsequently referred to as the Hyogo Framework for Action ). These priorities were developed from an acknowledgment that the most appropriate way to reduce disaster risk was to integrate the various efforts “systematically into policies, plans and programmes for sustainable development and poverty reduction”, supported at all levels and by all partners. It went on to state that “… accelerated efforts must be made to build the necessary capacities at the community and national levels to manage and reduce risk” … which in turn would contribute towards the achievement of the Millennium Development Goals (MDGs) and other development objectives.

The five areas identified were the following:
• Governance: organizational, legal and policy frameworks
• Risk identification, assessment, monitoring and early warning
• Knowledge management and education
• Reducing underlying risk factors
• Preparedness for effective response and recovery.

As explained in the original JP 6.2 project document, Tanzania, like many countries in Africa and elsewhere, traditionally responded to emergencies as they occurred - and then did not really address the need again until some further situation presented itself . More recently attitudes have changed. Given a greater recurrence of disasters, not unconnected to climate change, and given man-made hazards (terrorism, piracy, accidents etc) the issue of emergency preparedness and response has gained greater importance.

The Government established the Disaster Management Department (DMD) in 1990, to ensure the coordination of disaster management activities in the country. In Zanzibar, the DMD was only established in 2006. The Departments now seek to ensure that in times of disaster appropriate response systems, procedures and resources are in place to assist those affected and to support them to help themselves. The Department is also charged with the responsibility of coordinating disaster preparedness efforts and activities in order to minimize the adverse effects of hazards, through effective precautionary measures and to ensure timely, appropriate and efficient organization and delivery of emergency response.

With these structural changes have come related amendments in Government policies and strategies. Disaster risk reduction identified in the MKUKUTA (I & II) and MKUZA (I & II) strategies to address the challenge of disaster, its mitigation and the transition from humanitarian assistance to development. The GoT identified weaknesses in these areas of emergency management and highlighted the need to improve on certain aspects under their ‘Cluster 3’ sections, covering good governance and the rule of law.

In particular, MKUKUTA II outlines (chapter 4.3.1.3) that targeted growth should be strong enough to withstand threats, including in particular shocks due to natural or man-made disasters. Direct reference is made to the National Disaster Management Policy (2004) and the National Operational Guidelines for Disaster Management (NOG, 2003), (both of which are under review). Man-made and natural disasters and their impact on infrastructure and the national economy were recognized as posing substantial risks to economic growth and poverty reduction, while growth and development could in themselves create the circumstances for increased risk.

The current Zanzibar Strategy for Growth and Reduction of Poverty (MKUZA II) 2010 – 2015 includes the following disaster risk reduction intervention measures: (i) review and harmonize disaster related laws and policies; (ii) improve infrastructure and capacity to deal with disasters and strengthen emergency preparedness system; (iii) build capacity of institutions and individuals in terms of equipment and necessary skills and knowledge; (iv) empower community members, especially women and children, to prevent and respond to hazards and disasters. All these are encompassed under Operational Target 3.2.7: “Preparedness and response to disasters enhanced by 2015”.

Key Outcome & Outputs of Joint Programme 6.2
The principal outcome expected of the JP 6.2 was enhanced institutional, organizational and individual capacities of the Government of the United Republic of Tanzania (GoT), at different levels, to be able to respond to natural disasters, epidemics and civil unrest. This was working towards the outcome identified for the overall UN Country Programme which was to work towards supporting enhanced Government capacity for disaster preparedness, response to refugee flows and management of the transition from humanitarian assistance to development.

However, it was clear from feedback from several key informants and from various reports studied that during the first year of implementation the overall logic of the JP 6.2 was not obvious between the wide range of interventions listed and the outcomes expected, and although some activities had been underway during the first year, some significant revision work was later done, producing a more coherent work plan for the second year of the programme. It was also mentioned at this early stage that the original plans were considered to include “over-ambitious expectations … and a lack of clarity and vertical & horizontal linkage in the outcomes as presented in the work plan”.

The evaluation team confirms the lack of clarity and considers the original list of activities in AWP for Y1 to be a collection of rather disparate disaster related projects rather than a number of programme areas with obvious coherence, and it is inevitable that the implementing partners and UN Agency staff found them to be difficult to follow through with. The AWP for Year 2 (July 2008 – June 2009) clarified the detailed outputs expected and identified the activities to support them. Some activities had been implemented during the previous 12 months, and new ones came about later in the programme cycle. The evaluation team therefore took this Y2 matrix to be the most developed planning document for the JP 6.2, and this was used as another framework for the evaluation work. [More details of this point around documentation are given below in the section entitled “Planning and Planning Cycles”.]

Significant achievements of the JP 6.2 have been identified in a number of areas despite some of the early problems.

Disaster Risk Assessment
The Disaster Management Training Centre (DMTC) of Ardhi University in Dar es Salaam, a long-term partner of national disaster risk management authorities and development agencies, carried out an initial assessment study in late 2007 supported by JP 6.2. The process was carried out down to district and community levels to develop disaggregated disaster vulnerability profiles with the aim of identifying strategic priority areas for implementation, based on the different elements of national policies.

The risk assessments identified:
• the disaster risk profiles for the two areas of Tanzania Mainland and Zanzibar (suggesting hazards that affect or may affect the country, their frequency, magnitude and geographical spread, as well as potential vulnerabilities of populations, economies and environments);
• the current functional and technical capacity for responding to disasters and vulnerabilities, including existing policies and legislation;
• the actual capacity needs to respond to disasters, including revision of policies and legislation, contingency plans, information management, human resources, facilities and equipment, communication and early warning systems;
• the degree to which these hazards can be managed, and the coping mechanisms of affected communities.

The resultant report was intended to be used for revisions of the first year plans, but was so delayed in its presentation it only became useful for the Year 2 planning exercise. This delivery was allowed to drag for many months and the consultants should have been chased to deliver sooner than they did. However, that said, the fact that the second annual work plan was significantly improved from its predecessor was undoubtedly based on the important preliminary work done via this study, and the report itself, with its insights into available in-country capacity and critical needs, constitutes a key achievement of the JP 6.2.

Other assessment work was also done in the early part of the programme, looking at specific technical areas of the interventions. For example, for animal and plant health issues, an assessment was undertaken to study the capacity to detect, identify and monitor animal diseases and current information flows, based on which diseases were prioritized and high-risk areas identified. This preliminary work was vital for the direct interventions that followed.

The evaluators feel that the interventions made under JP 6.2 identified and addressed a number of pending disaster-related activities that IPs had already identified as needing completion. However, there still needs to be a much more robust engagement of disaster risk reduction thinking across all development interventions being done in Tanzania. It is insufficient to assume risks can be addressed by a limited series of interventions by a few partners – to make any significant difference the debate has to proactively engage with all partners and all supporting Agencies, and the identification of risk features and the opportunities to work towards reducing them has to be factored into all developmental work. There is much already written about this inter-relationship, including by a number of UN Agencies, but it was not clear to the evaluators of JP 6.2 that such thinking was evident in Tanzania beyond the key Ministries and institutions directly involved in disaster management. The UN in Tanzania, though, through the new UNDAP process and its commitment to ‘Deliver as One’, is in an ideal position to advocate for and contribute to a much more integrated approach.

Disaster Preparedness and Early Warning Mechanisms
This programmatic area encompassed a number of principal work components focusing on general policy development and management capacity, with a number of health and food security related issues also included.
2.1 General disaster risk management, preparedness and early warning
Inputs in this area concentrated on addressing some of the identified areas of risk and developing measures to mitigate against them. Support to structural inputs to the two DMDs and work on policy guidelines was critical, as well as working at lower levels to help establish and train regional and district disaster management committees in some areas. Significant progress was made in Zanzibar, with over 2,250 people benefitting from such awareness and training sessions; progress was also made on the Mainland but more towards the lower level structures such as the local disaster management committees and other local administration bodies.

It was reported that the split between coordination of, and direct involvement with, disaster management activities was fairly blurred, often with the PMO/DMD getting directly involved in field operations. The Standard Operating Procedures (SOPs) indicate that their role is to support when lower level capacity requests additional input and to coordinate the overall response approach – several discussions around the threshold or trigger points for DMD’s involvement indicated that some thinking has gone into the matter but no specific points have been set. It would be useful to work on clarifying this with specific indicators for future engagements. Also, the coordinating units should resist the temptation to double-up as direct implementers, which could compromise their coordination function.

The evaluators were left with a feeling that the focus of the programme was still largely on disaster response more than preparedness and risk reduction. However, it is acknowledged that the Zanzibar Disaster Management Department was only relatively recently established, and they have had to take on all disaster-related activities. This has so far focused on strengthening the response capacity as a priority. Although the work is moving in the right direction and the required focus on preparedness and risk reduction is not being ignored, more effort needs to be put into some of these specific activities in the future.

2.2 Food Security
In an effort to improve the overall preparedness and response capacity to shocks that affect household level food security, especially of the most vulnerable and disaster-prone households, WFP and FAO, under the coordination of the MAFSC, jointly conducted a Food Security Assessment training to strengthen the capacity of the Government and other partners in food security disaster preparedness. Some 31 participants (16 from Mainland and 15 from Pemba Island; 15 female) followed the two-week training course.

Subsequent feedback and analysis of the work they have since done has shown that the quality of their later assessment work had improved significantly – there was better analysis, broader checking and verification of the data and information gathered, thereby producing a better quality report, and it is considered that this is a direct result of the workshop. These individuals are to become members of the district based multi-disciplinary Food Security Information Teams with the mandate to monitor food security and nutrition in their respective districts and provide input into the Rapid Vulnerability Assessment (RVA) fieldwork and analysis.

2.3 Surveillance Systems for Trans-boundary Human, Plant and Animal Emergency Diseases
A number of the programme’s inputs were in the area of establishing and strengthening a structured animal disease information system, and building on the capacity assessment undertaken at the outset, considerable training work was done. In addition, a range of IEC materials was produced, printed and distributed as part of a public information campaign on disease identification and control. Several laboratories were upgraded with equipment and materials. This work was supported technically by FAO.

For human health, WHO worked with the Ministries of Health in Tanzania Mainland and Zanzibar to strengthen the integrated disease surveillance and response (IDSR) system by completing the disease surveillance guidelines and conducting training for 25 workers at regional and district levels in Zanzibar on the new guidelines of the IDSR; and conducting training of health workers in case management and disease surveillance in relation to the Rift Valley Fever (RVF) outbreak in 2007, when outbreak investigations and ensuing reporting were conducted.

A major focus was towards diseases that could affect both human and animal health, such as Avian Human Influenza Pandemic (AHIP) and RVF, and a number of IPs and PUNs were involved in this part of the programme. The MoH and MoHSW, supported by WHO, conducted AHIP awareness training to health staff in border districts, and UNICEF took the lead in facilitating the public awareness campaigns covering the production of leaflets and posters, which were distributed to 29 districts in Mainland and all the districts in Zanzibar; and the production of TV episodes, radio programmes and promotional spots.

With the support of UNICEF, MLDF and ML&F (respectively in Mainland and Zanzibar), together with the secretariat of the National AHIP Taskforce, carried out an extensive training programme to medical and veterinary staff, media representatives, community groups and military personnel.

In Zanzibar, the evaluators looked into the livestock health awareness work done at school and community levels. While there had been some significant improvement in knowledge and behavioral change reported (an increase in the number of referrals to animal health centres following the campaigns, for example, and changed practices regarding the use and disposal of dead poultry, as reported by the focus groups), the workshops and training sessions held had been undertaken by the ML&F alone. While this messaging was a clear part of their strategy, and had obviously been successful at this level, opportunities to look wider and involve other partners with their complementary messages were missed. School curricula are already overloaded and cannot incorporate a wide range of disaster-related content, so it is considered that the opportunity of accessing the school clubs and teachers’ groups with further health messaging will be more challenging in the future. With the UN Agencies and IPs working more closely together under the ‘DaO’ initiative, the opportunity to develop and deliver more integrated messages could have been better utilized.

The evaluators also explored the area of cooperation beyond Tanzania – whether, perhaps, similar documentation and strategies worked on under JP 6.2 had already been developed elsewhere. Avian influenza and RVF, for example, are not Tanzania specific, so had any of the other countries in the region developed materials that could be adapted by Tanzania, and/or could the Tanzanian materials be shared for use by neighboring countries? This would ensure a cross-border coherence of approaches and interventions between Ministries in these countries. The only specific feedback to these questions was provided by FAO who indicated that much of their technical input to the Tanzanian side of the work under JP 6.2 had been sourced from their regional emergency office in Nairobi, and that it came with a regional overview and knowledge, and that such factors were taken into account.

Some direct feedback was also received in Dodoma when the staff dealing with disasters in the Regional Administrative Secretary’s office explained their involvement in the local design and implementation of animal health messaging, including how the training - especially on RVF - had impacted positively on hygienic improvements in butcheries in Dodoma and the surrounding areas. These have been transformed as a result of the awareness raised and are now said to be the best in Tanzania, with air conditioned meat storage rooms, hygienic procedures for cutting and preparing the meat, and with the increased involvement of women (which is rare elsewhere).

Building Response Capacity
The revision and finalization of legal and political frameworks for disaster risk management were a core part of this JP 6.2, to set the policy frameworks to guide the coordination and cooperation amongst key players at all levels concerning disaster management. The PMO/DMD on the Mainland and the 2nd VPO/DMD in Zanzibar retain the overall coordination responsibility for this work.

UNICEF collaborated with the PMO/DMD to strengthen the implementation of the National Disaster Management Policy of Tanzania at regional, district and community levels with the ultimate objective of fostering the development of District Emergency Plans and setting up early warning systems. Under the umbrella of the JP, regional disaster management focal points from the four regions of Iringa, Mbeya, Ruvuma and Rukwa, in Mainland Tanzania, were trained. Overall, 14 out of a total of 21 regions in Tanzania Mainland now have trained Regional and District Disaster Committees in place to support emergency planning processes. The PMO/DMD in conjunction with Regional/District Disaster Committees assesses and verifies the degree of disasters before assistance is made available. There is still substantial work ahead, however, in terms of completing regional level training and cascading this down to district and community levels, and in setting up an effective early warning and preparedness system to minimize the impact of hazards.

In Zanzibar the 2nd VPO/DMD coordinates disaster management initiatives and works to raise awareness on disaster preparedness and response at community level. Unlike Mainland Tanzania, the Revolutionary Government of Zanzibar did not have a Disaster Management Policy so with the support of UNDP as part of JP 6.2, a Disaster Risk Reduction (DRR) Policy for the islands was developed. The draft policy document was finalized in March 2008 and is now (still) awaiting Ministerial approval. UNDP also supported the establishment of an emergency operation centre for the DMD, plus training for the staff; and support for a sub-national DRR platform meeting and inputs towards the activities undertaken in the International DRR Day in 2010. The 2nd VPO/DMD, with the support of UNICEF, worked to manage the shehia assessment and trainings: to date eight Shehias have been covered and 23 Shehia District Management Committees have been formed. The development and implementation of emergency plans at Shehia level will be carried forward under UNDAP.

Notwithstanding certain capacity deficiencies due to staff movement or vacancies, there is general appreciation in the GoT and the UN that the two DM Departments take their overall coordination role seriously, as do counterpart Ministries. The MoHSW has an inherent strength in having a full-fledged Disaster Unit; while the MAFSC and MLDF have long track records of dealing with early warnings, especially over drought and pests respectively.

The starting point for building emergency response capacity hinges on putting in place rapid reaction capacities, and one of the main JP 6.2 interventions in this area was the rehabilitation of five warehouses and stocking them with emergency relief items. These warehouses are at Zone level, each covering a number of regions, and were selected to be in areas known to be susceptible to emergencies, and thus ready to enable a quicker and more organized response in the event of a disaster. The Dodoma one, for example, covers Dodoma, Singida and Manyara regions. Prior to this, all such supplies were mobilized and sent from Dar es Salaam when the need arose.

The support, provided by UNICEF, covered inputs to pre-position emergency supplies for approximately 55,000 people (to date). The PMO/DMD (initially with the cooperation of the Tanzania Red Cross Society), selected a number of warehouses in five zones of the country, based on an assessment of those areas at greatest risk, plus the availability of suitable warehouse facilities and logistical issues such as road access and infrastructure. Where necessary, the units were rehabilitated prior to the goods being procured and stored. Management of these warehouses is currently being done by the host regional authorities under the overall guidance of PMO/DMD, which retains the authority to order the release of supplies to disaster-stricken communities. Previously, the Tanzania Red Cross (TRCS) managed such warehouses. While the evaluators were unable to find out exactly why the TRCS had had this mandate taken away (in October 2010), it is suggested that they have considerably more experience in such activities, and in addition to their community-level structures and volunteer base, the Red Cross still offers significant skills and experience to the emergency response inputs in Tanzania.

At the time of this evaluation, two of the four Mainland units were completed and stocked with relief materials sufficient for about 25,000 people each; and a third in Zanzibar contained goods sufficient for approximately 5,000 people. Two more units (Kilimanjaro and Lindi) were still undergoing rehabilitation work, and should be completed during 2011. A sixth warehouse in Shinyanga, principally used for the storage of educational materials for the refugees in the North-West that had been relocated, also contains several hundred blankets but none of the other relief materials. This was placed in Shinyanga to cover the regions around Lake Victoria under threat of rains from the El Niño effect in 2009.

The authorities in Dodoma explained that in case of a disaster, the District Disaster Committee would inform the Regional Authorities, which would send experts (sometimes backstopped by the PMO/DMD) to undertake an assessment. Upon authorization to release any required supplies, the Disaster Committees use established social networks (such as the TRCS) to distribute them. The evaluators concluded that the whole process is fully participatory, involving various stakeholders including NGOs, including the Red Cross with its a strong network of volunteers.

The evaluator found the state of the warehouse in Dodoma to be significantly poorer than the one seen in Zanzibar. It was covered by corrugated iron sheets, which although still reasonably strong were old and in a state of decay allowing the elements to enter. However, like in Zanzibar, the warehouse was clean and organized and the materials were well stocked, but the building itself may need some additional work to make it watertight and more secure. There is also an outstanding debt of 16 million shillings to be paid to the Government Procurement Services Agency (GPSA) from which this particular warehouse has been hired. Table 2 on the next page indicates the location and current state of preparedness of these warehouses.


Table 2: Warehouses: Location and the Current State of Readiness:
UNICEF supported the rehabilitation of the buildings and the purchase and delivery of the initial stocks and the management, allocation and resupply of the goods now sits under the management of the GoT authorities. At the time of this evaluation, the local government officers where the warehouses are located manage the facilities, and the decisions on allocating the goods are taken by the central PMO/DMD.

However, the PMO/DMD and 2nd VPO/DMD still need to finalise the decisions around how replacement stocks should be ordered and paid for to maintain sufficient emergency response capabilities. UNICEF has stated that they will not be in a position to replenish the goods used. A number of possibilities are currently under discussion, but a solution to ensure the sustainability of this asset still has to be decided upon. Until it is resolved, there are two possible scenarios: firstly, that the goods may not be used as they are intended, for quick emergency response, for fear of not being able to maintain a ‘strategic stock’ - because no-one knows how the materials will be replaced; or otherwise, using up the stocks to respond to community needs but being left with an empty warehouse, again with no idea about how any resupply would work.

It has already been suggested that the regional government staff – those who would expect to draw on the emergency stocks in the five warehouses – are brought together to determine the best ways of managing and releasing the stocks, and to decide upon ways to replace the contents.

The Government currently has a National Relief Fund which can be used to respond in many ways, including the purchase of relief goods at times of emergency, but it cannot be used – like many other ‘emergency funds’ – to restock warehouses outside times of crisis. A proposal to rename it as a Disaster Management Fund could allow such flexibility. At the same time other alternatives may exist: it is increasingly possible to negotiate framework agreements for basic supplies from manufacturers / importers, who will hold agreed quantities (and quality) of required goods in their own warehouses until called for by the Authorities, which does not require upfront payment or storage costs. While this may not negate the need for some regional stocks entirely, it may provide an adequate buffer at times of urgent need and would enable ‘emergency funds’ – whether from the GoT or donors – to be used to restock immediately at these times, on the basis of paying for what had just been distributed from the existing buffer stocks. This – or a similar arrangement – would provide a greater degree of sustainability to this programme component than is currently evident.

There was also some debate on what the threshold level was for a ‘disaster’, and therefore at what point the DMDs became involved in coordination, or in releasing stocks. While on the Mainland this was clearer (district level assessments, requests for additional support beyond what was locally available etc), in Zanzibar the central DMD was working directly at community levels, and it was noticed that the limited number of goods that had been released from the warehouse had gone to individual families in need. The evaluators would suggest that this level of assistance should be left to the social agencies, and that DMD needs to define some clarity on what level of event they become involved in.

Planning and Planning Cycles
The planning of the JP 6.2 was undertaken in early 2007 when joint programmes were identified as one approach to be taken to both align the UN’s contributions towards the GoT’s priorities as already set out in MKUKUTA and MKUZA, and to advance the ‘DaO’ agenda. The JP 6.2 was designed to contribute to the expected output of the UN’s Country Plan (“Disaster risk reduction policies and disaster management capacities strengthened for the GoT’s and RGOZ’s emergency relief, rehabilitation and recovery activities”), which was in turn aligned with the goals in MKUKUTA and MKUZA.

However, the evaluation team considers that the programme brought together a loose gathering of the various emergency related projects being implemented at the time by various IPs and/or UN Agencies and the expected results were developed based on ongoing projects rather than on identified needs. This shortfall contributed one of the major bottlenecks that hampered implementation throughout the life of the JP, as the choice of partners to be involved and the design of the programme itself was governed by what was already happening, and it was made to fit the ‘DaO’ concept rather than the JP being based on a true analysis of what was actually needed.

It would appear that ‘DaO’ was being rolled out, time was short, plans had to be developed and Agencies did not want to be left out, and hence the easiest way forward was to collate ongoing interventions to form a programme. It was also reported that subsequent cooperation from the PUNs could have been more engaged in support of UNICEF, as the Managing Agent, in developing the programme.

There was no clearly identified start date for the JP 6.2 – indeed this was a point of discussion at the inception meeting for this evaluation. It ‘developed’ during 2007. The Memorandum of Understanding (MoU) between the five PUNs was drawn up in November that year, but actually only signed in March 2008. The Joint Programme Document (for JP 6.2) – referred to in the MoU – is undated but indicates a programme period from June 2007 to December 2008. Activities, however, began to be implemented during the second half of 2007 once the AWP had pulled together the various activities to be supported.

The AWP at this stage listed 18 fairly disparate outputs (in fact two of them are ± identical, so 17) and while they were all ‘disaster focused’ they were not closely inter-connected. What has subsequently developed, of inter-Ministry joint planning and complementarity, for example on the coordinated messaging and awareness raising around the pandemic threats or the border control mechanisms, was not identified at this stage, but in the evaluators’ view this has been one of the more positive outcomes of the JP 6.2.

One challenge faced was to align the annual planning cycle of the UN Agencies (previously January-December) with that of the Government (July-June). While the UN’s Country Programme as a whole had to work on this realignment (now achieved with UNDAP 2011-2015), in fact the JP 6.2 AWP was fairly close from the outset.
All partners indicated that they – or at least their Departments/Agencies – felt they had been adequately engaged in the planning process. In reality, though, many of those individuals interviewed for this evaluation had not been in their current positions at the planning stages of JP 6.2 in 2007. Staff turnover was considerable within the IPs/Ministries and PUNs, and continuity to the process suffered accordingly. At the PMO/DMD, for example, the focal points changed three times during this period. This information was expanded upon by several IPs who reported that, although their Ministry was represented throughout the process, the individuals attending the planning and implementation meetings changed frequently. There was often no handover or lateral briefings done, and thus new people had to catch up with the process as and how they could, and were unaware of what had been decided upon in previous rounds. In the evaluators’ opinion, better performance was registered where there was continuity, as in the case of the MLDF (which had the same FP throughout this period).

At three meetings held between the evaluation team and the IPs, both in the Mainland and Zanzibar, former staff members (now either retired or having changed jobs) were brought into the sessions. This indicated also that the present implementation teams still have recourse to some of the institutional knowledge and memory from the earlier periods, and this is seen as very positive.

It was acknowledged (by the JP 6.2 implementation partners) during the first year that the AWP was probably too ambitious in its plans and targets, given numerous constraints that ensued in the initial period – a point already acknowledged in the Mid-Year Report for 2008. At this same time, it became clear that the reasons and objectives for the project were still unclear to many of the focal points within the Ministries and Agencies, and there was no common understanding on the concepts of disaster preparedness and response capacity. It was also realized that there was a range of ongoing interventions between individual UN Agencies and their traditional counterpart Government Ministries, and this information needed to be explored and shared across the JP 6.2 partners more effectively.

Therefore, in May 2008, a 1½-day workshop was held for all involved stakeholders to concentrate on the overall programme logic of the ‘DaO’ process, to develop a new (and improved) AWP for the following year, and to understand better the obligations regarding monitoring and reporting. Twenty-six stakeholders were represented at the workshop. With an external chair and via the use of various problem trees and discussions, the group revisited/explored the rationale for the programme, and identified the ways forward.

It was interesting to note that the consensus gained from this exercise, from the practitioners themselves, indicated three principal points:
• that the JP 6.2 was mainly a “compilation of emergency preparedness projects … and there was little evidence of ‘DaO’ in practice”;
• that none of the underlying causes identified in the problem analysis were being fully addressed within the timeframe of the JP 6.2, and there was little or no vertical and horizontal logic to the inputs; and
• that certain activities where the UN had a comparative advantage to address some of the underlying causes were not adequately identified.

While these points were identified and discussed, the eventual outcome of the exercise was a much more comprehensive and inclusive work plan for Year 2 (October 2008 to June 2009). It reduced the 17 rather vague outputs from the previous year to just nine more focused ones (see the matrix at the end of this section), and brought the associated programme activities closer together. It identified all related activities, the range of implementing and supporting partners, the timeframes for action and a budget breakdown by partner. With this plan, on a single page, all partners could identify the breadth of the inputs and how their components to the process inter-related to others’ contributions and thus developed towards the overall outcome.

The MA indicated that they had been well aware from the early days of the need to revise the first AWP and raised the matter often with the RCO, which was slow to respond in calling the partners together due to their own capacity constraints in the early days. Although a real difference can be noticed following the replanning exercise, it would have been beneficial for the workshop to have been held somewhat earlier.

While the second AWP was more inclusive and clearer in its layout and content, the evaluators noted that there was still an inconsistent use of terminologies and formats between the first two AWPs, which also continued into subsequent work plans. Having achieved a clearer format by Year 2, it would have been beneficial to retain this format into later years. The Y2 AWP groups a number of sub-activities together under nine primary ‘Activity’ headings; the following year, ‘Activity’ was replaced by ‘Result’, and the ‘Activities’ column actually listed the sub-activities. The Y3 AWP lost much of the useful detail of the previous version; it also introduced four new ‘thematic’ headings: ‘Policy and Planning’, ‘Capacity Building’, ‘Public Awareness’ and ‘Response Capacity’ which did nothing to help clarify the programme. These titles had not been used previously or elsewhere, and yet another new way of presenting the AWP appears confusing and unnecessary. The fourth AWP (July 2010 to June 2011) was prepared in a completely new format again.

While it is clear that the sub-activities obviously would be different from year to year and would build upon what had been previously achieved, the inconsistency in presentation is not helpful.

The Zanzibar Dimension
The components of JP 6.2 focusing on Zanzibar were structured more or less as in the Mainland and addressed the same needs (capacity building, various health issues, warehouse, etc). Its management structure was the same: with the DMD at the 2nd VP’s Office as the Coordinator. However, partly because of a much smaller population, the training and awareness sessions went down to the Shehia community level.

The development of the work plans was done with the participation of both the 2nd VPO/DMD and relevant Ministries (MoH, ML&F, MANR) and the ‘One UN’ Liaison Office(s) in Zanzibar. The evaluators noted that the IPs tended to bypass the UN Offices in Zanzibar and communicate directly with their Ministry counterparts in Dar, and/or the UN Agencies in Dar es Salaam. One PUN in particular dealt with the Zanzibari Ministries directly from Dar (though the contacts were reported to be staff other than the JP 6.2 focal points). This has reduced the value addition of having these UN staff in Zanzibar; and made it harder to deal efficiently with issues raised by the IPs there.

There was a widespread feeling amongst the IPs in Zanzibar that they are getting something of a raw deal; that they are not getting a fair share of expected JP 6.2 inputs; that at best they get ‘leftovers’. This is not a view necessarily shared by the evaluators. The IPs in Zanzibar suggested that although the implementing Ministries on the Mainland are non-Union Ministries (disaster management is not a Union matter), many of the PUN Agencies (and other external partners) tended to see the Dar es Salaam IPs as the main focal points for the interventions. [UNDP, though, has focused its JP 6.2 support exclusively to Zanzibar since 2009.]

In a practical discussion about these perceptions, it was suggested that the Zanzibar IPs should get more of the inputs, and they felt that they were often ‘overlooked’ in terms of receiving what was procured. For example, specifically mentioned was a second batch of items (mainly personal protection kits (PPE kits), but also some laboratory reagents and equipment) delivered to the MoHSW and MLDF respectively. Subsequent verification with the two UN agencies and the Ministries involved clarified that these PPE kits were for use during an outbreak of disease. As there had been no such outbreak since they had received the previous stocks, these should still be available as an adequate contingency, with the possibility of quickly mobilizing additional stocks from the Central Medical Stores should an outbreak occur. However, on further checking with the IPs (MLDF, MoHSW) and the PUNs (FAO, WHO) it was found that the second batches of supplies had either not been purchased at all in one case, or had only very recently been received in the other. Laboratory equipment had been provided earlier, as requested by the IPs themselves, though not the full quantity expected due to budget constraints at the time.
Whatever the final deliveries, it is important that the Zanzibar IPs are kept informed of situations such as these to avoid frustrations building up.

It was significant to note in this regard that none of the Zanzibar IPs was invited by the MA to the inception or feedback meetings held with the evaluation team in Dar es Salaam, due to time constraints. However, the presentations made at the partner meetings were shared with the partners in Zanzibar.

Budgets and Expenditures
Funding for JP 6.2 came from two sources: directly from the PUNs (known as parallel funds) which were from the Agencies’ own resources and directed towards this programme; or from an allocation made by the UN’s ‘One Fund’, which were unearmarked funds from donors made available to the JP to supplement the parallel funding available.

A budget of USD 3.088 million was quoted in the initial work plan. The final approved budget from the ‘One Fund’ (2007 to June 2011) was USD 3,389,358 plus additional parallel funding inputs from the five UN Partners totaling USD 1,326,653. Although this evaluation was not asked to look into the financing of the JP 6.2 in detail, it must be said that the financial figures given to the evaluation team by the MA, plus those included as annexes to the three annual reports on the programme and those presented on the UN’s Multi-Donor Trust Fund website contain some significant differences. Final figures for the first six months of 2011, including the actual expenditures by the IPs , will be available only from August 2011, and it can be expected that the final accounts for the JP will be available after that. It is therefore not possible to give specific comments on cost efficiency or effectiveness in this report.

Recommendations:

Some Specific Recommendations
1. The UN, in coordination with the donor community, should continue its pressure and advocacy work at the highest levels of the Government of Tanzania – and particularly within the Ministry of Finance – to find ways to speed up and improve the processing of donor funds to the separate implementing Ministries.
Suggested implementer: The UN’s HACT/Finance Working Group, with other senior UN Agency staff and donor leads.

2. There needs to be more cross-cutting debate and planning to integrate disaster risk reduction into all developmental work rather than leaving it as a separate sector of work, if a significant and cost-effective difference is to be made in the longer term.
Suggested implementer: The UN’s emergency coordination group and the new UNDAP process in Tanzania are ideally positioned to take this forward

3. It is important for the IPs and the PUNs to minimize staff turnover to ensure continuity, and certainly to avoid gaps between incumbents in key positions. While there may be many necessities for staff turnover in Government and in the UN Agencies, efforts should be made to reduce this to the minimum and to accept it only when absolutely necessary. Likewise it is important to motivate staff and to reduce the long period of key staff operating in an acting capacity.
Suggested implementer: All UN agencies and IPs.

4. It is suggested that the complaints of a ‘raw deal’ from Zanzibar, whether real or imagined, need to be addressed by taking extra care in future projects involving both parts of the United Republic to ensure that the islands get adequately included in the overall process and receive a realistic share of UN support.
Suggested implementer: All UN Agencies under the leadership of the RCO to show the true benefit of Delivering as One.

5. It is important that a solution is found to the question of sustainable options for resupply for the strategic warehouses, both to maximize the potential and to build on the initial investments. Solutions are possible but have to be explored and worked through.
Suggested implementer: PMO/DMD and 2nd VPO/DMD with technical advice from the UN’s emergency coordination group and/or other PUNs.

6. Delivering as One: While the ‘DaO’ is a pilot, it is important to emphasize that the initiative is THE way UN support will function in Tanzania. While closer integration will not happen overnight, the RCO needs to lead the process and Agencies and their staff need to engage with it more than is currently obvious. It is important to strengthen the UN Resident Coordinator Office/System so that it can better steer the ‘DaO’ initiative, and important that more progress is “seen rather than talked about”.
Suggested implementers: RC’s office and all UN Agencies and personnel.

7. Decision making authority in the disaster management process has to be moved to lower levels, and the officials there have to have the skills and tools to handle it. While decentralization will not happen overnight, JP 6.2 has already started by empowering district disaster committees and the regional administrations, and this work should definitely continue. Particularly for risk reduction and preparedness activities, the decision-making has to be as close to the community as possible, as well as including them.
Suggested implementers: PMO/DMD and the 2nd VPO/DMD to provide ongoing inputs and support to sub-national authorities responsible.

Lessons Learned (Optional):

A number of clear lessons have come out of this JP 6.2 over its implementation period. Many have already been taken into account to inform ongoing work and it is not a finite or finished process.

1. The planning for a programme such as this, especially when it is a new approach as it was under the ‘DaO’, needs to be comprehensive, clear and coherent. It should also be done at the start of the implementation period, and not one year into it. It can be ambitious in its expectations, but also needs to account for the operating realities and likely constraints that will be faced, and be flexible enough to deal with them. It needs to be inclusive and have clarity of outcomes.

2. Furthermore, coherence and continuity of documentation and terminologies allow a much easier follow-through of programme activities, especially for individuals coming into the programme after the initial planning work has been done.

3. Working on a Joint Programme such as this was a new experience for UN Agencies, IPs and individuals, and required a more cooperative approach and less of the traditional ‘silo thinking’. This challenged people’s comfort zones and there was limited support from the usual reference points such as Agency HQs. It was also a pilot – but it has worked and most of the issues have been worked through during the period.

4. This approach has – perhaps more than anything else – helped develop and maintain enhanced and more cooperative working relationships where they did not exist before. Robust working networks have been established between Ministries, between UN Agencies and between the IPs and the PUNs, and these have helped deliver a better understanding and more efficient ways of working together. This will be required under the coming UNDAP process also.

5. It makes good sense to align the initial AWP cycles to those of the Government, as well as to the identified national priorities. This allows the Government to take the required level of leadership in the activities, and overall ownership of the outcomes.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search