2013 United Rep. of Tanzania: Evaluation of Mobile Decision Support Applications in Tanzania
Author: BluWat Ltd
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labeled as ‘Part 2’ of the report."
UNICEF is working to support the Tanzania Ministry of Health & Social Welfare and the Zanzibar Ministry of Health to strengthen the quality of service delivery for children affected by Severe Acute Malnutrition (SAM) in Zanzibar through an eNutrition project in Zanzibar and Maternal Health through an eMNH project in Bagamoyo. This support was implemented as part of the United Nations Development Assistance Framework (2007-2011) and the United Nations Development Assistance Plan (2011-2015) priorities for Health & Nutrition within which UNICEF has committed to supporting the MoHSW and the MoH to strengthen health and nutrition monitoring and evaluation systems.
As part of this effort, UNICEF has been seeking innovative ways to improve the capacities of frontline health workers in the use of clinical guidelines and protocols to contribute toward improved quality of care. To achieve this UNICEF has been working with D-tree International, a non-profit organization specializing in the development of electronic decision trees to assist front-line, in-community healthcare workers.
In 2013, UNICEF commissioned an external evaluation of the two projects implemented by D-tree through a competitive bidding process in which Bluwat (Tanzania) Limited were selected to undertake the assignment. The external evaluation will contribute to accountability and learning for key stakeholders including the Tanzania Ministry of Health & Social Welfare and the Zanzibar Ministry of Health, D-tree, UNICEF, sub-national health teams and civil society; provide recommendations on future scale up; and inform UNICEF Tanzania Country Office programming strategies linked with the use of mobile technology for different areas of programme implementation and information systems strengthening.
The purpose of the evaluation is to assess the relevance, efficiency, effectiveness, and the sustainability of the use of mobile technologies for the screening and treatment purpose in the areas of nutrition and maternal health. Given the pilot nature of the project and duration of implementation over a two year period each, the criteria of impact is given lesser weight. Greater weight is given the criteria of relevance, efficiency, effectiveness and sustainability to strengthen the scope of examination on the pilot nature of the projects and their capacity to go to scale within the development context of Tanzania.
The evaluation uses a cross-sectional plausibility evaluation design using an external control group to rule out other confounding factors which may have resulted in observed effects on the treatment group. Comparator facilities were selected based on similarity to the treatment sites in terms of socioeconomic and livelihood systems but where they were not exposed to the benefits of the project. For the maternal health project, two sites Chalinze and Kwaruhondo were selected for comparison with Chalinze and Miono in Bagamoyo district, whilst for the nutrition project, Sebuleni was selected as a match to Chumbun in Unguja Urban and Chaani Msingini, Kidoti and Gamba were selected from Unguja North A. All control areas were in proximity but an adequate distance from the treatment sites to prevent any spillover effects from the intervention.
The evaluation covers all the eight facilities that benefited from the intervention including the two eMNH project sites in Miono and Lugoba, Bagamoyo district and the six eNutrition projects in Zanzibar urban (Chumbuni) and those in Unguja A District (Chaani Kubwa, Kivunge, Tazari, Nungwi and Gomani).
Findings and Conclusions:
Relevancy – Overall the project objectives were found to be relevant and in accordance with the priorities and policies of the MoHSW on the Mainland and the MoH in Zanzibar.
Effectiveness – Overall the project sites where eMNH and eNutrition had been introduced had improved the accuracy and comprehensiveness of treatment. In these same facilities however, the accuracy and comprehensiveness of completion of the existing paper based systems for data gathering declined even below the levels of the comparators due to the additional burden of work in completing the task twice on the device and on paper.
Efficiency – Cost analysis of the eMNH project finds fixed costs to a total of TSH 399,669,649 and variable costs as TSH 29,039,992 ; for eNutrition project finds total fixed costs at TSH 350,292,034 and variable costs as TSH 35,400,000 over 2010 to 2012. On the basis of the review of variable costs, TSH 28,249 was expended per case (approximately $17/case) for the eMNH project as a result of the higher caseload for ANC over TSH 177,889 was expended per case (approximately $109/case) for eNutrition due to the smaller number of client interactions over the project period.
Impact – Clients of both project noted positive interactions with health workers using the eMNH and eNutrition tools compared with the experience of clients in comparator sites. Compared to the Sphere standards for nutrition however, the projects sites underperformed in terms of recovery and default rate.
Sustainability – The huge investment for the extremely limited scale of coverage in just six facilities in Zanzibar and two facilities in Bagamoyo limits the ability to advocate for investment on the basis of value for money. Implementation of a product such as eMNH and eNutrition requires a substantial resource investment both in terms of human resources for technical support and in terms of financial resources.
1. Strengthen process and evidence to inform problem identification and programme design
2. Improve linkages between the use of mHealth initiatives such as eNutrition and eMNH with other health systems strengthening initiatives
3. Ensure a consistent and active presence in maintained in the mHealth CoP
4. Ensure plans for scale up factor in the requirements of other complementary interventions and resourcing requirements
5. Strengthen programme design and evidence base to improve project evaluability
6. Strengthen mHealth Community of Practice in Zanzibar
7. Strengthen regulatory environment for mHealth in Zanzibar to promote integration with national health information systems
8. Strengthen community management of acute malnutrition
1. The importance of having fully functional programmes including maintaining a continuous supply pipeline for essential commodities and equipment cannot be under-estimated, otherwise diminishing the contribution of the mobile applications toward improved quality of care.
2. Integration between facility based service delivery and community outreach and support pays dividends in strengthening access to health and nutrition services and reducing the burden of facility based care
3. Assessments of network coverage and system requirements should be established as part of programme/project design.
4. Identifying an area of intervention with the potential for high coverage creates a more substantive evidence base for advocacy and leverage of resources.
5. Maintaining a rolling training plan to address staff turnover
Full report in PDF
PDF files require Acrobat Reader.