Author: John Snow, Inc.
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report.”
For many years, the Expanded Programme on Immunization (EPI) in Indonesia has shown high levels of overall performance, with WHO/UNICEF estimated national coverage rates of 80% or higher indicated for most key EPI antigens. Survey data indicates somewhat lower national coverage figures, however, and also reveal substantial performance variations within regions, with pockets of low coverage in some areas. These are often inaccessible or remote locations, where available transport options are limited, terrain is challenging and populations are small and dispersed. As a result, service delivery can be difficult, time-consuming and expensive to provide or sustain, and in such areas, infants and children typically do not have access to the high levels of health care provided in more accessible parts of the country.
In an attempt to address these inequities and to achieve its Millennium Development Goals, (MDGs), the Ministry of Health (MOH), through its National EPI unit located within the Directorate General of Disease Control and Environmental Health, has piloted a strategy known as ‘Sustained Outreach Services’(SOS) in selected areas of Indonesia. This intervention is based on a WHO global strategy, (Annex 1) and adapted by the MOH to provide populations in remote areas with a package of basic immunization and maternal health services, on a regular and sustainable basis. Beginning In 2009, and with financial support from the UNICEF Indonesia office, and technical support from both UNICEF and WHO, this approach has been progressively phased into a number of remote locations of priority districts in three provinces of Eastern Indonesia. Lessons learned from this pilot activity will be used to assist in formulating national EPI policy and in enhancing programme strategies, and after an initial two years of operation, MOH required an evaluation of the experiences and resulting impact on programme performance of the SOS intervention.
At the request of the UNICEF Indonesia country office, the evaluation was carried out by John Snow Inc.(JSI), an international public health organization with substantial experience in immunization programmes worldwide, and of their monitoring, evaluation and implementation (Annex 2). Appropriate Terms of Reference for the activity were prepared by the UNICEF office in May 2011 (Annex 3), and the evaluation itself was conducted between September and December of 2011. In view of the relatively short time since its introduction, evaluation of the SOS strategy was designed to focus on process issues rather than on outcomes and programmatic achievements. The latter will only become apparent after a longer period of implementation in the pilot provinces and districts, and the accumulation of a larger body of programme data and operational results.
The evaluation was designed to assess the relevance, effectiveness and efficiency of the Sustained Outreach Services (SOS) strategy as it is applied in the current target areas of Indonesia. The specific purposes of the activity were to:
1. Document factors that supported SOS programming in Indonesia, including those that led to changes in immunization coverage in the target areas,
2. Recommend sustainable mechanisms as considered appropriate for wider application of SOS in the national EPI and the broader child survival programmes of the government,
3. Identify any needs for additional human and financial resources required towards solving inequities in the long term, that will increase and sustain immunization coverage levels in hard to reach areas, and
4. Contribute to global knowledge on equity and evidence-based, cost effective/ cost efficient strategies for reaching the poorest populations in geographically hard-to-reach areas.
In addition, the evaluation would attempt to identify key elements contributing to SOS programme implementation, to determine gaps in achieving EPI coverage targets, and to summarize the lessons learned.
The overall design for conducting the evaluation was detailed in a proposal submitted by JSI to the UNICEF country office in Indonesia on 15 June 2011. (Annex 4) The design outlined 3 main phases for the evaluation:
Phase 1: Desk Review and Draft Evaluation Design (September – October 2011)
Phase 2: Field Work (October – November 2011)
Phase 3: Preparation of the final report (November – December 2011):
A quantitative and qualitative data collection instrument for use during phase 2 field work was prepared, with specific sections designed for each group of individuals to be interviewed and each administrative level involved in delivering the SOS programme. The instrument was field tested and a Indonesia language translation was added to the text to produce a dual-language instrument for ease and flexibility of use in the field. (Annex 5).
Findings and Conclusions:
The Expanded Programme on Immunization in Indonesia has long shown high levels of overall performance, with WHO/UNICEF estimated national coverage rates of 80% or higher indicated for most key EPI antigens over many years. Survey data shows somewhat lower national coverage figures, but more important, reveals substantial performance variations within the country that are not apparent from the national figures. Many of these areas of lower performance are inaccessible or remote locations, where available transport options are limited, terrain is challenging and populations are small and dispersed. Service delivery can be difficult, time-consuming and expensive to provide or sustain in such areas, and typically, infants and children do not have access to the high levels of health care provided in the more accessible parts of the country.
To address such inequities WHO devised a strategy comprising a package of immunization and other health services that could be delivered to isolated and remote areas on a more regular and sustainable basis. This strategy, know as the Sustainable Outreach Services (SOS) approach, was not a replacement or alternative to routine immunization programmes, but was designed to build upon an existing service, and enable it to extend its range of activities to areas not previously served by the routine programme alone. Beginning in 2009, Indonesia has piloted this approach in selected areas of some of its remote Eastern provinces as a means of addressing low immunization coverage in those areas, and thereby, contribute to achieving its Millennium Development Goals nationally. The SOS strategy has been progressively phased into selected districts of three of these Eastern provinces to date, and MOH required an evaluation of the resulting impact of the new approach on immunization services in the target areas.
At the request of the UNICEF Indonesia country office, the evaluation was carried out by John Snow Inc.(JSI), an international public health organization. In view of the relatively short time that most SOS-implementing areas had been operational, it was agreed that the focus of the evaluation would be on the process of implementation, rather than its programmatic impact. The five main programme areas to be covered by the evaluation were the intervention design, its implementation, the results, the costs and a summary of strengths, weaknesses and lessons learned from the pilot phase. In addition, the evaluation assessed some key indicators of the routine immunization programme upon which the SOS strategy is built, and whose infrastructure it largely shares. The main method of qualitative data collection was through interviews of health staff and officials at national, provincial, district, health facility and community levels, and was supplemented by observational, quantitative and financial data gathered wherever possible. The evaluation was conducted between September and December 2011, and included field visits to all of the remote SOS-implementing areas in 6 districts of 3 Eastern provinces of Indonesia.
Results of the evaluation show that in the overall design and rationale for the intervention, there have been significant shortcomings in documentation and communication from the national level. While the specific objectives and characteristics of the local version of SOS may have been understood by those involved in its design, these fundamentals have not been adequately articulated, documented or disseminated for national use. As a result staff interviewed at province, district and health facility levels did not have a good understanding of these issues, and did not have a clear idea of how SOS was a related but distinct activity from the routine programme.
Some shortcomings were also noted in implementation of SOS. Some districts completed all key preparation steps, but others omitted important items, in particular, micro-planning and mapping of hard-to-reach areas. For staff training, emphasis was placed on socializing and mobilizing for SOS, but little attention was given to implementation guidelines and technical issues. None of the districts made changes to the schedule of vaccines offered, and all continued to target the same age groups as routine services. No specific provisions were noted for identifying, immunizing and recording doses given to older age groups, or for reporting these to district level. Much of the inherent flexibility offered by the SOS approach has not been used in Indonesia, and in effect, SOS was implemented as a pulsed routine EPI activity only, with some other services added. An ideal schedule for the timing of SOS rounds in a typical year shows that up to 4 SOS rounds per year are possible, although it is unclear whether funding limitations will actually permit this in practice. This schedule applies only to Unicef-funded SOS activities, but for any future government funding of SOS, typical delays of 6 months or more in release of local government funds may only permit one or two SOS rounds to be conducted per year.
The strategy was apparently well-accepted in all 6 SOS-implementing districts to date and interviews reflect a belief that results are positive and better than expected. There is little quantitative data so far to support such views, although the first district to implement has provided limited data which shows some decrease in performance over the brief period when SOS was implemented. This is too small a sample to draw any firm conclusions on the SOS strategy, but a larger body of data was collected on key indicators of the routine programme across all health facilities visited. This reveals a number of serious weaknesses and failures in basic functions of the routine programme, and shows that many of these are either being ignored completely, or are being carried out at completely inadequate levels. Such widespread weaknesses must inevitably compromise service quality at all delivery points and undermine any advances made, both by the routine programme itself, and thus by the SOS strategy.
Targeting small, remote communities for SOS has resulted in significantly higher costs and for this initial stage of implementation, operational costs are estimated at 3 to 5 times higher per village than for the routine programme. Benefits from these higher costs have yet to be realized, partly due to a limited operating period and thus limited data, but also due to programmatic weaknesses and compromised service quality. It was found that operational costs for SOS were only part of total costs for the strategy, and implementation to date has relied mainly on existing infrastructures for both land and sea transport, and on existing human resources. This approach can only offer a temporary solution, and to become a sustainable intervention, substantial additional investment for SOS will be needed in infrastructure, capital and human resources.
• appropriate guidelines, explanatory and training materials on the local SOS programme to be prepared as a matter of priority
• planning, preparation and staff training prior to introducing SOS to be improved, and districts to include all key activities in preparations
• a detailed assessment of land and sea transport and of manpower needs in all SOS-implementing areas to be carried out
• immediate and intensive efforts to re-focus attention on ensuring routine programme quality at all levels
• urgent measures to ensure that all basic immunization programme functions are performed regularly and comprehensively across all health facilities
• these steps to take priority over any consideration of expanding SOS, which would be premature and ineffective until basic programme quality can be assured.
PDF files require Acrobat Reader.