Evaluation database

Evaluation report

2013 India: Evaluation of Social Mobilisation Network in India for Eradication of Polio



Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report."

Background:

The Social Mobilization Network (SMNet) was established by UNICEF in Uttar Pradesh (UP) in 2002 and expanded to Bihar in 2005 – the two Indian states with the highest incidence of polio cases – to generate community support for polio immunization activities. The SMNet deployed community mobilizers in areas identified as high risk with the main task to work with resistant communities and to encourage uptake of the oral polio vaccine (OPV) during Supplementary Immunization Activities (SIA). Over the years the programme has built a strong decentralized base of trained human resources – 6500 Community Mobilization Coordinators (CMCs) (1 per 500 households) across UP and Bihar; Block Mobilization Coordinators (BMCs) for mentorship and supportive supervision of CMCs (1 per 8-12 CMCs); District Mobilization Coordinators (DMCs) and District Undeserved Coordinators (DUCs) for every district; 3 Sub-Regional Coordinators (SRCs) in UP and 2 in Bihar for the delineated regions; and the state level polio units for leadership.

The programme has evolved over the years in response to changing national and local priorities. High Risk Groups and Areas (HRGs and HRAs) were identified and prioritized. The Underserved Strategy was developed to reach high risk communities and was expanded from including Muslim sub-sects to also including groups which have been missed during immunization rounds due to their occupations and lifestyles. These included groups like nomads, brick kiln and construction workers, slum dwellers etc. The field level interventions were strengthened over time in Bihar in response to repeated outbreaks of polio. The ‘X’ code, used to mark missed households, was expanded to allow CMCs to identify the causes for not immunizing a child. This allowed a targeted strategy for converting ‘X’ households to ‘P’- households where all children have received OPV in the current immunization round. By involving community leaders and building local ownership, the intervention was able to develop locally relevant strategies to strengthen polio eradication efforts.

Purpose/Objective:

The objectives of the SMNet strategy were to:

  • Maximize the impact of communication efforts at the national, state, district and block level through strengthened coordination amongst partners and effective advocacy 
  • Ensure children most at risk – particularly those under the age of two, Muslim and boys – are adequately protected from polio by intensifying efforts in blocks where wild polio virus transmission is sustained 
  • Increase the total number of children immunized and turnout at the booth by achieving a critical mass of communication activities in all high-risk areas of priority blocks in states with on-going wild poliovirus transmission 
  • Ensure polio eradication by strengthening communication for routine immunization especially in polio endemic states

Methodology:

Keeping in mind the complexity and scale of the SMNet, this evaluation used a mixed-method design in order to comprehensively and judiciously cover each aspect of the intervention. The research methods used for the evaluation were:

 The literature review and desk research helped the team understand the objectives, rationale, operationalization and outputs/outcomes of the programme and provided the basis for conceptualization of the study design and tools. Additionally, review of published peer reviewed papers and reports provided an understanding of the macro-systemic context of SMNet. Existing evidence was used for comparison of findings from the current evaluation.

 The secondary analysis used data from UNICEF state offices, SIA rounds, National Polio Surveillance Programme (NPSP) that was made available to the evaluation team. Besides this, data in the public domain from Annual Health Survey 2010-11 (AHS), District Level Household Survey 2007-08 (DLHS), data from the Ministry of Health and Family Welfare (MoHFW) on budgets, polio/routine immunization were used. The secondary data analysis helped understand the equity focus of the interventions including targeting of High Risk Groups (HRGs) and High Risk Areas (HRAs). The data analysis also helped in understanding the evolution of the SMNet strategy. A trend and comparative analysis was undertaken to analyse the impact and effectiveness of the intervention.

 The primary qualitative research involved in-depth interviews with stakeholders from across the government system at state, district, block and sub-block levels, representatives from partner organizations, and frontline workers at the community level (ASHA, ANM, AWW, CMC); and focus group discussions with community members (mothers of children in the 0-5 year age group, other caregivers and influencers like grandmothers, fathers, local leaders, etc.). Stakeholders at each level were selected on the basis of a sampling design. This was used to understand stakeholder perceptions about the SMNet and the processes of change catalysed by it.

Findings and Conclusions:

Relevance

The key evaluation findings are provided below. 

  • The design and interventions of SMNet are aligned with community needs 
  • The SMNet approach has been relevant to achieve the results of the polio eradication programme by reducing resistance to vaccination and reaching the unreached in polio endemic states of UP and Bihar 
  • SMNet has accounted for contextual realities in the programming environment and responded to evolving priorities over the years through relevant strategies

Effectiveness

The main findings and inferences are as follows: 

  • Knowledge, Awareness, Behaviour and Practices related to OPV have improved in SMNet areas 
  • Reduction in refusal rates can be attributed to SMNet awareness raising and mobilization measures 
  • There has been a net increase in number of children accessing the OPV at vaccination booths in UP 
  • The community has a high level of faith and trust in the CMCs 
  • SMNet has worked in close collaboration with the public health system 
  • The SMNet interventions original objective have been achieved to a large extent 
  • The SMNet focus has expanded over time from only polio immunization to other child health issues through the Polio Plus initiative and increased focus on Routine Immunization 
  • The intervention has responded to numerous challenges and constraints with innovative solutions 
  • By focusing on high risk areas and high risk groups/ underserved community- Muslim community, Brick kiln and construction workers, Nomadic groups and Slum dwellers, the intervention was designed to address issues of equity. This approach ensured that those who were worst affected were prioritized. 
  • The intervention engaged women at decentralized levels as change agents. Preference was given to employing and training female CMCs in order to reach mothers/caregivers meetings. As a secondary result of the intervention, women have been empowered and are active decision makers in household decision making. Through the SMNet, CMCs have access to a platform for growth and learning, there is a large degree of social recognition and an increase in mobility.

Impact

The main findings and inferences are: 

  • The SMNet intervention has led to a decline in refusal to OPV. Resistant households (XR) reduced by approximately 61% in UP and by 42% in Bihar between 2007 and 2012. 
  • The intervention has led to consistent and significant increase in Knowledge Attitude Behavior Practices with the access of communities to FLW s and CMC visits. (Meta-Analysis of KABP studies showed a strong linear relationship (Correlation coefficient (r) =0.51 for KA and r=0.90 for BP). 
  • The intervention’s Underserved Strategy was developed to cover those in hard to reach areas and chronically missed communities due to issues in access, reach and health seeking behaviours of these communities. Through locally relevant IEC and outreach activities, the intervention sought to reach the unreached.

Efficiency

The main findings are as follows: 

  • SMNet has utilized funds in an economical manner and has indicated allocative efficiencies 
  • SMNet’s financial monitoring systems and processes need strengthening
  • The lean staffing of SMNet for national and state offices indicates efficiency of HR management 
  • While reporting and monitoring mechanisms are in place, there is scope for strengthening 
  • The outputs of SMNet in terms of coverage and unit costs indicate a cost-efficient and fairly economical programme 
  • Forecasted costs of SMNet for the next decade support a strong case for continuing eradication interventions as the most cost-effective option

Sustainability

The key findings and inferences are: 

  • CMCs are accepted by the community and by other stakeholders as FLWs and change agent 
  • SMNet is convergent with the public system and its frontline workers 
  • Keeping sustainability concerns in mind, SMNet has started building a strategy on "Polio Plus" interventions. SMNet needs to develop sustainable plans for quality assurance, stringent monitoring/surveillance and capacity strengthening of its personnel in the context of a broadened scope and ensure clear role definition to complement role of other frontline health workers (ASHA and AWW) 
  • Effective strategies of SMNet like utilizing local health workers, involving religious and local leaders to address resistance, area specific and population specific approaches etc. can be considered for replication in other contexts.

Conclusions and Learnings from the SMNet Programme

  • The SMNet was evidence-based in its strategic planning and programmatic focus, which ensured that its objectives, spatial targeting and interventions were efficacious and effective towards the main goal of eradicating polio. 
  • The strategic positioning of the programme along with convergence with the public system has been crucial in determining its impactful functioning and sustainability. 
  • The SMNet programme structure provided for effective linkages with the public health system at all levels – national, state, district and blocks with participatory relationships with PRIs. These need to be institutionalized. Besides this, the SMNet needs to maintain links with civil society. This can help in bringing about convergence of activities and efforts towards better programme implementation, especially given the Polio-Plus focus going forward. 
  • The SMNet has remained relevant to the needs of the community and to the changing national context. Many factors may independently, or in interaction with each other, contribute to a need for shifting the focus areas of a complex and large scale programme like the SMNet such as, national policies, policy context in the state, development priorities, donor priorities, emergencies, etc. However the programme has to be flexible and dynamic to respond to such changes. In doing so, the programme needs to ensure that it remains relevant – this is possible by being cognizant of the objectives and goals and ensuring that it serves the need of the most vulnerable. The SMNet responded to the large outbreak of polio in Bihar in 2009 by introducing CMCs and strengthening field level interventions. Additionally, since 2012, the intervention has also placed CMCs at border areas to control importation of the virus. 
  • Clear statement of goals and objectives and process/outcome/impact indicators of the SMNet have been lacking and it is important to have these in place, along with rigorous systems and mechanisms to review these regularly against activities and changing policy needs. 
  • Given the importance of eradicating polio, not just as a country priority but also as a global concern, and the intensive resources that have been invested in these efforts, it is imperative that a programme like SMNet plans for a results monitoring/evaluation framework right at the outset. This is crucial to create evidence on efficacy and impact of the interventions. 
  • The SMNet has maintained lean but optimal staffing human resource aligned with the objectives of the intervention. Lean staffing at administrative/managerial levels with right expertise is important to ensure quality of services and technical inputs. This contributes to efficiency in human resource management, while focusing on building an optimal operational cadre – which is important for a decentralized community-based programme like SMNet. 
  • A rationalization of human resources is important in the context of existing personnel at the ground level in the public system. The SMNet in Bihar undertook this in involving existing AWWs instead of introducing CMCs in most areas. This is important to avoid duplication of roles and responsibilities of personnel and maintain high resource efficiencies. 
  • Balancing expanding scope while maintaining quality and focus of the programme on polio eradication is essential to ensure effectiveness and sustainability. This assumes importance in the context of the Polio-Plus focus and SMNet’s increasing involvement in RI and other child health interventions.
  • Stringent systems and processes for financial management with rigorous records of budgets, allocations and expenditures are crucial for a programme the size and scale of SMNet. This is especially true with several stakeholders and lines of funding involved in the programme and needs to be strengthened further. 
  • Knowledge management is critical along the life cycle of the programme. SMNet had several innovations that were undertaken at decentralized levels as responses to local contextual realities. In order to learn, make course corrections, and create knowledge about these innovative strategies and operational mechanisms of SMNet, it is important to have official documentation of the evolution and rationale of change along the project life cycle.

Recommendations:

The broad recommendations based on the evaluation are put forth in four broad areas – 

  1. Institutional and Policy Level 
  2. Structural Level
  3. Operational Level 
  4. Replication Level

A. Institutional and Policy Level

  1. Continue and replicate the effective interventions of SMNet
  2. Government and donors should extend support to continue eradication efforts, compared to available alternatives
  3. The objectives of the SMNet should be fine-tuned based on contextual needs and should reflect the current health policy/programmatic context
  4. Continue to maintain a balance between flexibility and focus on goals

B. Structural Level

  1. Staffing of SMNet should remain lean but optimal to match the increase in scope/work load and human resources should be rationalized in alignment with existing resources
  2. Imperative to strengthen financial management systems and processes
  3. Need to design a results monitoring framework for tracking achievements and indicators
  4. Need to strengthen linkage with civil society

C. Operational Level

  1. Need to review human resource and training plans in the context of the expanded scope of SMNet
  2. Undertake formal financial reviews
  3. Continue to ensure an appropriate mix of activities
  4. Strengthen knowledge management
  5. Organize regular capacity building of staff for knowledge updation, motivation and supportive supervision
  6. SMNet needs to develop strategic role rationalizations for its operational staff vis-à-vis the public system FLWs
  7. SMNet can develop its CMCs as mentors of the ASHAs in a child health and nutrition role

D. Replication Level

  1. Replication models need to adapt specific relevant interventions/strategies of the SMNet rather than the model as a whole
  2. Strategic positioning and partnerships is crucial in determining the model’s impactful functioning and sustainability
  3. The model must be flexible and dynamic capable of emergency response
  4. Objectives and role of the model/programme must factor in specific needs and context of the local geography
  5. The model must have convergence with the public health system and be designed to suit local realities
  6. The model/programme should be staffed with lean but optimal human resource, with expertise in alignment with core objectives and balancing with existing resources
  7. The model should design a rigorous and robust evaluation framework with defined primary indicators for processes/outcomes/impact
  8. Develop systematic and needs-based capacity-building plans  


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
India

Region:
ROSA

Theme:
Immunization

Type:
Evaluation

Language:
English

New enhanced search