2000 PAK: Third Party Evaluation of Expanded Programme on Immunization, Punjab
Author: Consultants Consortium SoSec KEMC
The Expanded Program on Immunization (EPI) of Pakistan was one of the most successful programs in the developing countries in the early '80s, but its coverage decreased with the withdrawal of international support in the mid-'90s. Presently, the program is working hard to achieve the objectives: to reduce morbidity and mortality resulting from the six EPI target diseases. Despite various interventions and significant inputs from donors, the results have not been encouraging and there is continuous low coverage. To improve the coverage, the Government of Punjab, Health Department, organized the Crash Program from July 1999 to October 1999. It was decided that the evaluation of the EPI program should be entrusted to an independent, credible Third Party Evaluation (TPE) to avoid any bias factor.
Purpose / Objective
The study objectives were to determine the extent to which the objectives of EPI Crash Program were met satisfactorily through a Third Party Evaluation, assess the efficiency and effectiveness of the Health Department in terms of mobilization and utilization of resource, and identify factors that contributed towards the success and the shortcomings, and draw lessons for future intervention.
A quantitative survey was designed according to the World Health Organizations (WHO) standard for EPI Evaluations i.e., 30 Cluster-Sampling Technique. Planning of the survey was done both at macro and micro level. Punjab was divided into four regions, namely: Bahawalpur, Faisalabad, Lahore, and Rawalpindi; with each region having 8-9 districts.
A qualitative part included structured interviews with policy makers, senior & mid-level managers, service providers and community opinion leaders. Discussions were held with the Secretary Health, Director General Health Services, Additional Secretary Health (Technical) and Director EPI, along with his provincial program team. TCC Teams also conducted detailed interviews with the operational staff including Divisional Directors, District Health Officers, Deputy District Health Officers, District Supervisors Vaccination (DSV), Assistant Supervisors Vaccination, Medical Officers, Vaccinators and CDC Supervisors. International donor Agencies like UNICEF, WHO and World Bank were also visited.
Key Findings and Conclusions
Immunization coverage of children by cards and history in Vehari was 95% and 92% in Multan; the lowest was in Rajan Pur (28.9%), although the overall completed vaccination coverage in Punjab was 70%. The rest of the districts showed a coverage lower than 60% for all the antigens. 68% of the children were fully vaccinated, 26.8% were partially vaccinated while only 5.4% of the children remained unvaccinated on the whole. The highest proportion of fully vaccinated children was reported from Vehari (94.9%) and Multan (91.2%); partially vaccinated children were highest in Rajan Pur (48.3%) followed by Faisalabad (45%), Muzzafargarh and Bahawalpur (44%). Twenty-seven percent of the children remained unvaccinated in Rajan Pur, followed by Okara and Layyah at 10%.
Vaccination Coverage of Mothers for first 2 doses of TT in Khushab was (76%), Lahore urban (84%) and Sialkot (82%). These districts showed the highest coverage. Seventeen out of the rest of the 32 districts showed a coverage of more than fifty percent. On the whole, coverage with two doses of TT came up to 63%. However, the coverage for three doses came down to 18% and for the fourth and fifth dose, it came as low as 5% for all the districts. The coverage for TT1 and TT5 ranges from 69% to 5%.
National Immunization Days and Sub National Immunization Days (NID/SNIDs): The coverage in Punjab in the first round was 98% (7,233 out of 7,372) while 92.3% had their second dose of OPV along with Vitamin A. In the districts Rajan Pur, Layyah and Attock, SNID coverage reported for 2nd dose with Vitamin A was the lowest (81-84%).
The coverage during the Crash program was assessed by immunization cards (n= 3,450). The number of children vaccinated with BCG was 789. Out of this, 627 (79.5) benefited from the 'crash' program for BCG vaccination when we considered their age at vaccination at least one month outside the recommended age for BCG vaccination. For OPV first dose, out of 899 children, 574 (63.8%) of the children were immunized, giving a margin of one month after the due age of vaccination i.e., 2 months. For OPV third dose, out of 1175 children who missed their third dose, 741 (63%) benefited. For DPT first and third doses, nearly 64% of the children were vaccinated outside the routine immunization. For measles, 41.6% (381 out of 916) of the children vaccinated during this 'crash' period were those who had missed their opportunity in routine. Thus, the 'crash' program seems to have covered a large number of 'missed opportunities' and had increased the proportion of children fully vaccinated as a whole.
During discussions with policy makers, operational managers and field staff, a number of issues were raised, which have been grouped under various components, as follows: Program Resources; Technical Aspects; Program Management and Organization; Communication and Community Participation; and Monitoring, Evaluation and Feedback.
Presently, all the supplies are procured at the federal or provincial level and distributed to the districts, with frequent complaints of erratic supply, thus adversely affecting the Program. Inadequate and worn out Cold Chain Equipment. Restricted mobility of the supervisory and field staff due to shortage of vehicles and POL. Mal-deployment of vaccinators.
There is no regular training program for all categories of EPI staff, though there is a well-established training institution network of PHDC - DHDCs operational in the province. WHO training manuals exist for training different categories of EPI staff. The sporadic trainings offered were not focused on skill building.
Many health care providers lack knowledge regarding TT vaccination schedule. Moreover, deficient communication to pregnant women regarding TT vaccination poses another hurdle. Field staff sometimes lack the very basic knowledge and skills to operate and maintain the cold chain. Recording and maintaining vaccine temperatures have been cited as a problem in the field.
The managers and supervisory staff lack supportive supervision concept. Most supervisors lack the technical knowledge essential for effective supervision. Stress is more on administrative and side issues, rather than the technical ones.
Program Management and Organization:
Organizational communication is deficient in a number of ways and it was felt that key functionaries like DHO and MS DHQ, DDHO and MS THQ lack coordination amongst them, though their supervisor (i.e. Divisional DHS) is the same. Lack of coordination between various departments working for similar objectives (e.g. Department for Population Welfare, Department of Education, Department of Social Welfare).
Though in principle it has been agreed that the Medical Officer in charge of the health facility shall be responsible for the EPI coverage within his catchment area, the DHO office is still managing the field staff directly. Clear-cut job descriptions are lacking at all levels, making the division of roles and responsibilities ambiguous. People responsible lack the knowledge of proper planning and are usually unaware of its real importance. District level managers and field officers are not sufficiently authorized to get their equipment repaired or replaced.
Accountability is patchy and initiating disciplinary actions against the non-performers is not an easy task. The Annual Confidential Reports (ACR) system is one of the institutional tools to gauge the performance. Unfortunately, ACR is neither annual nor confidential, resulting in lack of confidence in the mechanism.
The system does not offer any performance-based incentives to the staff, either in form of written recognition or monetary benefit. Salary increments or posting and transfer to better places are not linked to work performance. This affects the attitude and morale of the staff, as excellent or poor performance has no effect on immediate gains or losses.
Communication and Community Participation:
IEC is given very low priority and budgets. Vague, unclear, incomplete or mixed messages are promoted, which fails to convey the directive to families and communities. Involvement of the community people in planning, implementation and monitoring of health services is minimal. Promotion material generally developed at central level and field staff involvement is minimal. Proper training program on the use of material is virtually non-existent.
Monitoring, Evaluation and Feedback:
It is reported to be a major issue during program implementation. Irregular monitoring: Although there is a defined monitoring system, starting from federal level to the tehsil level, but the impact of monitoring is not visible. Inefficient use of collected information: Visiting supervisors are not in the habit of carrying pertinent information already available while visiting the health facilities. The decisions are, therefore, whimsical rather than evidence based.
Decentralization of Supplies: Distribution system for vaccines and supplies needs to be redesigned with storage facilities at the divisional level. Decades-old Cold Chain equipment at the peripheral needs to be replaced.
Deployment of Staff: There are instances of mal-deployment of vaccinators in the field. This needs correction through administrative measures. Job Descriptions for all categories of staff need to be developed and implemented.
Training and Skill building: It is recommended that appropriate and focused training programs be initiated for different categories of EPI staff and skill building through hands-on training. WHO manuals need to be modified according to local needs and be used for such trainings. Trainings should have special emphasis on IPC, maternal TT and cold chain maintenance. Training of staff in AFP surveillance also needs improvement.
Community Involvement: Civil Society may be involved in program advocacy, planning and implementation through CBOs, NGOs and community leaders. LHWs have formed Health Committees, which can play an important role in program propagation for community involvement.
Crash Program was a successful experience. It is recommended that the cadence made during Crash Program be extended.
NIDs/SNIDs: A general impression regarding NIDs and SNIDs, which prevailed amongst some mid-level mangers, is that these occasions consume significant time and resources from the routine activities, thus affecting routine coverage. However, it was admitted that NIDs and SNIDs generated awareness and motivated the supervisory staff to look into the routine EPI coverage and plan accordingly. It is clearly a false impression that the routine coverage has declined because of Immunization Days. It is recommended that this pace of NIDs/SNIDs be maintained and the target of Polio-free Pakistan be achieved accordingly.
Mobility of Staff: Operational vehicles be supplied to the supervisory and field staff, and ensuring their proper use. Vaccinators generally have bicycles and each vaccinator has to look after one Union Council. Most of the time, it is difficult for them to cover all that area with a bicycle. They should be made more mobile, as is being done in one of the Tehsils in Rawalpindi division.
Full report in PDF
PDF files require Acrobat Reader.