2015 Pakistan: End of project Evaluation - GAVI CSO project
Author: Dr. Ayesha Khan
End Project Evaluation of GAVI Supported CSOs Project to Strengthen CSO Involvement in Immunisation and Maternal-Child Health Services in Pakistan
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.
Pakistan is the sixth most populous country in the world with maternal-child health indicators that lag behind its regional neighbours. Pakistan’s public sector spending on health is approximately 1% of its GDP US $ 234 billion2 with the cost of immunisation activities (routine and campaigns by the EPI program) ranging from USD 47 -175 million per year. The goals of the EPI program are to achieve immunisation coverage of at least 80% across all districts of Pakistan against eight vaccine preventable diseases (DPT, measles, Hepatitis B, BCG, polio, and pneumococcal) and mothers with tetanus.
Despite high level government support Pakistan’s full immunisation coverage is low at 54% and polio, measles cases continue to emerge across the country. In 2008, as part of enhancing immunisation outcomes in hard to reach areas, Government of Pakistan requested GAVI support for pilot testing an innovative model of CSO-Government partnership to facilitate immunisation and MCH service delivery, demand creation, and engagement of CSOs to expedite policy and practice change.
GAVI UNICEF CSOs PROJECT
UNICEF-EPI -Government of Pakistan was successfully awarded a Type B US $ 7.5 million (after the initial US $ 100,000 mapping Type A grant) by GAVI in partnership with the core group of 14 CSOs to work on improving immunisation and MCH outcomes in 21 districts across all 4 provinces, AJK and GB. The GAVI CSOs project was managed and supervised by UNICEF and the CSO Monitoring Unit (M&E and coordination role) and implementation activities were carried out by the CSOs.
The main purpose of the End Project Evaluation is to assess the extent to which the CSOs activities and involvement has brought (or started bringing) about anticipated changes in immunisation and MCH outcomes in these hard to reach communities, examine which factors were critical in helping or hindering change and draw lessons for future programming. The findings of this evaluation will help key stakeholders in understanding potential models of service delivery, limitations of the current approach, and identify existing capacity gaps that need to be addressed for effective program implementation and policy change.
The primary audience for the evaluation are Government of Pakistan (EPI and Health Departments) and GAVI. The secondary audience includes CSOs, local partners, other donors/partners/ NGOs and community beneficiaries.
The main objectives of the Evaluation are:
- Review whether the project interventions were aligned with the objectives of the GAVI CSOs grant.
- Assess the extent to which this project has been able to strengthen and institutionalise meaningful participation of CSOs in national and provincial health strategy development and its implementation.
- Assess whether CSOs supported by GAVI have been able to advocate for increasing coverage of EPI vaccines to marginalised children of selected hard to reach intervention districts.
- Evaluate to what extent the resources/inputs (funds, experts, time, etc..) used by the programme were converted to results across genders and various socio-economic groups.
- Determine whether the CSOs have been able to build strong CSOs coalition and linkage with the provincial health department and advocacy for CSOs engagement at policy level.
- Identify lessons learned and formulate recommendations on how to best engage CSOs in future Health Systems Strengthening projects and enhance equitable results.
In June 2015, the CSO project reached its final conclusion and UNICEF Pakistan commissioned an end project evaluation to review project outcomes, lessons learned, and recommendations for future CSO-Government partnership model. The current summative evaluation has used a mixed method of 1) desk review and secondary data analysis (CSOs progress/monitoring reports) plus 2) the baseline report 2014, along with 3) primary interviews/FGDs with all relevant stakeholder categories – community, local LHW/vaccinators, district and provincial EPI/Health staff, Federal EPI, CSOs, CSO Monitoring unit, and UNICEF team to compile an overview of the findings, lessons learned and recommendations. As per UNICEF TORs the main thrust of the evaluation methodology was qualitative. The evaluation used desk review, data triangulation, interviews with key stakeholders, site visits and observations to formulate this evaluation report. Data was analyzed using content analysis and thematic analysis, using, OECD/Development Assistance Committee (DAC) criteria of Relevance, Effectiveness, Efficiency, and Sustainability with the addition of Gender Equity and Human Rights dimensions.
The evaluation methodology triangulates findings from CSOs progress reports, M&E reports, the Baseline study (2014), along with qualitative interviews from the key stakeholders of this program- Federal EPI and provincial counterparts, Ministry of Health Services, Coordination and Regulation, provincial departments of Health, district partners, UNICEF, WHO, the advisory group, CSOs, and most importantly the end beneficiaries i.e. the communities served (or not served) throughout the process. The evaluation has tried to cover management and outreach staff of all 14 CSOs and randomly selected one intervention area to have in-depth or focus group discussions with community beneficiaries and local partners.
Findings and Conclusions:
RELEVANCE: The CSOs project activities were well aligned with the original proposal and the national EPI priorities, UNICEF and GAVI mandates. Original proposal documents do not specify how the CSOs project design and selection was responsive to specific immunisation and MNCH needs of the selected areas.
EFFECTIVENESS AND PERFORMANCE: In nearly 90% CSOs achieved their stated targets and results. Considerable improvements (15% points increase in immunisation coverage), TT and HBV vaccination, rehabilitation of severely malnourished children, reduction in default and refusal rates, and increased SBA were some of the salient achievements.
EFFICIENCY: Accurate costing of the CSOs project is not possible due to absence of disaggregated UNICEF cost data prior to 2012.-13 With limited information provided to the evaluation team the CSOs project appears to be a moderately high cost model that may not be sustainable in its current form. CSOs provided immunisation services at approximately USD 19.2 per child immunized (without including the cost of immunisation supplies), and MNCH services are estimated to be in the range of USD 15-25. Costing of implementation models by areas and activities was not possible due to lack of data.
SUSTAINABILITY: A well-planned or clear exit strategy was not present in the CSO project design. Individual CSOs did have some have level of planning however for most CSOs project activities closed as of June 2015.
GENDER, EQUITY AND HUMAN RIGHTS: The primary beneficiaries of CSO project and activities were mainly women and children. Indirectly the project activities increased women’s access to information and services, thereby improved gender equity, and potentially empowerment of women through regular engagement at the community level. However the project design did not have a clear strategy for gender mainstreaming within the project to address socio-cultural inequities in health care access and outcomes.
(pls see report for details)
1. CSOs-Government Model and Project Design
1.1 Continue to support the CSOs-Government partnership model with modifications/changes as learned through six years of on-ground program implementation. Performance monitoring and work plans should be routine part of the initial project design.
1.2 M&E - should include CSOs driven frameworks and feedback from communities and district governments on how CSOs work is affecting district/sub-district indicators.
1.3 Efficiency – a proper costing exercise with disaggregation by geographical challenges and looking at inputs versus outputs and alternative scenarios should be undertaken.
2. Partner Selection and Capacity Building
2.1 Revisit and expand the partner mapping exercise with goal of expanding the pool of eligible CSOs and building social capital in small-medium size CSOs at the grass-roots in Pakistan.
3. Implementation and Results
3.1 Baseline measurements should be a mandatory part of program design and implementation.
3.2 Objective and measurable indicators of progress should be a routine part of work plans and progress reports.
4. Monitoring and Evaluation
4.1 Outcome and participatory monitoring to enable real time learning for the CSOs and course corrections in the project activities
5. Management Model and Coordination
5.1 Test out different models of fund management such as 1) province or district specific stand alone management by a large CSO(with 2 co-signatories), 2) CSO-Provincial agreements with supervision by district governments model, 3) CSO coalition with supervision by Government in a combined HSS model, and 4) Externally managed UNICEF or any other large donor agency model. The advantages and disadvantages of each model can be carefully reviewed in terms of costs, long term social capital, accountability and ownership of the key partners.
(pls see report for details)
- Project Design: Balance of Clarity and Flexibility the lesson learned is that CSOs project flexibility while a positive strength should have been coupled with self-derived but clearly articulated (and documented) results.
- Project Design: Course Corrections and Feedback
Course corrections are a necessary part of responsive and accountable programming. Having regular and specific issue directed feedback sessions between UNICEF and the CSOs and/or directly with the communities (without the presence of CSOs) would promote a more open environment of information sharing and program responsiveness. This aspect was lacking in the current CSO project.
- Mapping and Broadening CSOs Selection
Pakistan has a moderate sized and vibrant body of CSOs approx 3000+ or so in different categories and sectors. In future GAVI funding calls new CSOs with strong provincial and district roots should be actively recruited to expand the pool of eligible partners and to build lasting social capital.
- Focusing Activities and Results one District at a Time
The scale (i.e. absolute numbers) of project activities and beneficiaries influences outcomes on health and immunisation coverage. While CSOs were providing a comprehensive spectrum of activities within health and MCH, their geographic reach and target beneficiaries were a very small percentage of the UC or sub-district population to make a significant impact on the indicators. Enhancing the scale of CSOs reach and using a more focused approach would be beneficial in improving health indicators.
- Participatory Monitoring
Output based monitoring was deployed in the project throughout the project via the CSO Monitoring Unit and the GAVI missions. The lesson learned is to engage in actual participatory monitoring along with provincial partners (both within and outside of routine scheduled visits) and to think of monitoring as a change enabling incremental process.
Full report in PDF
PDF files require Acrobat Reader.