We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2015 Pakistan: Mid-term Evaluation of IKEAF funded programme on Promoting Child Rights in Cotton Farming Areas of Pakistan



Author: AAN Associates

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

The IKEA Foundation funded programme, ‘Promoting Child Rights in Cotton Farming Areas of Pakistan (CRCF)’, is a multi-year (2011–2017) multi-sector programme implemented in two phases to identify and address areas underpinning the protective environment for enabling access to children’s rights. It focuses on the rights to survival, education, and protection. Delivery approaches include building government capacities (especially at provincial and district levels), communities and families. It features a multi-sector integrated model, encompassing complementary interventions around community economic and human development, social protection benefits and services, legislative and policy reform, and strengthening public administration and governance. At strategic level, programme interventions can be grouped into:

  • Public sector capacity development for policy reviews and revisions, capacity building, systems  development/strengthening, to create protective environment
  • Community mobilisation and awareness to create multiple and inclusive community representative groups, education, and training, enabling them to take organized action and engage with public authorities. This sensitizes and educate communities to demand services, and mobilize support.

The original plan envisaged implementing activities in 1,625 villages across eight districts in Punjab, Sindh, and Balochistan provinces. This was reduced for operational reasons to six districts, but maintaining the number of beneficiary villages. Phase-I activities were implemented in four districts: Lasbela (Balochistan), Bahawalpur and Rahim Yar Khan (Punjab), and Ghotki (Sindh). In Phase II, initiated in 2014, activities expanded to Khairpur (Sindh) and Rajanpur (Punjab). The programme was designed to benefit a range of beneficiaries but focused on supporting women, children and adolescents, as direct beneficiaries. It targets over 1.2 million children as direct and indirect beneficiaries.

Purpose/Objective:

UNICEF commissioned this Mid-term Evaluation to get an objective assessment of programme performance, delivery approaches, partnerships arrangements, key learning and suggestions for the future. The evaluators designed and carried out an enabling evaluation to offer the programme partners with unbiased and considered views as to programme achievements, challenges, and lessons learned. This being a forward-looking evaluation it carries programmatic and operational recommendations to guide the course correction for the remaining programme period.

Methodology:

The Descriptive Evaluation Design was used for its focus on assessing programme approaches.
These are normally used in settings to ascertain if a programme is operating as planned, gather feedback (from beneficiaries and other stakeholders), determine whether it is producing expected outcomes and outputs, and clarify programme processes, goals and objectives.
The evaluators used a mixed-method approach, utilizing both secondary and primary information (qualitative only). Besides key programme documents, the evaluators consulted national and international research, databases, and media reports. The primary information was generated using qualitative tools including 94 Key Informant Interviews, 31 Focus Group Discussions field (three districts).
The respondents included public officials (national, provincial and district levels), UNICEF staff, IP managers and field staff, and beneficiary groups of men, women, adolescent and children. Purposive sampling was used to select stakeholders and the number of times to consult each of them. The evaluation features design and application of innovative methods whereby the evaluators developed and applied child friendly methods and tools (interactive) to seek information from children and adolescent. This enable productive engagement with children to help them reflect, share experiences and suggest ideas for improvement.
Quality assurance processes were developed and implemented. The evaluators took pre-emptive and responsive mitigation measures. The ODEC/DAC framework (efficiency, effectiveness, relevance, and sustainability) was used as an analytical framework. Additional findings and analysis with respect to UNICEF programming principles and priorities such as human rights based programming (HRBA), equity, gender equality, environment sustainability and disaster risk reduction, was embedded into the OECD-DAC criteria and also presented separately.

Findings and Conclusions:

The programme was assessed to be relevant in focusing on priority community needs as substantiated by poor baseline development indicators. The communities (women and children especially) further validated the assertion that the programme prioritized and addressed basic needs. The thematic focus on social protection (poverty reduction), healthcare, WASH, education and child protection was relevant for these districts.
Programme delivery approaches underpinning public sector capacity development was highly relevant in triggering government-led sustainable change. Design and delivery conformed to UNICEF HRBA principles and gender equality.
Although implementation demonstrated an equity-based pro-poor focus, selection of project areas was done by district governments, but without specific equity-driven criteria. Districts selected, though agreed with IKEAF, were not necessarily the poorest or largest cotton-farming districts. Lasbela was selected for being relatively secure, with a forthcoming district government while Bahawalnagar was dropped despite being among the top three cotton-producing districts.
The programme design was incongruent with the principles of Results Based Management (RBM).
It was aligned to multiple national development policies, strategies and plans in target sectors. The programme was approved before the launch of the UNICEF Social Protection Strategy, but design was related to the strategy (social transfers; access to services; social support/care services; and legislative and policy reform). Programme objectives were coherent with those of other development partners, and with IKEAF corporate social responsibility approaches.
Linkages with government were appropriate and adequate, with demonstrated application of rights-based approaches, and the programme engaged with duty bearers at all levels (especially district). Successes included mainstreaming non-formal education, allocation of PKR400 million for PATS in Punjab, among others.

Recommendations:

  • Using the Results Based Management Approach revisit the definition of outputs, intermediate results and outcomes, and reformulate the Theory of Change.
  • Use the proposed indicators (in five domains: child survival, education, WASH, child protection, shelter and information) in the revision to help remain compliant with the Social Protection strategy and to data supplied for the MDGs (not SDGs).
  • Adjust targets and budgets accordingly, ensuring consensus but maintaining flexibility to  produce mutually acceptable centres of responsibility and uniform ownership. Document rationale for excluded districts in Phase II.
  • Define criteria for marginalization and enable district governments to internalize and select project areas accordingly.
  • Introduce environment and DRR into training and for communities, public stakeholders and civil society partners.
  • Reformulate a comprehensive and integrated BCC approach, based on district reviews and, if appropriate, leveraging ICT tools.
  • Deliberate on the gaps in management, reporting, data generation and management, development accounting issues; and bring uniformity in reporting.
  • Revisit implementation approaches to improve integration of activities.
  • Revisit monitoring to develop and implement an integrated monitoring framework/system which is process oriented, measures rate and efficiency of progress and responds to TPFM reports documents response.
  • Review and revise data management standards to prevent the possibility of double counting and train partners accordingly. Revalidate existing information where possible.
  • Improve data and knowledge management as per report recommendations.
  • Create standard cloud-based formats for all data collection requirements.
  • Review existing PCAs to implement mandatory pre-qualifications for sub-agreement partners to reduce the risk of indirectly engaging non-compliant partners.

Lessons Learned:

  • Baselines only remain useable if adjusted for drastic changes due to disaster/ emergency situations. Three district surveys will need to be adjusted following the 2015 floods.
  • Ambitious targets without adequate resources may result in poor outcomes unless targets and results are synchronized early in the start-up stage. When revising targets and approaches, the level of funding may be made commensurate with IKEAF’s high expectations and matching current outputs; or the geographical area may be constrained to only the most cotton-producing districts (implying discontinuation in Balochistan). Maintaining the same set and level of outputs may affect impacts and sustainability.
  • Sustained integration of programme components can only be achieved if there is consensus on common interventions and district-specific customizations;  communication, dissemination and training must be centrally developed and uniformly applied; during and post-emergency situations, the integration of development approaches is useful.
  • The lack of a Theory of Change hindered full understanding of the programme and in leveraging synergies and learning. Outsourced field monitoring may be effective if a standard is developed and monitoring is synchronized. Monitoring information must be reviewed and strategic decisions taken to keep potential deviations to the minimum, and circumvent unrecoverable failures. Effective (timely and accurate) information-sharing and dissemination leads to proper understanding of field staff about programme design, Theory of Change, processes and quality assurance, without which there are challenges in delivering better results. Investing in knowledge management can inform and improve programme design.
  • Social mobilization is a useful strategy for modifying community behaviour. Involving top management in continuously engaging with high-ranking public authorities can help get required support during the entire lifecycle and ensure sustainability.


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2015

Country:
Pakistan

Region:
ROSA

Theme:
Child Protection - Child Labor

Type:
Evaluation

Partners:
Government offices in Sindh and Punjab provinces

Language:
English

New enhanced search