We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2018 Pakistan: End of Programme Evaluation on Promoting Child Rights in Cotton Farming Areas (CRCFA) Programme of Pakistan



Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.

Background:

(CRCFA) was an IKEA Foundation (IKEAF) funded programme implemented by UNICEF Pakistan Country Office from 2011-2017. The aim of the programme was to provide backstopping support to the governments at the provincial, district levels, as well as the communities, families and other stakeholders in order to strengthen the protective environment for children for the enhanced realization of child rights in the targeted areas. The programme was implemented in districts of Lasbela (Balochistan), Khairpur, Ghotki (Sindh), Bahawalpur and Rahim Yar Khan and Rajanpur (Punjab) in two phases.
The primary focus of the CRCFA programme was to strengthen the realization of a number of child rights, with a focus on child vulnerabilities in six of Pakistan’s cotton-producing districts. For this purpose, the programme was designed using an integrated approach focusing on cross-cutting themes of social protection, child protection, education, health and water, sanitation and hygiene (WASH). This integrated multi-sectoral design was proposed in response to multi-faceted issues and factors which affect child rights and child protection services.

Purpose/Objective:

The end-of-programme evaluation was commissioned to evaluate the performance, key achievements, challenges and lessons learnt in order to assess how far the programme achieved its intended objectives based on the criteria developed by the Organization for Economic Development’s Development Assistance Committee (OECD-DAC) criteria. The evaluators designed and carried out an unbiased and participatory evaluation approach where all the relevant stakeholders of the programme were consulted at each stage of the evaluation process (design, implementation, analysis and recommendations).
The key objectives were defined as follows:

  • An assessment of programme achievements at the end of term, vis-à-vis programme objectives and targets;
  • An overall assessment of the performance of the programme based on relevance, effectiveness, efficiency and sustainability of the CRCFA initiative; and
  • Identification of lessons learnt, formulation of conclusions and recommendations on improvement measures for consideration in any future programming, with a view to enhancing results, where applicable, including through an equity lens.

Methodology:

The evaluation utilized creative ways to combine different evaluation frameworks, tools and techniques including quantitative and qualitative components. The evaluation was a pre/post evaluation design with a mixed method (MM) approach that adopted a three-pronged strategy: a) desk review, b) primary qualitative and household survey, and c) secondary quantitative components. The MM approach was meaningful to explain and interpret phenomenon, address questions and theoretical perspective at different levels. This approach enabled the evaluation team to triangulate he findings of qualitative assessment and he household survey with the secondary data.
The data was collected through 63 in-depth interviews, two meetings, 24 focus group discussions and 1,134 household surveys. Following the UNICEF Procedure for Ethical Standards, the research team maintained professional standards, confidentiality of documents obtained, impartiality and transparency during the research process.

Findings and conclusions:

Relevance: The CRCFA programme is relevant in terms of focusing on the community needs. All the components (child protection, WASH, health, education) were found to be priority needs of the community based on the interaction of the evaluators with the targeted communities.
The programme strategies were very much relevant to the existing national priorities and policies on applicable child rights and the programme worked with the government in all the components to leverage its resources to maximize the achievements.
Effectiveness: Majority of the outcomes were met as the utilization of public services by the targeted communities evidently and significantly increased as a result of the programme, particularly in Punjab followed by Sindh and then Balochistan.
Efficiency: The delays in transfer of funds created challenges in implementation. Engagement of multiple partners also created problems in implementation.
Sustainability: The programme lacked a clearly defined implementable exit strategy, which created challenges with reference to the sustainability of the interventions. However, certain interventions were greatly appreciated by the government and replicated, which enhances sustainability of the impact.
Cross-Cutting Areas (Gender, Equity, Human Rights, Disaster Risk Reduction): Limited attention being paid by thematic units in integrating cross-cutting priorities in terms of design, implementation, monitoring and evaluation.

Recommendations:

Based on the successes and weaknesses of the programme, three levels; UNICEF Country Office, UNICEF programmes and government level are made:

  • Selection of districts should be based on criteria laid out in the initial proposal. Selection of districts/ union councils and targeted villages should be equity-based.
  • A Theory of Change should be formulated at the beginning of all future programmes.
  • Programme design and budgeting should be gender-responsive.
  • Cross-cutting themes should be integrated into the programme design.
  • Rather than engaging multiple partners, it is better to narrow down the pool and select a few partners after a thorough capacity assessment.
  • For future programmes it is important that UNICEF ensures selection of IPs in districts who have the potential to work on child protection. Otherwise, UNICEF should bring in experts/ consultants who can build capacity of communities on child protection.
  • In conservative areas like Balochistan, members of the jirgas should be sensitized towards child protection and their capacities should be built on this topic.
  • The need for any aspect of a programme should be assessed separately for each community.
  • The WASH component should also build in safe disposal of waste in its future programmes as most of communities pointed out garbage dumping is a major issue in their areas.
  • Strategy of feeder schools was also very successful especially for female and disabled students. This strategy should be used in other programmes related to girl education.
  • The non-formal based education model was a great success and should be replicated in other programmes.
  • The government should replicate the CHW model in areas that are not covered by the LHW programme. 

Lessons Learned:

The collected and analyzed data revealed the following lessons learned:

  • Working with one implementation partner was conducive to delivering results and led to improved ownership so is recommended as a good strategy for future programmes. 
  • Subcontracting should be better planned and reflective of ground realities and UNICEF Provincial Offices should be taken on board as their feedback is critical to designing successful programmes.
  • The authority to finally select IPs should rest with the UNICEF Country Office but after consultative process with UNICEF Provincial Offices.
  • SRSO should have been directly reporting to UNICEF. The chain of communication was unnecessarily long.
  • BBSYDP deviated from the regular protocols and extended their beneficiary criteria to cater to CRCFA needs and included adolescents. This provides a window for future programmes where BBSYDP might add a certain age group which is not a part of their regular programme.

 

Please find the attached labelled as follows:

  • Evaluation Report - Report
  • GEROS Evaluation Review - Part 2
  • GEROS Feedback Summary - Part 3
  • Evaluation Management Response (EMR) - Part 4


Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year: 2018

Office/Country: Pakistan

Region: ROSA

Type: Evaluation

Theme: Child Protection

Language: English

Sequence #: 2018/002

New enhanced search