We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Pakistan: Evaluation of Young Champions Initiative for Girls’ Education



Author: Aslam Aman

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System".   Within this system, an external independent company reviews and rates all evaluation reports.  Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it.  You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

The Young Champions Initiative (YCI) is a flagship initiative under the South Asia United Nations Girls’ Education Initiative  (UNGEI, www.ungei.org) which is a partnership of organizations committed to narrowing the gender gap in primary and secondary education. YCI also seeks to ensure that, by 2015, all children complete primary schooling, with girls and boys having equal access to free, quality education. UNGEI’s work is driven by Millennium Development Goals 2 and 3. YCI involves developing young men and women (aged 11 to 18) as role models to promote education for girls and boys. The YCI initiative was particularly important for Pakistan because the country lags behind in achieving the target for Universal Primary Education (UPE).

The YCI model was inspired by the success of similar models in addressing different social issues around the world.  The model is premised on clear evidence that young people are more receptive to the knowledge they receive from their peers and celebrities.
The implementation of YCI in Pakistan predates its launch by UNGEI in Pakistan in 2010. The pilot YCI was initiated in 2007 by a group of UNICEF staff members who had attended a Training of Trainers (ToT) course on YCI in Nepal.  At that time YCI was piloted in KP and Punjab. In KP more than 450 Scout Master Trainers from the KP Boy Scouts Association were trained. They then trained 5,700 Scouts  drawn from 300 Government schools in 16 districts as Young Champions. UNICEF facilitated the Scouts to undertake advocacy campaigns and to engage in co-curricular activities such as sports competitions, essay contests and Arts and Science Exhibitions. These Scouts also took part in relief activities for Internally Displaced People in KP in 2009.

(Read more in the main report annexed herewith)

Purpose/Objective:

Purpose:
The purpose of the evaluation, as stated in the TOR, was “to establish the extent to which the PGEI supported Young Champions initiative has achieved and continues achieving its intended outputs and outcomes at the country level generally and for youth particularly, and suggest ways to build on and strengthen existing initiatives.”  

The findings of the evaluation are expected to inform future programming of UNICEF, the implementing partners and the Boys Scouts Associations and Girl Guides Associations in the Provinces, FATA, AJK and GB.  It is hoped that the findings will inform the policy and practices of government Education Departments in those provinces.

Objectives of the Evaluation:
• Determine the relevance, effectiveness, efficiency,  early outcomes and sustainability of YCI (as per UNEG standards).
• Provide all parties/stakeholders with an understanding of the current achievements, successes, lessons learnt and areas for future improvements through actionable recommendations.
• Assess the implementing partners’ capacities, monitoring mechanisms and orientations on applying a result-based approach as well as ability to documenting results;
• Determine early outcomes that may have been achieved so far or the potential for the achievement of outcomes.
• Take stock of existing programs, projects, activities and partnerships under the Young Champions Initiative and recommend a strategy for future programming by defining new partnerships and potential areas of collaboration.

(Read more in the main report annexed herewith)

Methodology:

As advised in the TOR the main thrust of the evaluation methodology was qualitative.

• Desk review:  This included a review of key project documents provided by the UNICEF office, those obtained during the data collection process and other relevant literature (List of documents reviewed in Annex 10).

• Stakeholders’ workshop: This was held in Lahore and attended by representatives of partner organizations in Punjab including PBSA, PGGA, UNICEF, UNGEI, and SED.  The purpose was to inform the stakeholders about the evaluation process and to seek critical feedback about the proposed evaluation methodology.

• Key informant interviews: Interviews were held with UNICEF officials at the head office and in provincial offices, with key staff members of partner organizations, PGGA and PBSA officials, school heads, teachers, parents, community members and Young Champions.

• Group interviews: Interviews were held with Young Champions, parents, PBSA and PGGA representatives and teachers.

• School visits for verification of enrollments:

• Telephonic interviews:  In all, 23 telephonic interviews were held, of which 13 were with Young Champions, eight were teachers trained by Help in Need, one was a politician and one was a PGGA trainer in Kasur (Punjab) 

3.4 Data Analysis: Data was analyzed using a combination of two approaches for qualitative data analysis: Content Analysis and Analytical Induction.

(Read more in the main report annexed herewith)

Findings and Conclusions:

YCI, with its focus on out of school children, particularly girls, is highly relevant to community needs, country priorities and UNICEF’s mandate.

YCI directly addresses a key UNICEF mandate - to provide basic education and gender equality. YCI is also aligned with UNICEF’s current country programme with its focus on bringing out of school girls and boys to school.

The Implementing Partners in Punjab claim that a total of 12,105 out-of-school children (5,832 girls and 6,273 boys) were enrolled in school as a result of the YCI. The implementing partners in Punjab were considerably effective in enrolling out of school children, but claims regarding the total number of enrollments need to be treated with caution. 

The cost per child enrolled in Balochistan was USD 36.73. In Punjab the cost per child enrolled in the school was USD 24.71 for the Jahandad Society for Community Development and USD 19.10 for the Hayat Foundation.  A comparison of cost per unit for partners in Balochistan and partners in Punjab is not realistic, given the difference in context and the nature of the projects. No figures are available for FATA.

YCI was implemented as isolated projects in Punjab, Balochistan, KP and FATA with few vertical linkages with the country office and no horizontal linkages among the different projects.

The programme was not guided by a systematic, rigorous inquiry into the causes of children’s exclusion from school. YCI addressed only one of the many factors that keep children out of school. A UNICEF study on out of school children is available but the programme did not take stock of the findings. 

Limited monitoring or the absence of monitoring was a notable weakness of the projects implemented. The reporting mechanism was weak, with a resulting lack of progress reports and analysis of impediments. Considerable institutional memory has been lost...
 
(Read more in the main report annexed herewith)

Recommendations:

•While there is great merit in engaging Young Champions in enrolling out of school children, YCI should also be used as a broader framework to engage existing youth and adolescent groups across different programmes of UNICEF.
•With the recommendations outlined in Section 7 in mind, YCI should also be implemented across additional regions of Pakistan, including Sindh, Gilgit-Baltistan and Azad Jammu and Kashmir (AJK).  
•The logic framework for adolescent related programming should be developed at the UNICEF country office level.  With the support of experienced managerial support, the framework should include:  a well-developed theory of change; a strong GE and HR strategy that is integral to the framework of the programme; a stakeholders’ analysis; a strong monitoring system, including a well-developed Results Framework with detailed process, output and outcome indicators; a sufficient budget, a rigorous implementation plan and process and a sophisticated communications (i.e. reporting and analysis) plan.
•The process of identifying and selecting partner organizations should be more rigorous, combining objective criteria regarding experience and abilities, with capacity-building activities to fill identified gaps.
•YCI should be integrated with the UPE drives at the district and school levels, rather than the implementing partners and Young Champions working in parallel with existing enrollment drives in the schools and the District Education Departments.
•To maintain contact with Scouts or Girl Guides after they graduate, UNICEF should try to engage “open scouts”.  A database of such people should be maintained by implementing partners and PBSA and PGGA. It is acknowledged that gender-based limitations make this more feasible for male than female graduates.

(Read more in the main report annexed herewith)

Lessons Learned:

It is evident that the YCI programme suffered from a series of structural problems, each of which contributed to the issues and limitations identified in this evaluation report. The problems originated in two closely-related factors: one is insufficient knowledge (or application of knowledge) of the social and economic contexts of Pakistan as these relate to education, gender and disability. There is also little evidence of knowledge of these issues (e.g. poverty, distance from schools, the social demands for gender-segregation in schools, the limited space for growth in classrooms, the limited number of teachers) in the regions and Districts where the programme was implemented. There is no evidence of any lessons learned from the pilot projects being taken into account for the Pakistan project. Nor were large-scale Projects such as the Punjab Education Sector Reform Programme or the British Council implemented Active Citizenship Programme consulted for ‘lessons learned’. There is also no evidence of a social survey/scoping study being conducted before the Pakistan programme was designed.

Although there was reliance on country-wide education data, apparently there was no District-specific understanding of the number of out-of-school children, or of the specific factors in each District that keep children out of school. Incorporating knowledge of these local factors might have resulted in a more effective and efficient programme that UNICEF and others could learn from. This point will be especially salient when expansion to other parts of the country is being planned. It is also the case that locally ‘notable’ people can become another kind of Champion for this kind of project.

The second foundational problem was the lack of three critical planning tools: a realistic Theory of Change, an overarching Logic Framework and a tightly-integrated implementation plan.

(Read more in the main report annexed herewith)



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Pakistan

Region:
ROSA

Theme:
Education - Girls

Type:
Evaluation

Language:
English

New enhanced search