Evaluation database

Evaluation report

2011 Viet Nam: Provincial Child Friendly Programme (2006-2011) Evaluation



Author: Shanks, E., Dao, N. N., Duong, Q. H.

Executive summary

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 3’ of the report.”


Background:

1. The Provincial Child Friendly Program (PCFP) was introduced under the UNICEF Country Program in Viet Nam in the period from 2006 to 2010, with a one-year extension to 2011. The expected results of the program are: (1) the ‘realization of the rights of women and children’ through (2) ‘strengthened sub-national capacity for realizing these rights’, (3) the ‘development of replicable models of integrated programming and convergent services for women and children’ and (4) the ‘development and testing of national and provincial policies and standards’. Under the One UN Plan (2006-2010), the PCFP is positioned under Outcome 2: quality social and protection services are universally available to all Vietnamese people (Output 2.23), and Outcome 4: the principles of accountability, transparency, participation and rule of law are integrated into Viet Nam’s representative, administrative, judicial and legal systems (Output 4.3). According to the latter, under the One UN Initiative in Viet Nam, the PCFP is positioned under the Program Coordination Group (PCG) on Governance.

2. The PCFP has two broad components. The first national component deals with capacity building and knowledge management (the Capacity Building and M&E Project). This project is led by the Ministry of Planning and Investment (MPI) in collaboration with Ministry of Labour, Invalids and Social Affairs (MOLISA). The second sub-national component consists of six Provincial Child Friendly Projects in Dong Thap, Ninh Thuan, Dien Bien, Kon Tum and An Giang provinces and in Ho Chi Minh City. These projects are led by Provincial Peoples Committees (PPC), through a Province Steering Committee, with coordination by a Project Management Board (PMB) located under the Department of Planning and Investment (DPI) in each province and the Department of Labour, Invalids and Social Affairs in Ho Chi Minh City. Representation on the PMB includes other implementing agencies including the Department of Health (DOH), the Department of Labour, Invalids and Social Affairs (DOLISA), the Department of Education and Training (DOET), the Province Centre for Rural Water Supply and Sanitation (PCERWASS). In Kon Tum, the UNICEF-PCFP works with UNDP and UNFPA to demonstrate joint UN support at sub-national level.

3. In several important respects, the PCFP signified a strategic re-orientation of UNICEF Country Program in Viet Nam. Whereas UNICEF support was previously channeled primarily through sector ministries and programs at national level (with funds managed and disbursed at central level), the PCFP has worked directly at sub-national level (including the direct transfer of UNICEF funds to the provincial authorities). Combined with this, the PCFP has sought to enhance the integration of UNICEF support with the annual and 5-year Socio Economic Development Plan (SEDP) and sector plans at province level. This was justified in light of the Government’s commitment to continuing decentralization with more autonomy for sub-national government in the preparation, implementation and oversight of their SEDPs. The PCFP recognizes the importance of partnership with the provinces in delivering results for women and children, and capacity building of provincial government partners, including both executive agencies and elected legislative bodies, is one key strategy of the program.

4. This was also a new implementation modality for UNICEF, around which several other adjustments were made to enhance the focus of the Country Program. Sector specific programs in Education, Child Protection, and Child Survival and Development have been maintained alongside the PCFP. Some sector program activities at sub-national level have continued (both within PCFP provinces and in other geographical areas); while funding for all UNICEF supported activities in the selected PCFP provinces has been channeled through the PCFP fund-flow and disbursement mechanism. A PCFP Section was established in the UNICEF Country Office to support the new program. Initially this section consisted only of program coordination staff (with technical support coming from program specialists from other sections), while from 2008 the PCFP Section has included technical specialists to give more intensive support to implementation of activities in the PCFP provinces.

5. There has been a phased introduction of the sub-national projects, with the most recent addition being HCM City which began operations in the second half of 2010. Each project covers from 2 to 4 districts, plus 10 to 14 rural communes or urban wards, which represents between 24% and 44% of the total number of communes and wards in the PCFP districts.

6. In the original Program Plan of Operations (PPO) it was intended there would be 10 sub-national projects; and together with the CB-M&E Project and UNICEF management and administration, the total budget estimate was set at US$ 21.709 million. When the program began, it was found that an extended period was needed to prepare the Detailed Plan of Operations (DPO) for each project which slowed start-up to some extent. Accordingly, during the Mid-Term Review of the Country Program in 2007-08, it was agreed to reduce the number of provinces to six. Total program costs have therefore necessarily been reduced from the original estimates. As of the end of 2010, disbursement reached US$ 12.364 million. Three provinces have exceeded the expected disbursement given in the DPO: Dien Bien (123%), Ninh Thuan (113.5%) and An Giang (102%). Dong Thap has disbursed 94% and the CB-M&E Project has disbursed 85% of the planned project amount as given in the DPO.

Purpose/Objective:

7. According to the Terms of reference, the objectives of this evaluation are as follows: (i) to evaluate various aspects of the program in terms of its relevance, effectiveness, impact, efficiency and sustainability; (ii) to analyze the effectiveness of the overall strategy and approaches of the program both at the national and sub-national level on the fulfillment of children’s rights in provinces including the geographical focus and the mix of provinces from different regions; and (iii) to identify lessons learnt on program design and implementation and to provide recommendations for UNICEF’s future engagement in the next country program cycle, with respect to the overall strategy and approaches of the program both at national and sub-national level, and geographical focus in the new program cycle taking into account the current coverage, and presence of other UN agencies and donors.

Methodology:
8. The evaluation methodology includes several main components and qualitative and quantitative sources of information, as follows:
• Results framework analysis. The evaluation is structured around the four main expected Results or Outcomes at program level. At the outset of the assignment, a matrix of evaluation criteria was drawn-up which was used to define the detailed evaluation methods, questions and indicators. The matrix was also used to compile the main results and achievements of the program in relation to each Outcome.
• Province level evaluation workshops. Evaluation workshops were held in each province, including separate sessions with: (i) staff from the main province and district level implementing agencies; (ii) members of the PMB; (iii) members of the Core Planning Group involved in the SEDP and sector planning work; and (iv) representatives from 3 to 6 communes in each province. These workshops included a combination of Focus Group Discussion around prepared evaluation questions; SWOT Analysis exercises on specific aspects of the program; and a Participant Questionnaire for province and district staff and Core Planning Group members (but not including commune representatives).
• Commune case studies. Commune visits and case studies were made for three communes in Dien Bien, Ninh Thuan and Dong Thap provinces. These communes were selected on the basis of being involved in the program for an extended period, hence where program activities would be comparatively well advanced. The main purpose of the commune case studies has been to assess the cost-effectiveness and impacts of integrated and convergent services and approaches to working with children’s issues at community level.
9. The evaluation methodology has sought to actively engage a large number of participants from all stakeholder groups involved in the program. In total, around 400 people were involved. Around 300 people participated in the province evaluation workshops, while 145 province and district staff completed the questionnaire, which represents a valid sample of opinions from people working directly on the program at this level.

Findings and Conclusions:

10. Relevance. Across all aspects, the program rates highly in terms of its relevance. Detailed planning of the individual projects began in 2006, starting with Dong Thap and the CB-M&E Project which became operational by the end of 2006 and early 2007. It has taken about one-year to jointly prepare the DPO for each project. This was an intensive process. Even so, the Evaluation Team believes that, given the multi-sector approach and complexity of the PCFP, this was a realistic time-frame that was necessary to build up understanding and consensus amongst the province authorities and implementing agencies. In comparison to other donor-supported projects, a one-year preparation and lead-in period is also not excessive.

11. Throughout the preparation process and subsequent project implementation, UNICEF has been active in engaging with the Province People’s Committees. This has resulted in strong local ownership and understanding of the program and close coordination by the local government authorities. This has been essential and fundamental to the success of the program, especially in introducing the SEDP and sector planning reforms.

12. All stakeholders endorse the relevance of strategic approach to decentralization and integrated programming for women and children. The management arrangements, whereby PCFP is coordinated by MPI at national level and DPI at provincial level provides the necessary linkages to the planning system and coordination functions. The modality of combining a central project (responsible for capacity building and M&E) with province projects (responsible for detailed planning and implementation) is also clearly appropriate to the administrative context of decentralization in Viet Nam. The PCFP is timely and relevant in the emphasis given to strengthening cross-sector coordination and collaboration. This is not only because institutional responsibilities for children’s issues by nature span multiple sectors. At this point in time, it is widely recognized that cross-sector coordination is a weak aspect of the administrative system in Viet Nam, which is to some extent re-enforced by the strong vertical alignment and programmatic structure of the planning and budgeting system.

13. In essence, the PCFP is a programmatic mechanism which is serving as a vehicle to more intensively test and apply methods and technical interventions that have been under development and supported by the sector-specific programs of UNICEF for an extended period. Some components of the program, such as the child/adolescent friendly school environment, various maternal and child health and nutrition interventions and child injury prevention, are carried out in all provinces. At the same time, through the program preparation process in each province, locally-specific issues and priorities for women and children were identified. While the technical interventions and models of convergent services of the program are for the most part highly relevant to local needs and priorities, the key issues relate to their effectiveness in terms of lesson-learning, replication and sustainability.

14. The province selection is considered by the Evaluation Team to be sufficiently representative of different regions of the country and of different socio-economic contexts (i.e. coverage of emerging issues). This includes differing ethnic minority contexts (in the Mekong Delta, Central Highlands, South Central Coast and Northern Uplands); and migrant groups in areas of both out-migration (Mekong Delta) and in-migration (HCM City). The district and commune selection is also appropriate in covering the range of demographic situations, ethnic groups, poverty contexts and disadvantaged areas in each province.

Recommendations:

Recommendation 1: There is strong justification for continuing the Provincial Child Friendly Program into a second phase from 2012 to 2016.
Recommendation 2: The scope and coverage of the program should be further adapted and focused on priority children’s issues and population groups in each locality.
Recommendation 3: Planning reforms should focus on the strategic orientation and content of the SEDP and sector plans, their implementation, monitoring and evaluation.
Recommendation 4: Improved cross-sector coordination and integrated programming requires more refined adaptation to the decentralization context.
Recommendation 5: Increase the focus of the program on child protection, children’s participation, and cross-cutting child care and protection issues.
Recommendation 6: Designing and introducing strategies for cost-effective replication and scaling-up should be an integral part of the program itself.
Recommendation 7: Enhance technical expertise within UNICEF and support provided by the program on public financial management aspects
Recommendation 8: Scale-up the community-based child care and protection activities, alongside introduction of the social work system and social protection policies.
Recommendation 9: Focus improvements in service delivery and strengthening cross-sector coordination and collaboration through the IBCC work.
Recommendation 10: Extend the approach to public consultations on the SEDP and sector plans, in association with executive agencies and legislative bodies.
Recommendation 11: Clarify the program results framework and strengthen the M&E, lesson-learning and documentation methods and responsibilities.
Recommendation 12: A greater level of connectivity is required between the sub-national projects and the other programs within UNICEF and at national level.


Lessons Learned (Optional):

23. Institutional lesson-learning. The PCFP Results Framework articulates a clear relationship that lies at the heart of the rationale and strategic approach of the program, as follows:

Development of replicable models and testing of national and provincial standards…
That are properly assessed, evaluated, documented and disseminated…
And used as a basis for scaling-up and replication in other parts of the province and in other provinces.

These linkages suggest that ‘pilot activities’ need be guided as a learning process. This may include ‘learning-by-doing’ during implementation (including process documentation), with clearly identified outputs to support replication (e.g. structured evaluation, guidelines and training materials, policy briefings, or recommendations for revised legislation). It is important that the pilot activity achieves a satisfactory result (this may be called ‘internal effectiveness’); but the successful output of an individual pilot activity is perhaps less important than testing the process of replication itself (this may be called ‘external effectiveness’). To influence policy decisions, this needs to be an iterative process between different ‘levels-of-learning’ and may need a staged process of testing and expansion.

24. In some areas of activity, the program has introduced a structured approach to institutional lesson-learning. One example is in developing and introducing the standards for Communes Fit for Children; these guidelines were developed by MOLISA with support from the CB-M&E Project, and MOLISA has worked with several provinces to introduce the standards. Another example is piloting the guidelines for Communes Safe for Children according to Ministry of Health guidelines. Lesson-learning and documentation from the field, together with feedback to national policies and guidelines, has also taken place around Community Led Total Sanitation and M&E for rural water supply and sanitation.

25. But for some other areas of activity, lesson-learning and documentation has not been done systematically or fully. There appear to be several reasons for this. Firstly, it appears that in some cases the thinking of provincial staff implementing pilot activities has been dominated by achieving the tangible output, in accordance with a specified implementation schedule; whereas the depth of lesson-learning, not the speed of implementation, should be the criterion of success. Secondly, the PCFP and other UNICEF sector programs contain a multiplicity of different activities going on at the same time, such that it appears difficult to fully review and document all of them. Thirdly, the responsibilities for lesson-learning and documentation (between the province projects, the CB-M&E Project, UNICEF PCFP Section, other UNICEF sections, and/or other ministries) are not fully clear or established. Lastly, insufficient attention has been given to the appropriate or best forms of documentation, especially for distribution to a wider pool of policy-makers.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search