Author: Evaluation Office, UNICEF NYHQ
BackgroundThe Country Programme of Cooperation (CPC) 2001-2005 between the Royal Government of Cambodia (RGC) and UNICEF was the subject of a Country Programme Evaluation (CPE) in 2004. The UNICEF Country Office in Phnom Penh, upon a recommendation of the East Asia and Pacific Regional Office (EAPRO) and after consultation with the Ministry of Planning of the RGC, requested that the strategic directions for the new CPC (2006-2010) be informed by a CPE.
The CPE built on the Mid Term Review of the CPC conducted in October 2003. The exercise was coordinated by the Ministry of Planning of the RGC and the UNICEF Country Office in Phnom Penh with support from the Evaluation Office at New York Headquarters (NYHQ) and the East Asia and Pacific Regional Office (EAPRO).
The CPE Cambodia was a pilot case of the DFID-funded Project for CPE Methodology and Guidance Development, which is implemented by the Evaluation Office at New York Headquarters. The CPE Project serves UNICEF’s goal stated in the Medium-Term Strategic Plan (2002-2005), to make Country Programmes of Cooperation more effective in terms of focus, implementation strategies and coordination within UNDAF and with other development partners.
Purpose, scope, limitations, objectives and methodology
Purpose. The main purpose of the CPE Cambodia was to support the strategy development process for the next CPC (2006-2010). The new CPC should ideally align its objectives, targets and indicators with the Government’s development priorities and targets expressed in the planned National Strategic Development Plan (2006-2010), which will be based on the National Poverty Reduction Strategy (2003-2005) , the Socio-Economic Development Plan II (2001-2005), the Cambodia MDG Report (2003) as well as the Rectangular Strategy (2004).
For UNICEF, it is also important to assess progress made in Cambodia on the adoption of the Human Rights Based Approach to Programming (HRBAP) and Results Based Management (RBM) as well as the alignment of the CPC with organizational priorities defined in the Medium Term Strategic Plan (MTSP) and global goals until 2015 (World Fit for Children, the Millennium Declaration and Millennium Development Goals).
It was also decided that the CPE would focus on three strategic dimensions that received relatively little attention in the MTR of 2003 and that are usually addressed in CPEs: a) the alignment and contribution of the CPC to evolving national policy frameworks, especially as far as the progressive realisation of the rights of children and women are concerned; b) he partnerships and strategic alliances that need to be developed at all levels to mobilize an effective response to the challenges of realizing children’s and women’s rights; c) the implementation of the Human Rights Based Approach to Programming (HRBAP) of UNICEF and the application of Results Based Management (RBM) in the CPC.
The CPE has not given special attention to key dimensions related to efficiency, such as human resource planning and management and the supply function. There is, however, some information on and an assessment of the mobilization of human and financial resources by UNICEF and aspects related to the financial expenditure during the current CPC (2001-2005).
Objectives. The objectives of the CPE Cambodia stated in the TOR were to assess role and relevance of the CPC Cambodia, its design and focus, its niche and comparative advantage as well as its effectiveness and efficiency. An assessment was also to be made of the resource mobilization and the budget realization. Of particular interest were questions related to sustainability, replicability and mainstreaming of results. The evaluation was also to be useful beyond the Cambodian context to the extent that lessons were to be drawn from the experience for global CPE methodology and guidance development.
Methodology and evaluation process. Cambodia was the first pilot case in which the CPE followed an MTR. This scenario had the advantage that the CPE could build on results of the MTR conducted in 2003 and focus on strategic issues. The challenge was, however, to maintain stakeholder participation in the validation and triangulation of evaluation results with as little transactional cost as possible.
Key methods for the conduct of the evaluation involved: (a) a comprehensive desk review of external and internal documents, including past and recent studies, reviews and evaluations of projects and programmes; (b) an extensive round of interviews / focus group discussions with key stakeholders; (c) field visits to selected provinces, districts and communities; (d) an inception report and an annotated outline of the final report containing major lines of analysis as interim products of the CPE; (e) a participatory workshop during which main findings, conclusions and recommendations of the evaluation were discussed, and (f) a final CPE report. All information has to the extent possible been triangulated and validated.
The evaluation takes into consideration commonly agreed evaluation norms and standards (OECD/DAC and Evaluation Associations) to ensure utility (ownership by stakeholders), feasibility (political viability and cost-effectiveness), propriety (impartiality, respect of rights of stakeholders, cultural sensitivity) and accuracy (adequate information).
Factors that favourably affected the evaluation process were: (a) the existence of a comprehensive MTR report containing information on all projects and programmes; (b) the availability of all staff of the UNICEF Country Office for information and discussion; (c) the opportunity to consult with key external stakeholders and make field visits; and (d) continuous support from the UNICEF Representative and the Senior Programme Officer.
The evaluation did face a number of challenges, the most important of which were the absence of an updated Situation Analysis that made it necessary to do more research on the situation of children’s and women’s rights than originally planned; the weak design of the CPC (absence of a clear results framework with “smart” objectives and indicators) and the lack of a good performance monitoring system. As the evaluation cannot by itself compensate for these weaknesses, the presentation of main experiences and results necessarily reports on activities as well as some results and also remain illustrative rather than comprehensive.
Key findings and conclusionsRoles and relevance: As compared to other external aid mechanisms, the UNICEF supported CPC mobilizes a relatively small amount of external financial resources — an average of a little more than US$ 15 million per year between 2001 and 2003. Although the external resource mobilization is relatively small if compared with bilateral agencies or international financial institutions, the CPC exercises an influence in Cambodian society that exceeds the relative size of its financial contribution.
The CPC deals with a broad range of issues affecting children: child survival; early childhood care and development; formal and non-formal education for all ages; HIV-AIDS; as well as special protection issues. The CPC and its programme components play a multitude of roles at various levels (national, province, communities) and in different sectoral or thematic contexts, thereby making flexible use of niches and opportunities and developing a variety of comparative advantages. Through the CPC, UNICEF actively contributes to the formulation of national legislation, policies and strategies in all major social sectors, especially child protection, health and education.
The MPO includes improvement of child survival as a specific objective, and as a part of the rationale for five of the six programmes: Health and Nutrition, HIV/AIDS, Seth Koma, Education and Advocacy/Social Mobilization. The CPC as a whole addresses many of the major direct and indirect obstacles and challenges affecting child survival in Cambodia.
There is evidence that the CPC has addressed some of the specific issues raised by the Committee on the Rights of the Child. Examples include (a) a campaign for nation-wide civil registration; (b) awareness raising campaigns against trafficking and sexual exploitation; (c) a five-year policy framework, police training and the establishment of a special telephone hotline for abused children. These initiatives have lead to increased awareness in urban and rural Cambodia on the dangers of trafficking and sexual exploitation, and an increase in the numbers of reports and arrests of child sex crimes.
Design and focus. The Human Rights Based Approach to Programming (HRBAP) has a long tradition in the CPC Cambodia to the extent that a rights-based approach was developed at the village and community level as early as during the mid-1990s. Although the approach of the CASD / Seth Koma programmes was more successful in terms of good processes but less in terms of tangible results in the areas of health and nutrition (see effectiveness below), the design of the current CPC as laid down in the Master Plan of Operations (MPO) contains most of the elements of an adequate HRBAP. By design, the CPC aims at the realization of children’s rights in Cambodia, notably the right to survival, the right to development, the right to protection and right to participation. It also addresses in its goals and objectives the immediate, underlying and basic causes of the non-realization of children’s rights. One can also find most good programming practices. While there is some attention to gender disparities (especially in education), there is clearly much less focus on the challenge to promote gender equality and empower women.
The Country Programme of Cooperation did not adopt a results-based approach to programming (Results Based Management). A logical / results framework was not a feature of the MPO. The MPO only refers to four overall goals (safe pregnancies and deliveries; best possible start in life for infants, education and protection of school-age children, life skills and participation for adolescents) and to national objectives to which the CPC would make contributions. The MPO is not specific in defining the specific contribution of the CPC to these national objectives. Results at the outcome level are described in rather general and unspecific terms, e.g. “ownership / empowerment” and “knowledge and behaviour” at the community level or “social policies” and “national programmes” at the national level. There is no description of expected outputs. Since no logical approach was developed, there is also no information on vertical and horizontal relationships between outputs, outcomes and impact results.
Ambitions spelled out in the MPO as to focus, convergence and integration have not entirely materialized in the current CPC. The intention was that the CPC would be realized through six “synergistic components”, the description of which contained frequent references to overall goals (safe pregnancies and deliveries; best possible start in life for infants, education and protection of school-age children, life skills and participation for adolescents). While there are several examples of convergence and integration at the level of concrete project initiatives, there is still relatively limited convergence at the level of the various programme components.
Especially the Seth Koma Programme has so far not lived up to its potential to be an integrative force, as there has been a certain “disconnect” between Seth Koma and other programme activities at the decentralized level (especially related to health and nutrition). The re-orientation of Seth Koma resulting from the MTR of 2003 and the strategic partnership with the Seila programme increase its potential to overcome the relative isolation it had experienced in the past and to connect to mainstream policies and strategies of Deconcentration and decentralization of the RGC.
The Programme for Children in Need of Special Protection (CNSP) attracted increased interest and funding during the first half of the CPC and the programme consequently expanded in the area of prevention and early intervention without evaluating the piloted newly developed protection projects. In addition, the Programme had started to deal with a wide range of issues, e.g. alternative care, child trafficking, sexual exploitation and abuse and various aspects of legal protection of children (including birth registration) without adequate prioritisation or focus.
Effectiveness. The current CPC can claim reasonable attribution of many results in terms of processes and outcomes at all levels, i.e. villages and communities, provinces and nation-wide. What a CPC can hope to achieve, possibly in close cooperation with other agencies of the UN system, are outcomes in terms of good processes and behavioural and institutional changes among stakeholders, who are both rights-holders (children, families, communities, local government, civil society, NGOs etc) and duty-bearers (families, communities, local government, civil society, NGOs, national government, external aid agencies. Documentation of these outcomes requires specific, measurable, achievable and time-bound objectives and indicators (including baseline data), an adequate system of performance monitoring, reporting and evaluation and a clear understanding of the specificity and complementarity of the contributions of other stakeholders. The lack of these elements limits the chances to adequately assess the effectiveness of the CPC.
The Seth Koma Programme has been the subject of several studies and evaluations during the current CPC, which generated some significant findings as to its effectiveness. The most notable results of the Seth Koma could be found in terms of a strengthening of capacities of rights-holders and duty-bearers in supported villages to drive the development process. A more tangible result that could clearly be attributed to Seth Koma was improved access to water and latrines in the supported villages, although there had been no significant increase in latrine usage compared to control areas. Families in Seth Koma villages, however, did better than those in control villages with respect to health knowledge and care practices, as well as in coverage of immunization and micronutrient supplementation. Nevertheless, in terms of other tangible outcomes, e.g. improvements in nutritional status of mothers and children and improvements of income and use of social services, the programme had not been very effective.
The CPC focuses on a number of issues that are essential to child survival as they will produce a clear pay-off in the short or medium term and because the contribution is complementary to activities of others. For example, within the National Immunization Programme (NIP), the CPC focuses on EPI-plus, e.g. micronutrients and deworming tablets for women and children. In the area of HIV / AIDS, for which there is considerable external support, the CPC focuses on life skills for young people and parenting skills. But there have also been several areas, where efforts have been lower and where an increase of attention would be rewarded in terms of impact on child survival. These areas include, for example, support to birth spacing, maternal nutrition, delivery and newborn care, and literacy and life skills education for out of school youth. Given the important role of informal and private providers in service delivery for the CPC’s target beneficiaries, there was also less focus on this group than warranted.
The CPC has contributed considerably to the policy / legislative framework for Children in Need of Special Protection. One newly developed law was submitted to the Council of Ministers, which has been awaiting adoption by the Council of Ministers since 2001. At the field level, awareness raising campaigns against trafficking and sexual exploitation together with a whole range of activities (police training, phone hotline, database of cases, etc.) have led to an increased awareness in cities and rural areas on the issue and has resulted in an increase of the number of reports and arrests of abusers. Whenever possible, national policies have been informed by field experiences. One good example is the survey on alternative care which served as the basis for the subsequent development of a regulatory framework on alternative care for children without primary caregivers.
Sustainability, replicability and mainstreaming of results. There is no evidence as to what happened to villages and communities, which were supported more or less directly by Seth Koma, but where external assistance was reduced or even withdrawn. At least since 2003, the Seth Koma Programme has diminished its direct support that had been assisted through many years in many locations, as it shifted its focus to the commune level and cooperation with the Seila Programme. There is also no direct evidence whether the development models introduced by Seth Koma were used and / or adapted by national or external partners outside Seth Koma project areas and provinces.
There were no stated aims of sustainability and replicability of the main activities related to child survival, and there is no evidence that activities were selected with a view to their sustainability or replicability. The major effort on mainstreaming of interventions or experience which would impact on child survival has been through work on national policy and guidelines (including the use of pilot studies or test cases) and at province level, through support to public sector service planning and implementation. However, the sustainability of the important child survival related efforts of this CPC is considered unlikely in the short and medium term, due to dependence on funding and technical support from UNICEF. The expansion or replication of successful initiatives beyond their present areas of implementation will also require funding and technical support from UNICEF or other partners.
As far as the CNSP Programme is concerned, activities related to children in post-conflict have the longest track record. There are some examples of relative sustainability and mainstreaming of supported initiatives, e.g. the Cambodian Mine Victim Information System (CMVIS) which is a well-established, comprehensive and well-utilized database. There is the idea of using CMVIS as a model and apply it to the collection of data on other protection issues. The challenge, however, goes beyond data collection in view of the complexity of conceptualizing child protection issues. The creation of networks and structures is necessary, but not sufficient to ensure improved child protection. Referral mechanisms are limited in view of the scarcity of social services available. Qualified staff is insufficient. The CPC is promoting social service training, but it is still too early to expect sustainability and mainstreaming without available services and external support.
Strategic orientations and recommendations
General considerations. The CPE endorses the view of current CPC management that there is no need for major changes to the programme beyond what is required in terms of adjustments to the policy framework of the Government (notably NPRS and Rectangular Strategy) and CCA and UNDAF. In agreement with CPC management, the CPE makes recommendations as to the CPC’s contribution to deconcentration and decentralization, on issues related to child survival, as well as on the programmes geared at children in need of special protection. The CPE recommends some adjustments related to HRBAP, RBM and performance monitoring and evaluation.
Deconcentration and convergence. The re-orientation of the Seth Koma Programme that was undertaken during the MTR of 2003 dramatically increased its potential significance as a contribution to national policies and strategies. It transformed the programme from a set of discrete activities at the level of individual villages and communities — disconnected even from other programme components of the CPC — into an approach that has increased chances of replicability and mainstreaming of results because it is part of a national effort with a consolidated and coordinated external support mechanism.
The shift of focus from the village and community level to that of Commune Councils involves a number of new challenges that will need to be met. While it is an excellent initiative to mobilize Commune Councils beyond mere planning and budgeting of infrastructure projects and involve them in issues related to the realization of children’s rights, more particularly in health and education, there is still some lack of clarity as to what roles the Commune Councils should play in this regard. It will be important to include these issues in the Councils’ agenda’s, ensure that their members are willing and motivated to deal with them (e.g. by counting women among their members) and that specialized committees are created to tackle children’s and women’s rights. These are necessary, but not sufficient steps. There is a need to undertake a process to determine what responsibilities the Commune Councils should assume on these social issues. There is also a need to carry out a more refined capacity analysis as the basis for an appropriate programme that could strengthen these capacities.
As much as it is important to clarify what would be the tasks and responsibilities of Commune Councils, it is also important to determine what they should not be involved in directly. Over the years, the various line Ministries, often with UNICEF support, have stimulated the creation of associations, e.g. those that promote good mother and child care practices (mother support groups), contribute to appropriate education (parent-teacher associations) and manage water and sanitation infrastructure (water user groups). It would, however, be appropriate to determine what should be the relationship there should be between these associations and, for example, the children and women’s committee of the Commune Council.
In its new role, the Seth Koma Programme should also learn from its past experience that it requires much closer links with technical expertise if it is to produce significant outcomes in terms of behavioural and institutional changes, e.g. in the area of improved nutrition and health practices.
The key policy documents concerning deconcentration and decentralization emphasize the need for horizontal integration of the programmes of the line Ministries at all levels. Specific guidance is still under development as part of the formulation of the respective strategic framework. To the extent that the UNICEF supported CPC is to make an active contribution to this national process, it will have to overcome its own “silo-effect” and develop a more convergent intervention model at the province, district, commune and village levels. Seth Koma has the potential to be an important integrative force, provided it succeeds in mapping out with each of the other programmes concrete areas of cooperation and joint development initiatives. This should be an important part of strategy development as part of the preparation of the next CPC.
Child survival. The next CPC should re-affirm the primacy of the right to survival as an over-arching goal within which the contributions of each sector and programme can be defined in a logical and inter-active manner. Success in reducing child mortality (MDG-4) will be closely linked with reducing hunger (MDG-1), improved education (MDG-2), maternal health (MDG-5), reduced gender inequality (MDG-3), increased access to sanitation and water (MDG-7) as well as effective interaction between sectoral and local government entities.
Given the long-term nature of some of the challenges, it is recommended that the CPC 2006-2010 be designed as the first stage of a two stage strategy covering the decade to 2015 which is the target date for the MDGs. The CPC 2006-2010 would form the first stage of a two stage strategy covering the decade to 2015, the target date for achieving the MDGs. This “Stage I” CPC would particularly address factors underlying the high mortality rates, and in addition would test and disseminate feasible approaches to overcoming the more direct obstacles, which could be greatly scaled up in the “Stage 2” CPC (2011-2015).
Within this ten year vision, it is proposed that the next CPC (2006-2010) adopt two major objectives: a) improved uptake of key child survival interventions by the rural poor through intensified multi-sectoral support; b) empowerment of today’s young people, as tomorrow’s parents, for planned and successful childrearing. This group will be the parents of young children in the 5 years leading up to 2015.
Key actions for consideration under the first objective should include: a) translation of MDGs and related actions into monitorable targets within the development plans of provinces, districts and communes; b) testing of an improved model of social mobilization/health promotion for the empowerment of poor rural communities; c) provision of results-based incentives and support to increase coverage and uptake of key interventions or behaviours in health, education, water and sanitation; d) maintenance and expansion of the Equity Fund pilot, while seeking options and partnerships for scaling up this fund; e) exploring the most appropriate roles of the CPC in engaging non-governmental organizations and the private sector to improve child survival. The second objective would be realized through actions directed at improved knowledge and behaviour of young people and their service providers, and at strengthening community support for this group.
In deciding the overall strategy for improving child survival in the next CPC, it is recommended that UNICEF address the several important issues. Firstly, there is a need for continued recurrent funding and for the linking of expenditure to results. Increased use of skilled health services by the poor is needed to reduce deaths from respiratory infection and other childhood illnesses. However, the Government has insufficient operating funds for health even at the present level of utilization and a relatively small proportion of external funds is used for recurrent service costs. The CPE recommends the expansion of the Equity fund model piloted in Svay Rieng, its inclusion in the Health Sector Strategic Plan, and its funding.
While UNICEF has a range of effective partnerships with technical and funding partners for defined projects or sub-programmes in the health sector, the relationships are less clear with respect to area-based system strengthening especially in UNICEF’s priority provinces. While partnerships with other UN agencies are seen as cordial and productive, there is an opportunity in preparing the new CPC, to review the relationships, especially being alert for areas where roles might be adjusted, and to ensure that a cohesive approach is presented by the UN agencies when coordinating with other partners.
Communications support in the current CPC has emphasised two areas: (1) high level advocacy and (2) mass media approaches to behaviour change, with some attention to equipping health workers to conduct interpersonal communication. Assessment of the effectiveness of these approaches will be helpful in designing the next CPC. In addition, there should be attention to support for “settings-based” health promotion, which takes account of the particular environment in which change is to be achieved, and involves local participation in planning and conducting a range of feasible and mutually reinforcing activities for a defined community. UNICEF would explore the application of this approach in the next CPC, including analysis to identify key environmental barriers to the desired behaviour change, and allies in building community support.
Children in need of special protection. The origins of the CNSP programme were closely linked to the post-conflict situation which prevailed until the mid-1990s. At the present time, there is still much interest among donors concerning landmines and other aspects of the post-conflict situation. UNICEF remains a preferred partner in this process. It is likely that the post-conflict issues will become less prominent over time although this will be a gradual process. Where capacities exist, UNICEF should find an exit strategy in relation to landmine related activities and identify new protection issues, trying to replicate workable models (e.g. data collection).
The CNSP programme has apparently been somewhat issue-driven and also rather dependent on the availability of donor funding. The public notoriety of certain issues (e.g. child trafficking) diverted attention from other forms of neglect, abuse, violence or exploitation, which are less known, but maybe as serious or even more important. There is a need to build on the more structural elements of the current CPC, avoid mere “activism” on specific issues and adopt as systematic and programmatic an approach to major protection issues as possible.
A more programmatic approach for the CNSP could be articulated around four major dimensions, each of which would describe a contribution to filling important national capacity gaps: a) research, monitoring and advocacy; b) legal reform and policy advice;c) experimentation of interventions on new protection issues; d) direct protection capacity.
Policy advice and support to legal reform, including capacity building in the areas of law enforcement and legal aid, should constitute the core of the CNSP programme in the future. The RGC and UNICEF should engage in a long-term visioning exercise and design a plan of action against the background of the Cambodian MDGs and overall social policies of the RGC. The plan of action should define the targets to be achieved on all major issues over the next five years (as has already been done for birth registration which should attain 50 percent by 2005).
Human Rights Based Approach to Programming. While HRBAP has been given adequate attention in the design and implementation of the current CPC, there has been insufficient emphasis on gender equality and empowerment of women. MDG 3 aims to eliminate gender disparities in primary and secondary education preferably by 2005 and at all levels by 2015, but also addresses the more ambitious challenge to achieve gender equality in education and throughout the various spheres of society for all age groups, which is a much more complex and profoundly challenging endeavour than reaching gender parity.
The gender equality and empowerment approach has implications far beyond the education sector. In virtually all spheres of development, it involves reconsidering stereotypes that historically and culturally determine role matters of girls and boys and women and men. It is necessary not only to work closely with women and girls, but, also, with men and boys and the public at large on eliminating gender bias and discrimination Children entering puberty require special attention and an emphasis on gender equality needs to be integrated in life skills education. There is also a need for ongoing contextual / holistic monitoring and evaluative analysis of and reporting on gender relations. At all levels and with all partners, it is necessary to strengthen gender sensitivity and enhance the respective capacity in programming efforts. To enhance gender sensitivity, there is a general need for gender-disaggregated data. Most importantly, however, even if empowerment of women is the ultimate aim, gender relations need to be addressed in all programmatic approaches that are addressed to both girls and boys and to women and men.
The CPE recommends re-activating the working group of gender focal points with the UNICEF Country Office and perhaps expanding it to counterparts in governmental and non-governmental partner organizations. A functional network of gender focal points in various line Ministries can be a good tool to enhance government attention to gender equality and women’s empowerment. Another important tool to boost attention for gender equality is a policy specifically aiming at gender parity in the staffing of both UNICEF and in line Ministries and partner NGOs. Although some progress has been made in the UNICEF Country Office, there is still a notable gender disparity especially among Cambodian staff. Gender disparities are much more pronounced in national government institutions, but there is a policy to increase the presence of women at all levels.
Results-based management and performance monitoring and evaluation. RBM is part of the common framework for 2006-2010, which the agencies of the UN family in Cambodia agreed upon in October 2004. The UNDAF framework is related to the Rectangular Strategy 2004-2008 as well as the new National Development Plan. The UNDAF is articulated with the Cambodia Millennium Development Goals in mind as well as the need to take into account the dimension of human rights.
There is scope for strengthening the design of the next CPC by applying the basic principles and practices of RBM. A causality analysis will have to be followed by support to implementation in the form of experimentation (limited in time), identification of partners and capacity analysis and identification of actions to address the capacity gaps (particularly in terms of influencing policies and strategies). Results based programme planning will have to demonstrate the causal relationship between the intended results at different levels (outputs, outcomes and impacts). The sum of planned interventions should be sufficient to achieve the expected results. The logical framework of the UNICEF supported CPC should be aligned to the UNDAF Results Matrix (a requirement for the Country Programme Action Plan or CPAP, which replaces the Master Plan of Operations or MPO).
The recent adoption in the Cambodia CPC of a five-year Integrated Monitoring and Evaluation Plan (IMEP) is important as it identifies needs for data and analytical information and plans monitoring and evaluation in relation to major milestones in CPC implementation. Once there is an adequate logical framework, studies and evaluations can measure results as compared to baselines and situate results in the broader framework of development work.
PDF files require Acrobat Reader.