2000 KEN: Summative Evaluation of the 1994-1998 GOK/UNICEF Programme
Author: Kairu, E. E.; Kola, H. O.; Momanyi, M. M.
During the 1994-98 program cycle, GOK/UNICEF Kenya Country Programme experienced many challenges and underwent tremendous changes, but the program also achieved a lot in its primary areas of concern; for example, basic education and child protection; water and environmental sanitation; advocacy, communication and social mobilization; social planning, monitoring and evaluation, and health and nutrition. Before the mid-term review of May 1996, the program had a total of 11 programs and 46 projects. During the mid-term review, the program was made more focused and the number of programs and projects were revised.
Unfortunately, mismanagement of funds and other malpractices led to the infamous 1995 audit. Between 1995 and 1996, very few project activities were undertaken mainly as a fallout of the 1995 audit. At the same time, most of the project staff was replaced with a new group of professionals. Finally, between 1997 and 1998, there was a resurgence of project activities in all the key sectors and focus districts. While all these were going on, the program had to deal with a major drought in 1996 and El Nino-related flood emergencies between 1997-98. In 1998, UNICEF decided to channel some of its program support through NGOs while it made direct payments for activities carried out by the government. This was as a result of funds given in 1997 that had not been liquidated by the government (approximately 10 million shillings as of December 1998).
Purpose / Objective
The evaluation was commissioned in order to enable UNICEF/GoK, donors, and NGOs to identify achievements, constraints and lessons learned, and to use these information to improve the quality and rate of project implementation in the next country program. The evaluation was meant to provide answers to four fundamental questions. Firstly, was the extent to which the main program objectives had been achieved at the district as well as the national levels? Secondly, to establish the key achievements of the country program and the key areas in which the program was expected to achieve specific results but was unable to do so. Thirdly, to determine the major external factors that affected the program implementation either negatively or positively. Finally, to establish the extent to which the original and the revised integrated monitoring and evaluation plans were actually achieved.
The evaluation methodology employed was participatory, encompassing literature review, field assessments and consultative meetings. The evaluation team applied the following methods:
- Desk review of program documents. Main components of the literature included reports (annual, review meetings, implemented activities, field visits), results of studies and surveys specifically commissioned by the program, and results of the Situation Analyses of Children and Women in Kenya. A SWOT (Strengths, Weaknesses, Opportunities and Threats) method of analyzing documents was employed, following the evaluation framework given in the Terms of Reference.
- Interviews were held with relevant GoK, NGO and UNICEF officers as key informants to give an input into various issues as well as clarifying the implementation process. Interviews of GoK staff were held at national and district levels. Using information from the various literature together with interviews, the team assessed the effectiveness of program strategies at national, district and community levels.
- Focus group discussions were held with implementing government staff at national and district levels and at community level on Bamako Initiative (BI) sites. Checklists to capture information on implemented program activities, achievements, constraints and lessons learned were used. The process of data interpretation ensured that key decision makers, especially the Chiefs of programs at UNICEF and their GOK counterparts, contributed to the process.
Key Findings and Conclusions
Programme Objectives and Structure
The MTR of 1996 contributed greatly in restructuring and streamlining program projects into manageable units. However, the GoK / UNICEF Programme Objectives encompassed sectoral concerns that were addressed by others in the sector. At the same time, the spread of sectoral projects in different GoK Ministries and Departments made program management cumbersome and difficult to track progress. For effective programming, delivery of services and tracking of progress, it may be necessary to relate program objectives to inputs, inputs to expected outputs and set indicators that relate only to the GoK / UNICEF Programme.
Input, Output and Process Indicators
Several surveys and studies characterized the program period and provided useful information for appropriate action. However, while addressing NPA and Mid-decade Goals, project outputs and process indicators were not well documented to gauge the level of program performance and objective attainment. These were not clearly specified at community level. Aggregating data at district level made it difficult to assess program impact at community level.
GoK/UNICEF Programme 1994-98 had an MPO that specified the milestones for its implementation. Nevertheless, a systematic formulation of indicators of different intervention level was lacking. At input level, the activity targets were primarily set in PPAs. However, some of these targets were not used in reporting, especially in comparing the annual achievements against set targets. The Annual Progress Reports reviewed were not precise on camping the planned and achieved targets.
At the outcome level, the output and process indicators were not well articulated. It is, however, important to note that some PPAs contained output indicators but this was not uniform for all sectoral programs. It was difficult to determine output indicators that could be used to gauge the performance over time.
Information and Communication Project
The advocacy component of the program made commendable progress particularly with regard to facilitating the revision of the Children's Bill. Significant strides were made by the HFV/AIDS/STD sub-component. However, it is clear that the program was still far from achieving some of its objectives; for example, that which sought to reduce the risk behavior and promote responsible social living amongst Kenya's youth in- and out-of-school. Whereas messages were channeled through schools, the young people who did not attend school should have been given more attention. More studies, particularly in the high prevalence areas needed to have been undertaken.
To a good extent, the increasing number of child-related and issues-oriented articles and features that occurred in Kenya during 1994-98 could be attributed to the activities of the Information and Communication Programme. A suitable example was the facilitation of training in feature writing for 19 journalists by the Thompson Foundation Institute. Such efforts should have continued to be provided by the program in the next program cycle.
Water and Environmental Sanitation Project
The lengthy off-shore and local UNICEF procurement systems had a negative impact on the program's ability to deliver assistance in a timely manner. For instance, generators had to be bought in Copenhagen. This practice should have been changed and the sub-office allowed to buy the generators locally. This led to some communities e.g. in Garissa District, getting different generators than the ones they had requested for. Consequently, it led to the need for additional training on maintenance and operations.
There was an urgent need to coordinate the activities of all the key actors in water and environmental sanitation undertakings throughout the focus districts. At the end of the project cycle, many communities still depended on implementing agencies for the operation and maintenance of their water and sanitation systems. The Water and Environmental Sanitation program addressed this particular issue by availing training opportunities for community members, particularly women. The training was provided by GoK personnel.
The capacity of communities to take in capacity-building assistance varied a lot. Important determinants included the community's history of self-help in addition to the level of exposure that the community had had. In most Kenyan districts, sanitation coverage was far behind water in spite of the widespread implementation/demonstration of VIP latrines. Furthermore, differences in access to adequate sanitation between urban and rural environments were noticeable. Planned urban areas were better served than rural areas, slum areas and squatter settlements. In slum areas, incidence of illness was high, especially in children, where it was sometimes as high as 76%.
There was an urgent need for a well co-ordinated national campaign on the promotion of hygiene awareness and basic education on the health risks associated with some traditional behavior patterns. For instance, in Kisumu, the Luo community referred to modern toilets as "Choo ya Daktari" i.e. the doctor's toilet. This type of attitude led to very little usage of latrines. In other areas, like Kwale, water availability was a problem. The PHAST (Participatory Hygiene and Sanitation Transformation) initiative of WES had a lot of potential to change the attitudes of communities with a view to encouraging the communities to come up with solutions to water and sanitation problems that they had identified. The program moved slowly from communities to schools by targeting students. The linkages between the Ministry of Water Resources, Ministry of Health and Ministry of Education should have been enhanced, particularly at the district level. Whenever possible, NGOs and CBOs should have been used as vehicles of assisting communities to improve their water and sanitation facilities.
The refugee water project met the objectives for which it was set up. For instance, it has been able to sustainably supply 15 liters of water per capita to all refugees. The project has also extended support to communities neighboring the refugee camps with supply of spare parts, water-pumping equipment and technical assistance. This has added to the capacity of the communities to maintain their water supplies on a sustainable basis.
Health and Nutrition Project
From the studies carried out on BI sites during the program period and the evaluation team's findings during the field visits, the effectiveness of BIs depended on the following:
- Comprehensive induction of VHCs and CHWs to address community needs. First, by identifying the BI catchment (most BI committee members did not know the population they serve) and to evaluate the purpose the BI will serve. For example, BIs next to a health facility were not doing well as they rely on the sale of drugs (which were cheap in health facilities) as the main activity. BIs should play a more complementary role and supplement the facility, and not rely on the sale of drugs.
- On-site training of community health workers could be more appropriate, particularly if PHAST methodology (employed mainly in WES activities) was to be employed.
- Recognition of the diversity in communities by the implementers of the BI initiative is an important pre-requisite for planning activities with GOK / UNICEF support in the context of social, economic and even political situations. For example, in one community, the local assistant chief / chief may be supportive of BI and allow active community involvement whereas in another community, the situation could be different.
- Regular supervision by health management teams (GoK, NGO or Church) to address issues related to drugs and others not easily handled by ToTs/ToFs and CHWs. The sale of drugs, in particular, if left unsupervised, will get out of hand.
BI sites offered good opportunities for sensitizing and reaching communities on immunization, nutrition and other aspects. Except for Bangladesh BI who, on their own initiative, sought immunization services from the health facilities in Mombasa, very few health personnel within reach of these sites were utilizing them. There is need, therefore, to recognize BIs as important entry points to the communities.
Records on BIs were inadequate for planning and decision making. A format for keeping information on BIs needs to be developed and be part of information available with DHMTs. This will be valuable in coordinating and supervising the activities as well as getting support from organizations offering services in the district. It was observed that some BIs were started by NGOs when the performance of existing ones was wanting. The thrust should be placed on supporting existing BI and only start new ones where need dictates.
For a community-oriented program with a lot of emphasis on community participation in decision making, allocation of resources, ownership and sustainability, there was need to refocus the strategy of vertical planning and implementation. This required major considerations in following the lengthy government procedures of channeling resources to the communities that were tied to national priorities instead of community needs. These considerations would have improved community participation in planning, implementation and monitoring, thus moving towards sustainability.
Basic Education, Child Protection and Development Project
The girl child project made some progress in sensitization and awareness creation at various levels. However, non-participation of girls in education was tied to social and cultural practices specific to communities. Thrust for improved participation should have focused on practical impediments at community level to attract more children, particularly girls, in formal education. For example, food availability was a problem in drought-prone areas and girls' tasks included fending for families or being married off to get the family the necessary resources (cattle, goats, money) to support other members. In project areas, there was need to be strongly linked with other programs that could assist in alleviating the underlying causes (e.g. School-feeding Programme, Micro-financing Programmes).
The MOE & HR needed to review policies that could improve the status of female teachers in the system to act as role models for girls' education. It was indicated that there were few female head teachers, and school inspectors. This situation needed to be addressed in order to give support to improved participation of girls in education. The institutions (District Children's Advisory Committee) established at district level for the support of CEDC were in place in all focus districts. Some were making commendable progress on their own (e.g. Kwale) but others needed capacity building in realizing and actualizing their own potentials, capacities and opportunities within the district to support the cause of children in need of protection.
Operations and Finance
The mobilization of funds (both General Resource and Supplementary Funding) was fairly satisfactory considering the major disruption by the external audit. Nevertheless, the disbursement of funds for individual projects required serious attention due to the identified constraints. The most serious constraint was the delay of funds disbursed through the treasury. At the same time, the procurement and supply of materials to project site had experienced mismatch with planned time. To eliminate this mismatch, the supply section should be supplied with PPAs so that they can plan accordingly.
Information and Communication Project
The government needs to view the promotion of information, education and communication efforts as an important pre-requisite for the advancement of women and children and, by extension, the economy, and to appreciate the important role that was played by these two groups in the country's development. Such a vision needs to be amply demonstrated in the allocation of government resources (e.g. physical, financial, jobs, etc.) to the different structures that had been put in place by the Information and Communication Programme e.g. the different ministerial, district and community-based committees. Without such a commitment, there were limited prospects for sustainability for the Information and Communication Programme.
On the other hand, in realization of the limited government resources and the multiplicity of needs that required to be addressed, the program's facilitation of the involvement of the private sector was likely to contribute significantly to the sustainability of the program. This particular dimension needs to be explored to the fullest and accorded support by the program as well as the government.
Water and Environmental Sanitation Project
Future prospects of improving the Water and Environmental Sanitation program will be determined by the extent to which the program will approach a number of issues. First and foremost will be the continuation of its capacity-building activities. Unless the communities have enough confidence and sufficient resources (human, material and financial) to run the program activities on their own, it would be difficult to see the program improving over time. Given the capacity building efforts that were undertaken between 1994-98, there are good prospects that the program will improve its interventions.
The Water and Environmental Sanitation program will also need to work more closely with other programs that have complementary activities in the focus districts. Furthermore, the program will need to straighten out the procurement system in order to make it possible for contractors to keep to their construction schedules.
Finally, GoK will need to strengthen its policy towards more focus on rehabilitation, management and expansion of existing facilities, together with the implementation of new systems.
Social Planning Monitoring and Evaluation Project Policy Analysis and Planning:
The support to data generation was satisfactorily realized and the use for policy analysis was established; for instance, the use of data from CRC reporting for advocacy and identification of gap in resource allocation for basic social services. However, the assessment at the district level indicated a growing demand for district-desegregated data for local-level planning. In order to improve on this, the GoK/UNICEF program should facilitate the development of an effective and sustainable framework that produces district- and national-specific data. If this is possible for Development Plans, then it is also possible for surveys.
Monitoring and Evaluation:
A framework for 3-tier monitoring strategy existed, though it was not very strong at the district and community levels. In particular, there was still need to improve on the M&E tools to ensure the monitoring of NPA goals in quantitative terms as well as strengthening the district and community committees to undertake M&E. In view of this, the objectives for M&E for the program period were seen to be still relevant. This will imply: a) putting in place a deliberate effort to improve on the quality of monitoring and reporting. This will require the formulation of activity for improving the formats for reporting; b) specific evaluation of sectoral programs should be tied to the realization of NPA goals; c) there is need for continued training of district committees on the Training of Trainers aspect on M&E. This should enable them to support community-based committees on reporting.
Community Empowerment to Analyze Data:
BI's effort to collect, analyze and use data at the village level was commendable. However, it was established that only TOTs were given these skills. In their absence, like in the case of Nyalenda BI and Oboke Nyamrerwa in Kisumu District, this implied that there was need to train more community members at the BI sites on this aspect. In order to improve on this, the SPME program should formulate a training program to widen the human resource base for M&E at the community level.
Basic Education, Child Protection and Development Project
Although the project made progress in areas touching on policy, there is still need for more attention. The NFE policy guidelines required revisiting to redefine: roles and responsibilities of the Adult Education Department at all levels; and Integrating NFE with formal education - who meets the costs of those children who initially were not in the formal system due to lack of resources (funds)?
Full report in PDF
PDF files require Acrobat Reader.