Author: Patrick Ward, Alex Hurrell, Aly Visram, Nils Riemenschneider, Luca Pellerano, Clare O’Brien, Ian MacAuslan & Jack Willis
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
Kenya has been seriously affected by the HIV/AIDS epidemic. It has increased the number of orphans in the country and also the vulnerability of affected households, both through the loss of productive adults and through the impact of chronic illness. In response, the Department of Children’s Services (DCS) in the Ministry of Gender, Children and Social Development, with assistance from UNICEF, developed the Cash Transfer Programme for Orphans and Vulnerable Children (CT-OVC). After a small pre-pilot phase, a second larger pilot phase was initiated in seven districts in 2006. At the same time, the Government of Kenya expanded the Programme in other districts to an additional 30. The Programme expanded further in 2008/09, with a total of 30,315 households having received financial support by mid-2009. Additional expansion is planned, the eventual target being to support 100,000 households by 2012.
The objectives of the Programme were clarified as Phase 2 progressed. Its overall objective is to:
Provide a social protection system through regular and predictable cash transfers to families living with OVCs [orphans or vulnerable children] in order to encourage fostering and retention of OVCs within their families and communities, and to promote their human capital development.
The latter includes, specifically: to increase enrolment and attendance in basic school; to reduce the rates of mortality and morbidity in children aged five years and under, particularly through increasing the uptake of immunization, growth control and vitamin A supplements; to promote household nutrition and food security; to increase civil registration of children and caregivers; and to improve household knowledge and appropriate case management for individuals with HIV/AIDS through coordination with other service providers.
An important additional objective of Phase 2 of the Programme has been to undertake a rigorous evaluation. To that end, an independent evaluation was commissioned from Oxford Policy Management (OPM); this is its final report. The evaluation operated in the seven UNICEF/DFID-supported districts. Its focus has been to evaluate the impact of the Programme on recipient households; to assess the operational effectiveness of the Programme implementation systems; and to assess the cost of the Programme in the light of its effectiveness. The evaluation was also asked to determine the impact and cost of imposing conditions with penalties on recipients.
The evaluation undertook a quantitative survey of households and communities at baseline and follow-up, with an additional survey of basic schools in the follow-up survey. The baseline fieldwork was undertaken between March and August 2007, and the follow-up between March and July 2009. Qualitative data collection took place through focus groups and in-depth individual interviews; this was undertaken in 2008 and again in 2009. A costing study was undertaken in 2009.
The impact evaluation was based on a comparison of Programme recipients with a group of controls, interviewed before the Programme began and again two years later. Impact is assessed by comparing changes in the various measures of the welfare of recipients (such as consumption or school enrolment) with changes observed amongst the control households. In this way, the information on the control households is used to allow for any other changes that the population, in general, may experience that have nothing to do with the Programme. A sample of non-recipient households was included to assess the targeting of the Programme. At follow-up, an additional component of the questionnaire asked recipients about various aspects of Programme operations, providing the information for the operational evaluation.
The evaluation took place in the seven districts that had already been identified by the Programme. In each district, two locations were randomly selected to benefit from the Programme intervention, and two acted as controls. In three districts and one sub-location of Nairobi, the Programme decided to impose conditions with penalties; in the remainder, there was no systematic monitoring of compliance with conditions and no penalties were imposed. For the evaluation, recipient households were sampled from a list supplied by the Programme. Other households were sampled from a household listing undertaken in a random sample of census enumeration areas. A total of 2,759 households were included in the baseline sample; of these, 2,255 were interviewed again at follow-up. The analysis is weighted for differential selection probabilities.
The evaluation assessed the targeting effectiveness of the Programme. It examined the characteristics of recipients and the use of the transfer. It also described the operation of (post-targeting) Programme systems. It evaluated the impact of the Programme against its stated objectives, and assessed its cost.
Findings and Conclusions:
The survey showed that the vast majority of OVCs in the Programme are orphans, with some 46 per cent being double orphans. Nearly all OVCs are cared for by a relative, most commonly grandparents or the remaining parent. The proportion of OVCs in recipient households who are male appears to be slightly higher than would be expected, raising questions about whether girls are less likely to enter the Programme. OVCs, and all children in OVC households, have an older age distribution than that of all children, so that Programme resources will tend to be directed away from the youngest age groups in the population.
The vast majority of initial recipients identified at baseline were still receiving payments from the Programme at the time of the follow-up, with just 3 per cent having left the Programme. The main reason for leaving the Programme was due to the OVC reaching the age of 18 years, although it was found that a surprising proportion of those who left, and also community-level Programme workers, do not know why these OVCs have exited from the Programme.
The evaluation found that many aspects of ongoing Programme operations are working well.
The payment system is ensuring regular payments to recipients, and there is no evidence of extensive ‘skimming’ from payments. Only 2 per cent of households report having to pay money to the Post Office staff, and the amounts paid are small for those who do. A slightly larger proportion of beneficiaries had to pay money to someone else in the community, with an average estimated cost of Ksh 109 to those who pay. Qualitative research did not generally find concerns about unofficial payments being made. Overall, unofficial payments are clearly not a large component of the total costs faced by households in obtaining their payments.
For most recipients, travel times are generally manageable. However, recipients in Garissa (and some other remote areas) face much longer journeys and higher costs. In Garissa, recipients spend an average 19.2 hours making a return trip, and 83 per cent have to spend at least one night out of their home. Overall, on average, they spend almost Ksh 1,500 on transportation, accommodation and food for every payment cycle, somewhat more than the Ksh 1,000 compensation for expenses received by them. As the Programme expands to other districts and more remote sub-locations, these issues are likely to recur and planning will be needed to resolve these issues. The cost of collecting the payment is much smaller outside Garissa, although it is still around 5 per cent of the transfer.
The impact of the Programme to date has been mixed, with a number of areas showing substantial positive impact, while others do not.
Cash transfers from the Programme have increased the real household consumption levels of recipient households substantially – by some Ksh 274 per adult equivalent. The result is a reduction of poverty levels by some 13 percentage points. The benefit of increased consumption is concentrated in smaller households, since the value of the transfer (per capita) is diluted in larger households, reinforcing the case for indexing the payments in some way to household size.
The Programme has also increased food expenditure and dietary diversity, significantly increasing the frequency of consumption of five food groups – meat, fish, milk, sugar and fats. A simple dietary diversity score is increased by 15 per cent from the baseline. The extra income has also translated into increased household ownership of a number of assets, including mosquito nets, and beneficiary households are more likely than controls to hold savings. However, there has been no increase in livestock holdings, suggesting beneficiary households are not investing any of the transfer in (livestock) farming activities.
Mean health expenditure does not appear to have increased in Programme areas in real terms, although the analysis shows an impact due to a decline in this measure in control areas, possibly reflecting budget constraints. In contrast, mean education expenditure per child has increased significantly in Programme areas and declined slightly in control areas, although the net difference is not significant.
There has been a decline of over 10 percentage points in the proportion of Programme households reporting receiving assistance from other households, other members of the community or organisations. This no doubt reflects a perception that these households are less needy now that they receive support from the Programme, as would be expected.
The evaluation did not find evidence of increased enrolment or attendance in basic schooling, with around 88 per cent of children aged six to 13 years in Programme and control areas attending at follow-up, and no appreciable increase over the period of the Programme. It might be that the 12 per cent of children who do not attend basic schooling are constrained by other factors, including access to schools and cultural factors, which the Programme has not addressed. There is some weak evidence that the Programme may have increased enrolment in the youngest children. There does not appear to be a positive impact on attendance, which was already high, or on class repetition.
Overall, there is no evidence that the Programme has had an impact on child health indicators. Vitamin A supplementation has increased significantly in Programme areas (by 10 percentage points), although impact estimates are not significant. A number of the other health estimates are indicating a move in the right direction, but are also not statistically significant. The models find evidence of an impact on reducing the frequency of illnesses, and of an increase in poorer households in the proportion of children consulting an appropriate source of care when sick, which is encouraging.
The proportion of children fully immunised is in decline in all areas, significantly so in Programme areas, which appears to be due to a decline in polio and DPT (diphtheria, pertussis (whooping cough) and tetanus) coverage. There is no evidence of an impact on the uptake of growth monitoring, despite being a Programme stipulation. The Programme has not had an impact on the nutritional status of children, although the results need to be treated with some care due to data limitations.
The principal recommendations identified by the evaluation are:
• Strengthen the targeting process. In addition to improving the poverty indicators used (as has been done), an appropriate geographical allocation of recipients should be ensured. The process itself should also ensure the identification of all potential recipients, and support an effective community validation process in which the initial list can be challenged. The appeals process should be made operational.
• Over the medium term, develop permanent systems for allowing newly orphaned and vulnerable children in Programme areas to be identified and admitted to the Programme after the initial registration has closed.
• Plan for the mitigation of costs faced in collecting the transfer by households living in remote areas before any further expansion takes place, building on the lessons from Garissa.
• Strengthen the communication and case management processes so that beneficiaries are fully informed of their rights and obligations. Part of this procedure should be to incorporate the community volunteers who deal with households more effectively into Programme processes through terms of reference, training and payments.
• Information about the days when payments become available at the post offices should be provided in a way that makes beneficiaries feel safe, possibly by word of mouth. Resources should be allocated to support this.
• Consider indexing the value of the transfer to household size and also for inflation.
• Strengthen government financial systems so that DFID funds do not need to be paid via Unicef, freeing up the management fee for other purposes.
• Investigate the factors that are limiting the uptake of basic schooling and health services, and develop strategies to address them.
• Strengthen the complementary activities in supplying households with information, education and communication in relevant areas, and additional services.
• If imposing conditions with penalties remains part of the Programme, their operation needs to be strengthened; the practical and motivational problems of monitoring preventive attendance at health facilities need to be addressed.
• Ensure the next phase of monitoring and evaluation can provide information on the effectiveness of any revisions.
• The Programme should consider whether it should, in the medium term, extend support to a much wider group of children classed as vulnerable, of which orphans would be only one sub-group. This should be considered within the context of Kenya’s overall social protection framework and the protection for children that it intends.
PDF files require Acrobat Reader.