Author: Tufts University Friedman school of Nutrition Science and Policy. Institutions: UNICEF and Belgian Survival Fund
The objective of the “Improvement of Household Food Security and Woreda Integrated Basic Services (BSF/WIBS) Project” was to improve household food insecurity and to reduce the mortality and morbidity of vulnerable women and children. Funded by the Belgian Survival Fund (BSF), the program has been implemented by the Government of Ethiopia in partnership with UNICEF since the year 2001 in four woredas of the Oromia and Tigray regions. The Project’s approach is integrated and multi-sectoral, initiating improvements in household and community well-being across agriculture, health, education and water sectors. An important priority of the approach is to build the capacity of Woreda and community level administration to plan and mobilize resources for activities that can continue after the Project’s funding has ceased. A hallmark of this strategy is the extensive involvement of community members in the planning and implementation of a jointly developed workplan.
This report details the process and results of the end-line evaluation of the BSF/WIBS Project. The evaluation was carried out from March to June, 2007 by one faculty and one research staff member of the Tufts University Friedman School of Nutrition Science and Policy together with a team of 3 national experts in the fields of water, education, and rural development/food security. The Tufts Evaluation Team (TET) was charged with five primary tasks: to assess a) the relevance, b) effectiveness, c) impact, d) efficiency, and e) sustainability of the BSF/WIBS. This evaluation also aimed to contribute to the understanding of challenges related to the integration of multi-sectoral programs, scaling up of community-driven activities, and the management and oversight of programs in a recently decentralized environment.
A mixed method approach was employed in order to answer the evaluation questions. Data sources included 1) interviews with project officials and community representatives, 2) focus group discussions with project beneficiaries, 3) a structured household survey of a sample of households from each of the four project woredas, and 4) a community checklist to collect information on community level infrastructure that had been built as part of the project.
While the range of data sources allowed for an analysis of all of the evaluation questions, the study of impact was limited by the fact that no quantitative baseline data existed for the project in Tigray, while in Oromia only a report with summary statistics was accessible. As such, the TET could not assess statistical significance of pre-post changes in well-being in Oromia, and could not learn about project impacts in Tigray.
Findings and Conclusions
The TET concluded that the design of the BSF/WIBS Project is relevant to UNICEF, BSF and Ethiopian development priorities as well to the expressed needs of the target communities.
•One of the biggest strengths of the project was the participatory planning approach that engaged target communities in identifying key development challenges in their areas and in determining the activities that were most contextually suitable to addressing these problems.
•That said, while relevant problems and solutions were identified at the outset by the community, there was insufficient prioritization of activities and assessment of key preconditions for success. Facilitated discussions (between communities and experts) during the planning phase could have addressed these issues earlier and promoted the implementation of activities with track records of success.
•The decision to implement the BSF/WIBS project in the most vulnerable Kabeles and target to only the most vulnerable households was appropriate despite the trade-offs involved. The project was narrowly targeted (according to its objectives), resulting in a low rate of coverage. This design decision made it difficult to detect project impact as part of the evaluation, as the evaluation budget was not big enough to support a sample size with enough power to detect impacts if they existed. As such, the magnitude of impact is likely to be understated in the evaluation results.
•While the project objectives were appropriate given the nature of the project’s interventions, the targets stated in the project documents (e.g. reduction of malnutrition by 15%) were largely unrealistic given the level of inputs available and the number of beneficiaries that could be reached, even under conditions of perfect project implementation.
•Of the numerous project activities, community revolving funds were one of the most effectively delivered in three of the four Woredas (in all except Boke Woreda). For instance, in Bedeno though only 200 households were slated to receive an improved breed of chicken, over the course of the project 575 households (nearly 3 times the target) were allocated 2,281 birds. The development of, and adherence to, strict eligibility criteria ensured that only those poorest and vulnerable members of the community received loans as cash (Tigray) or livestock (Oromia).
•The construction of education and health facility infrastructure was also highly effective. For instance, Samre exceeded its target by 20% over the project implementation period. For example, 12 new classrooms were constructed and 676 pieces of furniture were provided, including desks, chairs, blackboards, and shelving, in 2005 alone. The 2005 monitoring data from Tsilemti show that 8 out of 10 planned classrooms were built and 676 pieces of classroom furniture were delivered. Focus groups reported that the new construction coupled with awareness raising activities motivated parents to send girl and boy children to school for an education.
•According to monitoring reports and the community-level checklist, water infrastructure was greatly improved in many of the BSF/WIBS communities. Water projects were more thoroughly implemented in Boke (92% completion of planned communal water schemes) and Samre (over six times the number of planned water related trainings for government officials) than in Bedeno and Tsilemti (average 20% and 33% of communal water projects completed, respectively). Nearly every focus group stressed that the improvements were welcome but not sufficient given the severity of the water scarcity and the need for water for agriculture, livestock, and personal consumption.
•Household-targeted activities pertaining to the provision of nutrition inputs (eg. supplementary feeding) were delivered at the lowest rates, whereas activities pertaining to community awareness raising and training of project officials were more successfully implemented. Possible explanations for the slow implementation of household-targeted activities include: low community demand for nutrition interventions given the relative invisibility of the problem, a scarcity of nutrition capacity at lower administrative levels, and the lack of an institutional home for nutrition within the local health sector. These types of constraints may have been magnified by more general implementation issues described in the ‘efficiency’ section. Given that activities to improve administrative knowledge and awareness of nutrition interventions were, in fact, carried out, the project may have succeeded in laying the ground-work for more targeted nutrition interventions in the future.
A comparison of food insecurity manifestations between beneficiary households and non-beneficiary households in BSF/WIBS project areas using the FANTA Household Food Insecurity Access Scale showed that, for seven of eight items in the food insecurity index, fewer BSF/WIBS households reported experiencing food insecurity conditions than non-BSF/WIBS households. Thus, at the end of the project, BSF households showed better food security situation than non-BSF households. Based on the assumption that BSF/WIBS beneficiaries were worse off at the beginning (as corroborated in focus groups), the project had a positive impact on the household food security situation.
•Several positive impacts were detected in the health and nutrition of women and children. For instance, visits for antenatal care improved by 8% more in BSF project communities than in non-project communities. Use of contraceptives, delivery using a trained TBA or health practitioner, receipt of tetanus vaccine during pregnancy, and mother’s self-reported perception of her health status had all improved, and had improved more in the project areas than not.
•There were significant reductions, of approximately 10 - 20 percentage points, in each of three anthropometric indicators of malnutrition (WAZ, WHZ, HAZ). Because the data showed improvements of similar magnitude in both project and non-project sites, it is not possible to entirely attribute these positive changes to the BSF/WIBS nutrition activities. Moreover, the only nutrition-related activities implemented to any degree were training of government officials and some community-level interventions (like school gardens) that would be unlikely to affect nutrition status directly. More likely, the observed changes were due to a secular trend or to interventions implemented in both project and control areas (like the EOS).
•There was scarce comparable baseline to endline data to be able to draw conclusions about the impacts of education and water activities. The data did suggest that the improvement in access to a protected water source was greater by 15% in BSF/WIBS than non-BSF/WIBS areas by the end of the project.
•Focus group participants felt that the project served to strengthen both the physical and human infrastructure in ways that are particularly valuable to improving future development investments. For instance, with the decentralization of fiscal and planning control to the Woreda, having the BSF program in place with the secondary aim to capacitate local government has produced a structure and mechanism through which other national programs can operate. In addition, qualitative interviews indicated that the BSF/WIBS project instilled a strong sense of ownership, optimism, and feelings of self-sufficiency within the community.
The evaluation identified problems related to the implementation and management of the BSF project that appeared to account for project achievements that were less than intended.
•All key informants and focus group discussants reported that the project was very well targeted to those most in need of support. However, respondents also felt that food insecurity was widespread among other households not receiving project benefits.
•Monitoring of project activities at all levels was insufficient. This problem can be largely attributed to the overburdened workloads of all project coordinating staff within BOFED and UNICEF and the relatively low priority of the small-scale BSF/WIBS project within these institutions.
•The institutional capacity of both BOFED and UNICEF in regards to food security programming was also limited. Adding other institutional relationships, for example with the Bureau of Food Security, may have been more appropriate.
•Inconsistency (during the middle of the project) in the release of funds from BSF and UNICEF also greatly reduced the efficiency of the project activities. Funds were not dispersed to any project woreda for a period of one-two years (the actual time varied by Woreda) bringing many BSF/WIBS activities to a stand-still and impeding the momentum that had been gathering since the inception of the participatory planning exercises. It has been widely reported that the delay in the release of funds was due to the 2002-03 Ethiopian drought emergency (which affected the BSF Woredas) and the absence of a BSF focal person in UNICEF.
The Tufts Evaluation Team concluded that the BSF/WIBS was a well-designed and very well-targeted project with a commendable commitment to participation, integration, and building local capacity and infrastructure. The project suffered, though, from various implementation challenges that were often exogenous to the community project coordination structure and outside the control of individual beneficiaries
The Tufts Evaluation Team recommends that funding for the BSF/WIBS activities be continued in the four project Woredas, however key changes should be made to the project design, implementation, and institutional framework:
•Proposed activities during future PRA/needs assessments should be prioritized by the communities in order of their perceived importance agreed through facilitated discussions with programs staff. The feasibility of implementing the activities and the ability of proposed activities to drive desired impacts should be assessed with technical experts.
•The importance of developing a strong monitoring information system for the next phase of the project cannot be emphasized enough. Many of the implementation problems that ultimately hindered the project’s efficiency and effectiveness could have been easily addressed and resolved if information about project implementation had been compiled regularly, aggregated, analyzed and acted on at each administrative level.
•BSF should fund at least one full-time National Project Coordinator to be housed in UNICEF. This person would be responsible for coordinating and monitoring the project in all four Woredas and would provide bi-annual reports to BSF.
•A ‘phased approach’ to the implementation of project activities is recommended, with the first phase addressing issues of infrastructure and technical administrative capacity at the Woreda and Kabele level.
•Since the Woreda BSF Coordinator is typically overburdened with other competing priorities, incentives for the Woreda BSF Coordinator should be put in place to encourage his/her careful oversight of the project.
•In coordinating future BSF Projects, UNICEF should partner with regional Food Security Offices located within Regional Agricultural Bureaus rather than partnering only with Regional BOFED offices.
•In future the BSF/WIBS woredas should be used as a learning ground for the larger (but similar) integrated Community-based Nutrition program, in order to test technical strategies, activities, and management practices before scaling them up to 150 woredas.
•As a relatively small-scale project, the role of the BSF/WIBS should continue to be addressing the needs of the poorest, most food insecure and vulnerable households and communities when larger scale projects are focused on broader coverage with less narrow targeting to the ultra-poor.
•Drought is a regular phenomenon in Ethiopia. During the BSF/WIBS project, the redirection of funds and personnel from the existing project to “emergency response” in non-project areas caused a lapse in implementation that represented a major setback for the project and its beneficiaries. The prevention (or mitigation) of the effects of drought and other covariate shocks on the implementation of existing programs should be taken into account in the design of all food security interventions in these regions.
PDF files require Acrobat Reader.
Oromia and Tigray Regions of Ethiopia