2018 Burundi: Impact Evaluation of the Lumière Project
Author: Anousheh Alamir, Tillmann Heidelk, Julia Jadin, Juan Carlos Munoz-Mora, Rama Lionel Ngenzebuke, Philip Verwimp (Team Leader)
With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System (GEROS)". Within this system, an external independent company reviews and rates all evaluation reports. The quality rating scale for evaluation reports is as follows: “Highly Satisfactory”, “Satisfactory”, “Fair” or “Unsatisfactory”. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report, and the executive feedback summary labelled as ‘Part 3’.
In rural areas of Burundi, a very small percentage of all households have access to the electricity grid, resulting in a high reliance on other sources of energy such as wood, batteries and kerosene. These sources are costly, inefficient, unhealthy and not friendly for the environment. The use of energy is an important indicator of the welfare level of a household, a village and an entire country. By means of its Lumière Project, the UNICEF office in Burundi want to contribute to diminishing energy poverty by empowering communities with a reliable, clean and sustainable energy source. By implementing the project via the social enterprise model, the intervention aims to leverage the power of the market and create a chain reaction of mutual social and economic impacts. Access to clean energy technology enhances education, improves health and safety and provides economic opportunity for the communities, marking its relevance for UNICEF statutory objectives. In the Lumière project, the UNICEF office in Burundi addressed energy poverty by introducing rechargeable lamps in villages. The introduction of these lamps is meant to increase the availability of light in the household, to reduce the cost, to render its use more efficient, to reduce the use of unhealthy sources and to reduce negative environmental impact. The intended beneficiairies of the project are households who are not linked to the grid in the rural areas of Burundi. The vehicle for the intervention is FVS-AMADE, a Burundese NGO with many years of experience working in the rural areas. Working with them, UNICEF wanted to see the project as a social enterprise that over time could be self-financed. Rather than building new local groups, FVS-AMADE was able to work with existing village-based organisations (VBO), who had organized themselves as savings groups to support vulnerable children. In this report these organisations are also called “groupements”, the name used locally to identify them.
With this evaluation, the UNICEF office in Burundi wanted to obtain detailed information at two levels:
- at the level of the village-based organisations: how are they working? How important are the activities supported by the Lumiere Project in the overall portfolio of the VBO? Is the project increasing the capacity of the VBO to support its other activities? and,
- at the level of the final beneficiaries, defined the households residing in the rural areas chosen by the Lumière Project.
We use a mixed-methods approach in the evaluation, consisting of data collection at the level of the VBO, by means of focus group interviews of VBO members (termed savings group questionnaire later on in the report) in all locations under study. In each locations we also field a household questionnaire to members of the VBO and non-members. Randomized controlled evaluations have the capacity to measure the causal impact of a program. This is the case because on average the treatment and control groups are the same, which is a great outcome of the procedure of random selection into treatment and control. This procedure avoids selection into treatment/control based on a series of background characteristics, motivation, distance, political affiliation, wealth or any other. Accordingly the only difference between the two groups is that one of them will receive the intervention while the other does not. Changes at the end can then be attributed to the intervention.
Findings and Conclusions:
- Findings at the level of the village-based organisations (VBO). The village-based organisations (VBO) working with FVS-AMADE request each member to contribute weekly an amount which varies between 500 and 2500 BIF.
- Findings on Energy Use among the final beneficiairies. Lumière lamps mostly replaced battery lamps, kerosene and candles within surveyed households. A clear decrease in the use of battery torches and candles is observed amongst Lumière lamp users. User spent 32 BIF less per week on battery torches while non-users spent 71 BIF more. A general trend of decreasing use of kerosene is observed amongst all surveyed respondents, but more outspoken for users. The latter spent 189 BIF less per week, while non-users spent 148 BIF less.
- Findings on Take-up. Overall uptake of the technology is at about 12%, but increases to 18.5% among saving group members and to almost 30% among those HHs who live in collines which experienced the intervention.
- Findings on Welfare. We document that the Lumière Project is an important source of revenue at the local level for the VBO and it is responsible for the increase of welfare in the collines that are part of it. The point estimate of the increase in welfare is between 10 and 18%, depending on specification, with a preferred estimate of 14.5%.
- Conclusion. The evaluation used a mixed-methods approach, whereby the quantitative part consisted of a cluster randomize-controlled trial, with 34 treated villages and 29 control villages, interviews with 1.000 households at baseline and at endline, with half of the households members of the VBO and the other half not. The qualitative part consisted of focus group interviews with members of the VBO; Based on our quantitative analysis, we find an increase in welfare expressed as consumption per adult equivalents in treated villages compared to control villages of between 14 and 18%, relative to the baseline.
- Working through the VBO ensures a certain take-up among its members, but limites take-up outside. Indeed non-members perceive the re-chargeable lamps as club goods, accesible only to members, even when this is not the case. Hence, VBO should be encouraged to open-up and change this perception;
- The role of FVS-AMADE maintains to work in a very hands-off way, facilitating where they can, helping the VBO as much as possible. The evaluation team never received a clear and transparent answer from them on the many questions they asked. The donor, if it wishes to continue working with the NGO should demand full transparency, both on its finances as well as on the way it works with the VBO;
- Given the speed of technological evolution the Powercycles are already outdated, as solar energy is proving cheaper, more durable and more efficient. Some households in the endline survey were already using it, and do not wait for an NGO or a VBO to help them. We recommend a full transition to solar energy if the project continues;
- The business model, whereby a VBO is responsible for the sales of lamps, their re-charge as well as the management of the Powercycles, worked well. However, with the arrival of solar energy, households will not need a centrally or village-level managed supply anymore and the current business model will not stand a chance;
- The project does not address the use of wood as the main source of energy used in the household, for cooking. This source is responsible for health problems and air pollution. We recommend a (new) project to address alternatives for cooking with wood. As learning from peers is crucial here, the VBO could play an important role in showcasing alternative energy sources for cooking. It may have the potential to replace the current business model around the useof a Powercycle.
- The uptake of re-chargeable lamps outside the members of the VBO remains very limited. This means that the VBO functions as club with certain benefits for its members, but with few impact on non-members;
- Membership of a savings group is a strong determinant of take-up.
- Among households who use re-chargeable lamps, we find a clear decrease in the use of kerosene, battery lamps and candles.
Full report in PDF
PDF files require Acrobat Reader.