Evaluation database

Evaluation report

2005 YEM: Assessment of the Child Development Project



Author: Zabel, M.; Schuftan, C.; Dastgeer, A.

Executive summary

Background:

The Child Development Project (CDP) is a tripartite partnership among the Government of Yemen (GoY), UNICEF and the World Bank (WB). With a total budget of US$ 45.3 million, it is also the largest collaboration between UNICEF and the World Bank to-date. It is also the first time that the Government of Yemen took a loan from WB and delegated a significant part of the project implementation to another agency, UNICEF, which is, at the same time, a co-funder. CDP is a five-year project that started in 2001 and will come to an end in December 2005. It aimed to improve the basic social services to children and women in 30 districts in nine governorates in Yemen, selected on the basis of social deprivation.

The overall objective of CDP is "To assist the Government of Yemen in the implementation of a coordinated area-based programme for improving the health and nutritional status of children under five and the educational status of girls in primary schools in districts that are currently under-served in the areas of health and education". The project has six key components:

  • Project Management
  • Community Readiness Programme
  • Health Activity
  • Nutrition Activity
  • Education Activity
  • Pilot Early Childhood Development Activity

Purpose/Objective:

The purpose of the assessment was to examine the achievements of all project components against the targets for the selected districts and generate findings and recommendations for improving project processes and results. As the project will likely end in December 2005, there will be also lessons learned to be provided.

Methodology:

The governorates and districts were chosen in agreement by assessment team and UNICEF Yemen country office. The field visits encompassed three districts in three governorates: Al Hudain (Ibb), Al Milah (Lahej) and Zaidya (Hodeidah). Al Hudain and Zaidya are of the first phase of CDP, one district to be considered relatively easy to work with, and the other as relatively complicated. The third district, Al Milah, is of the second phase of CDP. In addition to the three districts visited, another 8 districts in the same governorates respective indicators have been analyzed, thus the assessment was based on data of 11 CDP districts.

The field visits were dedicated to semi-structured interviews with beneficiaries on different levels (community, district, governorate), and to field observations (status of construction of buildings, whether the buildings were used according to their destination, how the data is managed etc.). The beneficiary interviews were conducted as individual interviews or in smaller groups (e.g. female teachers, girls, fathers, water committees) applying semi-structured interview guidelines. Focus Group discussions, as proposed by UNICEF Yemen country office, did not always provide the results expected. Due to the large size of the groups, there was always a small group of opinion makers dominating the discussion.

At the end of the field phase, a presentation of the first preliminary findings of the CDP assessment was made in front of members of the CDP Steering Committee, followed by a Questions & Answers session.

Findings and Conclusions:

(1) Institutional Structure
The CDP is a rather complex and challenging project. It was the first project of its kind, implemented in a tripartite approach among UNICEF, The World Bank and the Government of Yemen. The challenges related to different procurement, financial management and reporting systems, and requirements of the organizations involved were underestimated in the beginning. The existing institutional capacity was not advanced enough for the approach considered, neither at the GoY nor at the UNICEF level. Inter-ministerial cooperation was widely unknown.

At the same time, the CDP design called for an expansion in scope and scale, compared to UNICEF's earlier area-based programme. The implementation area was enlarged from 10 districts in three governorates to 30 districts in nine governorates, from mainly education activities to a multi-sector approach.

(2) Monitoring
The PAD was calling for a strong monitoring function as a risk minimizing measure at project start. The importance of the internal monitoring function and system has been widely neglected by the project for almost four years of this five-year project. The position of an M&E officer in the UNICEF country office has been staffed only at the end of 2004. PCU succeeded only sporadically in recruiting an M&E officer, also just towards the end of CDP. The long absence of a stringent monitoring and quality assurance and the late staffing of the M&E position also had a negative impact on the M&E training of field office staff and on counterpart training at district, governorate and central levels. Although both organizations, UNICEF and the PCU, are sharing the same problem, the monitoring function of the PCU is not as crucial for the project as the one of UNICEF as the main implementer of CDP. This is felt as a weakness of the project.

The data generated for CDP at field office level is sometimes not comparable, as different data is retrieved by different field offices. In addition, sometimes data from different UNICEF sources are contradicting. The problem of data availability and reliability also applies to data sources of the government counterparts at the various levels. Internal monitoring carried out by UNICEF field offices is lacking clear guidance of the country office and related standards.

As regular training evaluation was not conducted throughout project implementation, a lot of valuable information to measure the quality, efficiency and sustainability of training and its impact on achieving the objective of CDP has not been gathered systematically.

(3) Coordination
A complex, integrated multi-sectoral project like CDP calls for continuous coordination among the different project components, partners and different levels of implementation involved. In spite of recent efforts made by UNICEF to intensify coordination between the different sectors responsible for the CDP implementation in the country office and between the country office and the field offices, there is still a lack of coordination between field offices and the country office, and between the components represented by sectors at country office level. There are promising attempts of coordination between HCMC and UNICEF at field level. However, these developments are fairly recent and may be too late to result in a revamping of the project.

The coordination function of CDP appears to be rather centralized; it is thus important to exchange all relevant developments with the field office staff on a regular basis. A crucial role lies with the area-based sector within the country office. This position was held only temporarily after the previous officer left and has now just recently been filled. De facto each CDP component has been established more as a stand-alone project.

The opportunity to share experiences with similar integrated projects and to use synergies, like with SFD's Integrated Community Development Programme, has been missed. However, in components such as education, good results have been achieved cooperating with GTZ and the NGO ADRA.

(4) Partnership and Visibility
The CDP has a good, local visibility overall. However, during the field visits, it became obvious that the project is not known as CDP, but as "the UNICEF project". Other CDP partners, such as the GoY represented by the HCMC/PCU, are not considered by the beneficiaries as major implementing partners.

This relates to the widely missed opportunity of UNICEF and the PCU to work in close cooperation at field level throughout CDP implementation. Only very recently, first positive steps have been taken in this direction. Three HCMC representatives are based in UNICEF field offices and a fourth is based in the PCU premises in Sana'a. have been recruited and assumed work.

Another observation is that UNICEF country office staff itself hardly distinct between the UNICEF country programme and the CDP. Although there are surely valuable synergies between the two, CDP is not "an extension of the UNICEF country programme". The fact that the Government of Yemen borrowed a substantial amount of funds at the WB and has commissioned UNICEF with the implementation of the main part of it, has to be appreciated in this partnership.

(5) Implementing Capacity
At the end of the fourth year of implementation, the overall expenditure rate was 52.60%, whereas, looking at the three sources of budget, the picture is as follows: 71.13% of UNICEF's own funds, 50.45% of IDA funds and 11.05% of GoY and Community funds.

UNICEF's own funds might also be used for purposes in line with CDP within the country programme 2002 to 2006, i.e. one year beyond the end of CDP. More critical is the assessment of the unspent IDA funds, i.e. US$ 14.27 million, which might remain with IDA and will not be accessible to the project beyond its official end date of 31 December 2005. At the time of the CDP assessment, a contract rider with a no cost extension of CDP appeared unlikely.

The expenditure rate of the education component at the end of 2004 was as low as 37% for community schools and 30% for women teacher training. Only textbook distribution achieved 92% expenditure. The Government of Yemen has so far only contributed to textbook distribution up to the district level.

The low expenditure rate in education, in particular in the women teacher training, has a quantitative and qualitative component. The training cost per person trained remained below those originally estimated. This indicates an increased cost effectiveness of this activity. However, there is a big gap between number of people expected to be trained and the actual numbers. This calls for an as rapid as possible implementation, as budget reallocation is not likely during the last semester of the contract period. As the funds for construction and rehabilitation in 2005 are only UNICEF's own funds, there may be certain flexibility in using them beyond the CDP contract. After a closer analysis of the funds spent and the remaining budget, the SC committee should take a decision.

(6) Female Rural Teachers
The target "the proportion of trained female teachers for grade 1-6 at schools in intervention areas increased by at least 15% by the end of the project" is unlikely to be achieved. Instead of increasing by 2000 female teachers nationwide per year, the number of female teachers during CDP implementation has even declined in CDP districts. This will diminish, if not jeopardize, the effects of the CDP education component, and also has consequences on the speed of closing the gender gap in education in Yemen.

The problem of lack of rural teachers, in particular of female teachers, is intrinsic. This is an important missing link to achieve the intended impact of the CDP, as well as other education projects.

The definition of quota for female rural teachers involves three ministries: the ministries of education, finance and civil services. Often, one Ministry is putting the blame for recruiting too few female rural teachers on another. This problem existed since pre-CDP times and has been repeatedly addressed by members of the Education SWAp. It also relates to the lack of inter-ministerial cooperation and coordination.

Female teachers assume several important and highly gender-relevant functions in Yemen's rural areas: (1) as trusted teachers for girls, in particular for girls in pre- and puberty age; (2) as possible links to the females of the community, i.e. mothers and other girls, communicating not only education-related issues; (3) as a role model within their communities and (4) as a possible professional alternative (of only a few) for women.

(7) Sustainability
Based on the financial contributions the GoY has made so far to CDP, the prospects for its overall financial sustainability have to be assessed as low. Putting finally in place HCMC representatives at governorate level is a good opportunity to enhance institutional knowledge and increase prospects for sustainability before the project ends. So far, institutional sustainability remains week.

A positive element for financial sustainability has been observed in some villages where the water components implemented water schemes, and where communities have collected substantial amounts of money for maintenance of the system. The sustainability of the nutrition component of CDP is not assured after project end. It is almost sure that the MOH does not have the financial resources to continue training and retraining volunteers. An expansion of the activity to additional districts is late in starting this year and may not yet establish CBN firmly by the end of the
project.

The sustainability of the school book distribution up to school level is not secured beyond project end either, as the GoY is contributing under CDP only to the distribution up to district level. An exit strategy was not available at the time of the CDP assessment.

(8) Gender
Women have not enjoyed equal access and representation in the community-based committees, such as water committees, parent-teacher associations (de facto mainly fathers' associations; no cases of mixed committees were seen during field visits). In the cases identified during the field visits, women were either not proposed or not applying for election or, if and when finally elected, assumed rather a "token women function," and not really actively participating as a committee member. The gender aspect has not sufficiently been built-in in the CDP project design.

(9) CDP Component Specific Findings

Health

  • A situation that makes little sense was found in which IMCI drugs are given out free and others not. IMCI drugs have been out of stock for months, creating serious problems for the staff in the health centers, since mothers are then asked to pay for the same drugs from the revolving fund.
  • Health Facility Committees do not participate in the management of the drug revolving funds or in the spending of the operational budget of the facilities.
  • The drug distribution to rural health facilities is on drug-by-drug basis, and is cumbersome for the staff to go and get.
  • There is no referral sheet in use for patients.
  • CDP has an anti-malarial component in one governorate that has not been monitored closely.
  • During their field visits, the assessment team did not find micronutrients in the stock of the health facilities visited.
  • Community communicators in the project are not yet trained on HIV and AIDS.

Nutrition

  • The coverage of benefits and services under the nutrition component is still limited, six months before project end.
  • UNICEF and WFP both have potential complementary nutrition interventions in different geographical locations.
  • CDP currently provides training to volunteer IMCI communicators and nutrition volunteers separately.
  • UNICEF runs separate ECD activities with its own funds, alongside with ECD activities under the CDP project.
  • No provisions have yet been taken to seek funding to continue the CBN programme. The assessment team found that CBN has neglected the micronutrient aspects of a nutrition intervention.

Education

  • During the field visits, cases of misallocation and inadequate use of classrooms were observed, e.g. teachers were sitting in a "management room" built as a classroom, whereas children were sitting on the stone floor outside.
  • CDP school construction is, after changing of the design to be in accordance with MoE requirements, still at the lower end of the models benchmarked (9423 US$). The requirement of the MoE to include teachers' room and latrines to all school constructions is justified.
  • Some of the teachers' rooms have been occupied by male teachers only, whereas the female teachers were meeting outside.
  • Some CDP schools visited were either not equipped with sanitary facilities or the existing facilities were broken and out of use.
  • Overall, the textbook distribution has been successfully implemented. The need to equip the textbook warehouses was only discovered during the course of the project. IDA funds were reallocated to purchase pallet racks, stackers and fork lifters. At the time of the assessment, these items were under procurement by PCU.
  • Under CDP, the Government of Yemen is only contributing to textbook distribution up to the district level. Although not included in the budget,
    UNICEF and the WB have distributed the books from the districts to the schools. This helps to reduce the direct cost of schooling for children, as the funds for bringing the textbooks from the district to the schools is usually covered by the parents. However, the question of introducing an exit strategy for the time beyond CDP support and the related sustainability issue, remain.

Early Childhood Development

  • The Early Childhood Development component is attached to the education sector at the UNICEF country office. It only started de facto in mid-2004, after the revamping of the workplan. It seems to have finally taken off well, although it is too early to comment in detail on the quality of results. The component started to allocate funds, and the expenditure rate after six months was 29.11%. Taking the short implementation time of 1.5 years for ECD into consideration, it still remains low.

Water

  • Overall, the sanitation sub-components have not been adequately addressed during CDP implementation.
  • The sanitation sub-component of WES should have been synchronised to take place together with the implementation of the water schemes. This did not occur for a number of reasons, including the unclear roles of UNICEF and GARWP in the area of sanitation, including the type of sanitation that the PAD envisaged and the lack of sanitation expertise to-date within GARWP.

Community Readiness

  •  It appears too late to start participatory planning training in the last six months of the CDP to achieve sustainable results. The participatory planning training should have been provided in the beginning of the project, to use it as a common element needed in all other components.
  • The capacity building of local government officers to contribute in the best way possible to the participatory planning exercises was weak. A greater degree of training of local government partners of CDP is needed in social mobilisation, conflict resolution, planning, monitoring and, last but not least, in record-keeping.
  • The Social Mobilisation Network, unless reinforced by funds and continued efforts of UNICEF, a third party or the government, will dwindle in its
    significance and relevance as time goes by.
  • Rather than working in a scattered manner, as is currently the case, resources and attention should have been focussed on particular areas where certain health issues were more pressing e.g. an area where malaria was more prevalent. This would have also increased feedback, backstopping and interaction amongst the different stakeholders involved, thus leading to greater effectiveness and visibility of the network.

Recommendations:

(1) Programme Management
To involve the newly-assigned HCMC representatives (so far in four governorates) at governorate and district levels actively into CDP field work; HCMC to recruit representatives for the remaining CDP governorates; to select potential candidates according to a transparent system and a defined profile. This should be done without further delay, to allow maximum use of the scarce time before project end for knowledge transfer. The activities of the different components should be coordinated as far down as village level.

For future integrated projects, to share experience and best practice with projects having similar target groups and objectives; to arrange a meeting between CDP, PCU staff and SFD staff to share their experience at field level with CDP-like, multi-sectoral programmes.

Further foster cooperation between UNICEF field office staff and the newly appointed HCMC representatives; to involve them actively in field monitoring and monitoring training; to proceed with a transparent and professional selection procedure for eventual further HCMC posts to be filled; GoY to provide more inputs during project preparation and implementation; to enhance the impact of the CDP, exchange experiences and best practices with other implementing agencies; to coordinate the work assumed by various agencies working in the same field and geographical region (SFD, CDP); to intensify, without further delay, the positive, recent trend of joint work between HCMC and UNICEF at field level; to prepare a work and financial plan on how to continue the HCMC presence in the districts beyond project end.

(2) Health
To find a solution to the current situation of free IMCI drugs versus paid other drugs; to take measures to assure no interruptions in the supply of IMCI drugs; to allow health facility committees to also participate in the management of the revolving drug fund; to make the new health facility committees accountable for the operational budget of the facilities; to consider distributing pre-packed drug kits for six months to health units and health centers; MoH and UNICEF to design, print, train-on and start using a formal referral sheet for patients sent to a higher level; to secure a closer monitoring of the anti-malaria intervention of CDP in 15 districts; to take measures to assure no interruptions in the supply of Vitamin A, iron and folic acid supplies in all health facilities; to add HIV and AIDS training to CDP community communicators' training now starting, using the newly-developed manual; to include adult family members in the safe motherhood training, as they are most important in home deliveries.

(3) Nutrition
To expand the nutrition component either to three more districts or, more preferably, to the current ten districts; both organizations – UNICEF and WFP - to focus their existing collaboration to cover the CDP districts; to train the already-trained nutrition volunteers also as IMCI communicators: they have initial skills and a high motivation.

The Nutrition Department of the MoH to start soonest the preparation of a proposal for the 2006 onwards funding of the CBN activities; to tackle the micronutrients problem of infants and women more proactively; to link future funding of the CBN component with ECD funding in UNICEF since these two components are intimately linked.

(4) Education
It is recommended to include the correct use of classrooms as an indicator for field monitoring; to monitor the use of classrooms constructed and clearly instruct the teachers about the attributed use of the classrooms; to consider in the new construction the necessity of one or, in case of female and male teachers, two separate teachers' room(s).

To build/remodel facilities with a pit or other latrines both for girls and boys, to assure water supply and to assure its maintenance; to assure that female teachers and students do equally benefit from this infrastructure; to follow up, in the course of the monthly school supervision missions, the appropriate use of infrastructure. To allocate a higher number of female teachers to the CDP-supported schools at least for the last school year before project end; to allow their training before project end; to publish the vacancies for rural female teachers with a secondary school degree at governorate, district and at school levels; to recruit, with priority, those volunteer teachers who have gained working experience already; to give priority to the CDP schools in the 30 districts; to have an inter-ministerial meeting ASAP to discuss and solve the endemic problem of the lack of female rural teachers in Yemen.

As soon as girls reach puberty age, they are, if not withdrawn from school, often segregated in the sitting arrangements in the classroom. To avoid segregating female students in mixed classes, to avoid squeezing girls in the far corner of the classroom; to consider for female teachers to show their faces while conducting the lessons, as interactive communication with the pupils remains otherwise very restricted; to include a profound gender sensitization component into teachers' training - for male and female teachers.

It is recommended to introduce an evaluation system for training activities at the beginning of any project, wherever possible, in line with the existing systems involved, e.g. of ministries, governorates and districts; to use standard templates as a tool for quality assurance and to facilitate aggregation of information.

The PCU/UNICEF should complete the procurement process of the equipment for the warehouses ASAP and should provide training on the equipment.

GoY to develop a sustainable strategy beyond project end for delivery of school books along the transport chain from the printing house to village/school level.

(5) Early Childhood Development
The only recommendation for ECD is to speed up the implementation of the ECD component according to the revised work plan before the end of 2005, making good use of the funds available for this component.

(6) Water
It is recommended to define clear roles and divide tasks within the sanitation subcomponent; GARWP to enhance their sanitation expertise; to share experience with similar projects with a sanitation component, e.g. SFD.

(7) Community Readiness
As far as possible in the remaining time, to do more training and in-country exposure visits to demonstrate how community participation works. This should enhance the capacity of district and local councils and of governorate officials, and thus increase the impact of the project.

The focal points need logistical support that would enable them to interact with liaison persons and community communicators at least once every two months; to provide incentives, financial and/or otherwise, to focal points, liaison persons and community communicators; to introduce regular refresher courses.

Lessons Learned (Optional):
There are a several valuable lessons learned from the CDP, being the "pioneer" project for this kind of tripartite approach. There are also comparative and complementary advantages between UNICEF and the WB.

(1) Planning
A lot of attention has to be drawn to the early phases of the programme innovation process. The innovation cycle has to be sufficiently progressed, i.e. packaging and institutionalizing have to be completed, before scaling and scoping up. These steps have to be analyzed and decided carefully and, more importantly, jointly by the partners involved.

A risk assessment should include the project size and scope, and the scale of money involved. Both should be in proportion to the existing capacities of the UNICEF country office and the institutional capacities at the government level. UNICEF should consider a full-time, high-profile manager for a contract of the dimension of CDP. Only if these elements are in place will such a project be in the position to face the challenges related to financial management, procurement and reporting. All partners must be dedicated to, and involved in, capacity building from the very start of the project.

An important decision has to be taken at planning stage by the organizations involved at HQ and country levels: Is the organization prepared, willing and capable to embark on this kind of partnership, and which resources are required. This includes harmonization of processes (reporting, financial management, procurement). Mid-term reviews should be carried out jointly by GoY, UNICEF and the WB. The transaction costs will likely decrease if harmonized approaches are applied.

Another aspect to be considered while using UNICEF staff in international contracts is their respective contract duration. As UNICEF expatriate contracts are usually for three years, whereas the CDP has a duration of five years, a harmonization of contract durations should be considered to minimize staff turnover during project implementation, as has occurred during the CDP implementation.

(2) Monitoring
At the time of the CDP assessment, it was almost too late to consider the implementation of a monitoring system for CDP. However, a further important lesson learned from the CDP is the need to set up an internal monitoring system at the very beginning of the project - to follow up project progress and to have an early warning tool. Project managers can retrieve and share data according to defined indicators on a regular basis.

This should be done in close cooperation between UNICEF and the government partner(s) (in the case of CDP PCU/HCMC and ministries). Involvement of the national partner is a crucial element for project sustainability, and has to start at the very first stage of the project cycle, identification, and should continue throughout implementation. This project monitoring system should be related to the monitoring system for the UNICEF country programme, e.g. by interlocking the logic models, but must have its own CDP-specific logic model and monitoring.

An important lesson is that the logic model has to be used as a working tool. The original logic model, set up at project appraisal stage, can always be modified at the lower levels of intervention. Should the indicators not be SMART (Specific, Measurable, Achievable, Realistic and Timely), as it was sometimes the case with the CDP, they should have been improved and the logic model adapted accordingly. Weaknesses in the initial logic model, like weak indicators or missing risks, are not an excuse for a lack of monitoring.

Any new area-based, multi-sectoral project should stipulate greater flexibility to change design according to changing needs, as long as the development objectives remain untouched.

On organization level, UNICEF headquarters should establish/reinforce the monitoring practice in a monitoring unit. This unit should develop a system and respective training in project/programme monitoring as a core practice for regional and country office staff. A handbook or manual should be compiled for daily use and for training purposes, explaining the monitoring system and providing advice on operational issues as procurement and reporting standards.

(3) Training
A relatively sophisticated, challenging project as CDP should make training in procurement regulations mandatory for each field officer and sector staff in the country office; this applies for both standards - UNICEF's own and the WB procurement regulations. This can be considered as a means to minimize delays in procurement and reporting and thus enhance implementation efficiency.

For future projects, procurement procedures, financial management and reporting should be considered to be harmonized from the very beginning of the project, to facilitate project implementation. There are already promising examples in other WB/UNICEF agreements, such as the IDEAL project in Bangladesh or the Multi-country Demobilization and Reintegration Programme (MDRP).

PCU/HCMC members should be in place in UNICEF field offices from the very beginning. The field offices must be staffed adequately to the task at stake, in quantity and in professional experience. UNICEF, if assigned to the task of transferring knowledge to the Government counterparts, must be one step ahead of their counterparts, to assume their role in a competent fashion.

(4) Female Teachers
A lesson drawn from the existing gender gap is to involve mothers more in the social environment of the rural schools. This includes organizing mothers-female teachers' associations and providing alphabetization courses. Furthermore, the gender dimension should be more pronounced in the project design of future projects. However, without solving the core problems of the recruitment of teachers proactively and the government living up to its promises made at project start, any future education project or component will have to face the same situation as CDP. Thus, efforts should be joined to overcome the practices that inhibit fair recruitment and, in particular, more involvement of female rural teachers.

(5) Other Lessons
For future projects, a Triple A approach should have been used in community mobilization from the very beginning of the project. The baseline study should include project and non-project districts. The supplementation of all micronutrients should be an integral part of the nutrition component.

The HRBAP approach should be considered as well as a pronounced gender approach in the planning stage of similar future projects. The country office can consider involving technical assistance for this approach.

Finally, the CDP has generated a number of important lessons and some good practices, which could be avoided or replicated in future projects, respectively. A booklet, summarizing the former, to share with UNICEF projects worldwide and other projects within Yemen, would be rather useful. A good example was provided by the UNICEF Bangladesh country office.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2005

Region:
MENA

Country:
Yemen Republic of

Type:
Evaluation

Theme:
ECD - Area Based Programme

Partners:
Government of Yemen, World Bank

PIDB:

Follow Up:

Language:
English

Sequence Number:
2005/001

New enhanced search