Evaluation database

Evaluation report

2009 Sudan: Evaluation of UNICEF-GOS 2002-2006 Country Health and Nutrition Programme



Author: Dr. Hongyi XU

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:
The baseline situation of women and children before the start of the 2002-2006 UNICEF Health and Nutrition Programme can be described from MICS 2000. This survey reported child (under 5) mortality and infant mortality in northern Sudan were 104 and 68, respectively. ARI, malaria, diarrhoea, combined with malnutrition were identified as the main causes of child deaths. A prevalence rate of 28.2% for diarrhoea, 16.7% for ARI were found in U5 children in northern Sudan in 2000.
MICS 2000 data also indicated that approximately 64.5% of children received a BCG vaccination by the age of 12 months. The coverage for measles vaccine was 51.5%. There was a considerable drop out rate for the second or third dose of DPT and Polio. The first dose of DPT and Polio was given to 65.9% and 74%, respectively. The third dose of DPT and Polio dropped to 46.2% and 44.2%, respectively. The percentage of children who had all eight recommended shots was only 26.4% in northern Sudan.
15.7% children under age five in Sudan were wasted (moderate and severe) and 3.8% were classified as severely wasted, according to MICS 2000. Approximately 31% of babies weighed less than 2500 grams at birth. About 44% of children received a high dose Vitamin A supplement. Approximately 19% of children aged less than four months in northern Sudan were exclusively breastfed.
Overall, 22.8% of U5 children were reported having fever in the last 2 weeks prior to the survey. Only 22.5% of febrile children got appropriate malarial treatment. Coverage of insecticide-treated nets within U5 children was only 2% though 24% of U5 children used a bed net, according to MICS. Only 0.6% of households had adequately iodised salt.
According to the SMS 1999, the MMR was 509 per 100,000 live births. According to MICS 2000, about 70% of the women in northern Sudan received antenatal care from skilled personnel. Skilled personnel delivered about 87% of births (midwives and trained-TBAs delivered 43% and 29% respectively). Only 21% of married women had ever used a method of family planning and 7% were currently using a method. Little activity had been done for HIV/AIDS in Sudan before 2002, though a prevalence of 1.6% was reported that year. The distribution of health facilities and health personnel was not equitable. On average, in 2002, less than half (43.6%) of all health facilities in northern Sudan offered RH services and some services, like EmOC, were seriously lacking.
The objectives of the evaluation were to carry out a comprehensive end of cycle review to gain an understanding of the successes and failures during implementation, draw lessons learned and make recommendations based on the findings. The methodology used included desk review, key informant interviews, cost-benefit analysis, and field visits.
Results showed that UNICEF deserved strong praise for its central role in the provision of primary health, immunisation, child health and nutrition, emergency essential heath care services in Sudan.
During the 2002-2006 programme cycle, UNICEF provided valuable support to the government in accordance with its obligation as set out under the Convention of the Rights of the Child, to protect and promote the rights of children to survival, development, protection and participation.
The dimension of the assistance from UNICEF was comprehensive. These dimensions included advocacy at policy level, service delivery on the ground, institutional support at federal and state level, and capacity building of service/care providers. UNICEF supported the establishment and rehabilitation of health facilities and provision of equipment, supplies and essential medicines. Routine operation and campaign activities were supported, as well as fund raising, programme design and planning, joint monitoring and supervision. In addition to these activities in focus states, the programme also covered activities in non-focus states with respect to projects of national thrust (namely EPI, emergency planning and response, malaria control, HIV/AIDS and micronutrient supplementation).
The years 2002-2006 were a period during which the country had not yet recovered from its decades of conflict. It was a period when natural and man-made emergencies frequently occurred with considerable impact on the population. The period also carried a mix of frustration and hope, ups and downs; during which many sectors demanded development and was a period in which primary health, child and maternal health care were waiting to build momentum. Despite the many constraints the programme had faced, it ended by fulfilling some of its targets and reaching noticeable achievements.
The major achievement of the programme was progress in immunization. UNICEF consistently provided leading support on routine immunization service, ranging from expansion of fixed sites delivery immunization, provision of vaccines and cold chain equipment, to training vaccinators and programme managers. As a result, the capacity of routine immunization in Sudan has largely improved across planning, management, information reporting, and surveillance. Health facilities with EPI service increased from 40% to 75%. Overall coverage of routine immunization, DPT3 as an example, increased from 70% to 85%. Together with other partners including WHO, UNICEF provided support on supplemental immunization activities such as polio NIDs, measles campaigns, and vaccination activities to prevent diseases such as meningitis, yellow fever, hepatitis, and tetanus. Though polio was not eradicated, reported cases and outbreaks which had been devastating in the past were largely avoided.
The second achievement of the programme was the effort exerted on Malaria control. During the latter half of the programme period, reported malaria cases and deaths were continuously decreasing. This was largely due to the combined improvements on prevention, diagnosis and treatment. UNICEF contributed to all these preventive and curative measures, with supplies of long lasting Insecticide treated nets, provision of ACT anti-malaria drugs, capacity building and upskilling of lab technicians and health workers, and with institutional support to malaria departments and their human resource development. During field visits, it was observed that primary health care providers showed confidence and skills in handling malaria cases. Supplies and management were an integral part of routine service delivery.
During 2002-2006, there were many emergency situations, due to conflict, natural disasters such as flooding, disease outbreaks, and Darfur crisis. UNICEF proved to be always available whenever and wherever the need was required. In addition to technical assistance and capacity building support, UNICEF was responsible for provision of supplies, such as life saving and essential drugs, vaccines, supplementary & therapeutic feeding supplies, PHC kits, health education materials, safe delivery kits, mosquito bed nets & insecticides, Non Food Items such as jerry cans, and water treatment. The coordination procedures and processes within UNICEF and externally with its partners have ensured that it fulfilled its core planning commitments towards emergency situations in Sudan, by conducting an assessment of the emergency situation within 48 hours, and being able to release prepositioned supplies to assist 35,000 persons within 10 days, for a period extending to 2 months.
UNICEF supported the primary health care system, through the establishment and rehabilitation of health facilities, provision of PHC kits and other equipment, the training of first-tier health providers to deliver essential care to address the main child diseases responsible for preventable child mortality. UNICEF, together with UNFPA, shared the responsibility to support midwifery school training, provision of midwifery kits, and installation of EmOC services. This effort, though still need more inputs to yield major reductions in maternal mortality, ensured greater access to quality basic obstetric care for many women previously without access. A number of nutrition surveys and sentinel surveys in northern Sudan were conducted with the support of UNICEF. UNICEF also supported Vitamin A supplementation through Polio NIDs, iodized salt consumption through policy and legislation and provision of iron supplementation to pregnant women. Through the activities of NGOs, the operation of SFCs and the TFCs were jointly supported by WFP and UNICEF. Though this strategy might not be the long-term solution to reverse children’s malnutrition status in conflict areas, it ensured that child malnutrition was not deteriorating.in the country.
The HIV/AIDS intervention during the cycle targeting vulnerable groups, such as youth and women, was slowly rolled out at the beginning, but was in a better position at the end of programme cycle. Seven PMTCT centres were established and operational in South Darfur, Khartoum, Kassala, North Kordofan, and Red Sea states. Those who tested HIV positive were subsequently provided with ARV treatment.
The GoS-UNICEF 2002-2006 programme did not achieve an impact on reducing child mortality and maternal mortality rates, based on the limited data available. The broader impact of the programme , which could not be measured in numbers, was evidenced in process indicators, like improvements in ANC delivery, deliveries by skilled personnel, reductions of disease burden and PHC provision to women and children. Positive effects of inter-sectoral activities between health, education, water and environmental sanitation in addressing mortality rates were also observed.
There was a noticeable impact on the programme environment. A variety of policies and plans were designed at both National and State levels. At the grass root level, community was aware of the Programme. The concerns of families about child immunization, child nutrition, child rights and disease prevention were strengthened. Despite its fragility in some areas, a peaceful environment became conducive to the various activities of the Programme. Previous areas of conflict started to return to normal allowing various activities to be sustained.
In light of the SHHS results, and as part of the extensive joint planning exercises undertaken across the health sector, key opportunities to have a distinct impact on the overall environment included promotion and implementation of the Accelerated Child Survival Initiative (ACSI). ACSI attempted to scale up existing projects, especially the Expanded Programme on Immunization and nutrition, while identifying capacity and resource shortfalls and prioritizing specific activities at local, state and national levels.

Purpose/Objective:
The objectives of the evaluation were to carry out a comprehensive end of cycle review to gain an understanding of the successes and failures during implementation, draw lessons learned and make recommendations based on the findings. The methodology used included desk review, key informant interviews, cost-benefit analysis, and field visits.

Methodology:
The following methods were used: desk review, key informant interviews, cost-benefit analysis, field visits and data analysis.
3.2.1 Desk review
A full range of documents related to planning and implementation, reporting, monitoring, reviews (including reports from field-visits), were made available for the review. The 2002-2006 Annual Work Plans and Annual Reports were the main documents reviewed, which described targets, provision of supplies and human resource capacity building activities in the three components of the programme, and the yearly progress. The routine, quarterly reports/reviews were utilised to examine the routine monitoring and delays experienced during each year. The programme policy and strategy plans were reviewed to example the appropriateness and coherence. Additional documents used included assessment and evaluation reports, 2006 Sudan Household Health Survey, 2000 MICS survey, 2005 Malaria prevalence and coverage survey, and several other assessments conducted by the UNICEF and Ministry of Health and partners on malaria and immunization activities. A complete list of documents utilised is provided for reference.
3.2.2 Key informants interview
Interviews were conducted with key national and state programme officers. At federal level discussions were conducted with the Assistant Under-Secretary for Planning, Policy and Research, Assistant Under-Secretary for Preventive Medicine and Primary Health Care, the Director General of International Health, the National programme managers of EPI, IMCI, RH, Nutrition and Malaria. At the State level, the Directors General of SMOH and State coordinators and managers of EPI, IMCI, RH, Nutrition, Malaria and CFCI were interviewed. The focus of these interviews and discussions was on assessing the management (administrative and financial) of the health and nutrition programme and the impressions on the achievements of the programme.
The programme had multiple partners and stakeholders, including communities to whom the programme interventions were intended. The Federal Ministry of Health was responsible for development, implementation and monitoring of the health and nutrition programme. Other key partners included UN agencies such as WHO, UNFPA, and other stakeholders. It was important to look at the partnerships of the programme and understand their strengths, and weaknesses. Therefore, key informants interviews were conducted with programme officers from UN organizations, international and national NGOs, as well as the community beneficiaries. Some of the interviews were conducted during field visits. A semi-constructed questionnaire was used to guide the interviews and discussions. The interviews were mostly conducted one by one, but were also conducted and combined with group discussions particularly at programme level.
The comments on programme partnership, objectives, obstacles and constraints, contributions, issues such as sustainability, policy, strategy and planning, implementation, coordination and communication, reporting, monitoring and supervision, as well as suggestions, were obtained during interviews with key informants guided through the questionnaire. Though the key national and state informants are mostly programme managers or officers with fair understanding of the collaborations between UNICEF and the country, some of them are newly assigned and were lack of information on the events on the previous programme cycle. Group discussions with UNICEF programme officers were conducted afterwards to discuss the comments and issues raised during interviews, to fill in the gaps.
Community beneficiaries such as women and children were interviewed during household visit. With permission from the chief of the village, female interviewers were allowed to visit the village huts. Normally women and children presented in the house at the time. The condition of the huts was examined. The interviews with the women and children beneficiaries (in this study, chosen conveniently during huts visit), focus on the means of living, children status and welfare, the availability of the health service, immunisation and common disease treatments.
3.2.3 Cost-benefit analysis
The cost-benefit evaluation was challenging since the programme had multiple partners. The important periodic data on financial expenditure linked with outputs was inconsistently completed to sufficient detail. Efforts were made to collect additional cost data at Federal, State and Facility level. Though some information was collected, in most cases it proved to be fruitless in terms of the details and quality of the information.
In the end, available child health costing data was used, to undertake cost-benefit analysis in key programme components such as immunization. Since the programme had multiple partners, estimations and assumptions were used to analyze the cost, to differentiate the contribution of UNICEF. As in many programmes, it was difficult to determine the exact outcome/impact of programmes attributable to UNICEF. On the other hand, regarding the extent of support, the feedbacks from the National counterparts was that UNICEF was a major contributor to Sudan’s routine immunization service and its coverage achievement. The cost of immunization was recalculated with an attempt to breakdown the routine immunization cost, the supplementary immunization cost, the recurrent cost of both services, and the capital cost of the immunization service, based on the cost of the child health data. The cost, matched with the output, were analyzed, the unit-cost was presented as well.
3.2.4 Field visits
GoS-UNICEF 2002-2006 health programme had 9 focus states, and health and nutrition interventions were conducted in the states and localities. Observation of the local situation, programme accomplishments and interaction with the local population was an important part of the evaluation of the implementation of the health and nutrition programme at State and Locality levels. Three states were visited, Kassala to represent the East, South Kordofan to represent the transitional states, South Darfur to represent the West.
The primary health unit was a key health facility that was visited, since it constituted the backbone of public health infrastructure in Northern Sudan and provided an obvious site where UNICEF had inputs in the primary health system. The GoS-UNICEF 2002-2006 programme implemented key strategies, such as focus on service delivery and community empowerment, and selected the most disadvantaged communities (CFCI communities) to deliver UNICEF’s support. Following consultation with programme officers in the health and evaluation sectors of UNICEF Sudan Country Office, one CFCI community and health facility available to that community was visited in each of the three states, Visits to one nearby non-CFCI community were conducted for comparison. Interaction with community leaders, was undertaken, as well as visits to some households (one relatively wealthy, one poor) to interact with beneficiaries such as women and children to gain an understanding of their views, and social and culture backgrounds. The observations during field visit were described in the findings.
Three days were arranged in the field, with Day 1 and/or Day 2 comprising discussions with key state stakeholders, and Day 3 spent on community and community health facility visits.
Due to the distance and time limitation of each visit, as well as security concerns in some areas, the selection of the CFCI community was not random. The CFCI community selected was generally a disadvantaged area and relatively close to the city (though some required more than two hours driving time from the city during dry season).
A semi-constructed questionnaire was used to collect information and comments and a set of data collection instruments targeting State and Facility level was used to collect quantitative information.
3.2.5 Data collection, verification and analysis
Attention went primarily to available data before attempting to collect data at the Federal, State, and Facility level, so as not to duplicate the data collected by targeting different levels of the programme on specific information.
Available data from routine reports and population surveys was analyzed first. The available data and indicators had various problems on quality. An extra section was added to comment on data quality, followed with some suggestions to improve quality in the future.
During data analysis, a set of indicators output/outcome/impact indicators were chosen based on the initial plan and internationally recognized guidelines. The available routine data, the reported data and the population data were cross-checked. The field visits were used to collect additional information for further verification.
To examine the outcome and impact, population survey data were given higher credit, based on consultation with peer professionals, observations from field visits, and the limited choice we had.
MICS 2000 and SHHS 2006 data were used, as well as programme surveys such as the Malaria survey 2005. As some of these surveys were conducted during periods when boundaries were different. (In instances, boundaries included a few selected towns of the South), we had to re-analyze the survey data to obtain northern Sudan results. To closely examine the trend and situation in the focus states, a state by state comparison was conducted (Annex 2). States like West and South Kordofan which existed in 2000 were aggregated to compare the newly created South Kordofan state in 2006. The method of allocation was inexact as parts of West Kordofan were assigned to North Kordofan, and the rest went to South Kordofan.

Findings and Conclusions:
There were many partners with whom UNICEF was working, categorized into the following groups: the recipient government, donor governments, UN agencies (namely WHO, UNFPA, WFP), NGOs, and local communities,. The main stakeholder for UNICEF-GoS country programme was GoS. The majority of funding of UNICEF-GoS 2002-2006 was raised by UNICEF itself. In some instances, funds provided had restrictions. For example, some funds were specified to support rehabilitation of health facilities, though on the ground the influx of returnees required new facilities and some deteriorated facilities were not a priority to rehabilitate, since the population had moved . To make sure the needy, vulnerable population were served in a timely manner, UNICEF made noticeable efforts to use a balance and fair approach to resource distribution and to be flexible to ensure timely support.
The three components of the Programme: the EPI project, Integrated Maternal and Child Health project, and the Healthy Growth and Nutrition project were targeted to achieve 90% coverage of routine immunization; 80% coverage of minimum care to children and pregnant women; 80% coverage of iron, vitamin A supplementation and iodized salt use within pregnant woman, U5 children and households respectively, and 50% reduction of malnutrition rates. Though some of the targets were ambitious at the time, they tended to tackle the fundamental child health problems in northern Sudan.
EPI indicators were the mostly clearly defined indicators, but some of them were not equally specified. In addition to outcomes indicators, process indicators were also needed to track periodic progress. There was no well laid plan at the beginning of the programming cycle regarding (i) continuous data collection, (ii) frequency of data collection, (iii) body responsible for collection, (iv) collections forms to be used, ,and the (v) verification of data collected.
For example, some indicators were designed to measure the coverage in the focus states, but were reported inconveniently as nationwide, or statewide, or locality/communitywide values. There was not consistency in application of indicators. In some instances, indicators were reported as a number, others as a percentage, and as an estimation, though it was not possible to determine how it was estimated. Since the administration was continuously changed (at Federal, State and Locality level, and at International level), and there were also population changes through movement (IDPs and returnees), number or boundary change on states or localities, lack of consideration of these changes at the beginning of the programme made the tracking, checking and reviewing the periodical progress difficult.
Another example of ambiguity of the indicators is evident in the final objective of reduction of malnutrition of 50% among children in the focus state. It was not clear from the beginning if/how 50% reduction could be achieved and if it was realistic. Though there were surveys and community monitoring conducted during 2002-2006, there were no plans on information use and therefore the information seemed not to inform further intervention. Though the nutrition programme mentioned that the intervention should be comprehensive, not just a problem of supplementary feeding, it did not tell what comprehensive exactly meant, except for vitamin A to children through NIDs, iron supply to pregnant woman through campaign, iodized salt to households through policy enforcement.
4.1.2 Program strategies
The GoS-UNICEF 2002-06 programme pursued the following broad strategies to achieve the main objectives of the programme.
Strategy 1: Child rights and peace building
This strategy formed the normative framework for all interventions. All components of the programme provided a base for protecting child rights, promoting conflict resolution and grassroots peace building. Examples included: the selection of intervention communities with a view to furthering rights protection and peace promotion; education and information activities in support of these priorities, and strong national partnerships and alliances across all sectors in support of children’s and women’s rights.
Strategy 2: Focusing on needy areas
In addition to the focus states, the programme covered also the non-focus states with respect to projects of national thrust (EPI, emergency response, malaria HIV/AIDS and micronutrient supplementation).
Sector field interventions and community-based initiatives of the programme converged on the most disadvantaged states and communities.
Firstly, key indicators from MICS 2000 and SMS 1999, including infant mortality, child mortality, maternal mortality ratio, literacy, female literacy, primary school enrolment, Immunisation coverage, malnutrition rate, access to safe drinking water, clean safe delivery, HIV/AIDS knowledge, etc., were used to rank and select nine “focus states” as the most vulnerable in northern Sudan (Figure 3). Red Sea state was added during the programme cycle. UNICEF zonal offices were strengthened to assist and monitor implementation. The population in the combined focus states was estimated between 13 -15 million at the time

Recommendations:
 Explore the capacity of current TFC and SFC facilities, with active identification and referral of malnutrition cases to increase enrolment. Improve the supervision of current facilities.
 Provide and scale up cost-effective approaches such community level nutrition promotion and practice.
 Integrate interventions with routine primary health care and community health. Continuous capacity building to community level service providers such as midwives, community health worker, medical assistant, etc. Enhance the supervision.
 Enhance policy and its implementation at state level such as iodized salt.
 Identify innovative integrated approaches to prevent and control child malnutrition. This could be done by strong spectral linkage with hygiene promotion, poverty reduction, food security, and mothers’ income generation and education.
 Strengthen nutritional surveillance; improve the quality of the data, the dissemination, and its use.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search