2009 Global: Evaluation of UNICEF Multiple Indicator Cluster Surveys Round 3 (MICS3)
The Multiple Indicator Cluster Surveys (MICS) managed by the United Nations Children’s Fund (UNICEF), were initiated in the mid-1990s to monitor progress toward the World Summit for Children (WSC) goals. They now help monitor progress toward the MDGs and many other global, regional and national commitments, as well as provide data used in making strategic program choices. Since the 1990’s, the program of household (HH) surveys has grown and changed rapidly and the visibility of the surveys and use of their results in myriad publications has increased dramatically. UNICEF committed itself to launch the 4th round of the MICS in 2009. The 17-24 months prior to the launch of the 4th round were identified as an opportune time to reflect on experiences to date and to make important decisions about future directions.
UNICEF commissioned this evaluation of the to specifically examine the MICS third round (MICS3) to judge if the results were justified by the expenses and commitments made, as well as to learn how best to use and improve the MICS initiatives and similar data in support of global goals and targets. The evaluation assessed the quality, governance and impact of the MICS. The evaluation was conducted by the independent consulting firm John Snow, Inc., (JSI).
The current environment for the MICS and other global monitoring tools is evolving rapidly, with new global partnerships increasing demand for data on program performance and progress toward global goals. Demands from donors span the whole gamut of information from cost-effectiveness information, to process and coverage measures, to measurable outcomes and overall impact.
The evaluation team found that data and analysis resulting from MICS3 was clearly an important part of the evidence base for programming and policy within UNICEF. The MICS3 is valued among global level partners for global monitoring and advocacy, particularly as related to the Millennium Development Goals (MDGs). At the country level, 9 out of 10 stakeholders interviewed felt that the MICS3 was an integral part of the country’s information base on women and children. In most countries examined, the quality of data was on a par with other ‘gold standard’ global household survey efforts such as the Demographic and Health Surveys. The evaluation team concluded that the quality of the MICS3, in general, in terms of the type and magnitude of nonsampling errors is quite similar to those found in global household survey programs.
UNICEF was able to make effective use of many aspects of its organizational structure, processes, and culture to make the MICS3 a high priority. The technical assistance, tools, materials, and training developed and provided by UNICEF Headquarters to support MICS implementation were held in very high regard. UNICEF Country Offices sought other donor support for financing. In many countries, multilateral and bilateral agencies contributed to the survey.
These positive findings have to be balanced against suggestions for further improvement and strengthening of the survey program. Although data quality in most countries was comparable with that in other global survey programs, significant data quality lapses were noted in several countries, as well as deviations from accepted norms in sampling and fieldwork procedures that did not adhere to recommended practice. A proliferation of indicators between MICS Round 2 and MICS 3 suggests weaknesses in processes and structures for strategic decision making. Final reports are often delayed for long periods to the point that the data are no longer seen as timely. UNICEF’s decentralized structure meant that critical technical decisions were negotiated by those with the least knowledge and experience in the conduct of household surveys. Stakeholders have serious concerns about a move to a three-year cycle of MICS implementation and the heightened demand for coordination of any such effort.
Recommendations for UNICEF that arise from this evaluation take two forms: (a) overarching recommendations and (b) more detailed technical recommendations. Technical recommendations appear within the relevant sections of this report while the overarching recommendations appear in the concluding section. Among the “big picture” recommendations, the evaluation team suggests that UNICEF do the following:
(1) Clarify and reinforce the primary objective of the MICS: at the global level for monitoring and advocacy and at the national level for advocacy related to child health and well-being priorities. UNICEF is strongly encouraged to resist the dual pressures of expanding sample sizes to generate subnational estimates and content particularly where indicators are not yet fully validated—by establishing parameters to better guide and support those decisions.
(2) Bolster quality assurance through several mechanisms. At the center of those efforts is the need to correct the “mismatch” arising from UNICEF’s decentralized organizational structure and its role in MICS decision making. As the team found, survey expertise resides at headquarters and, to a lesser extent, at regional levels, while the locus of technical decision making is at the country level. UNICEF is urged to rationalize decision-making processes so that those with greater expertise and experience are more closely involved, either directly or through consultant staff members. Resources for technical assistance should be expanded and processes formalized.
(3) Capitalize on its investment in the MICS to strengthen and appropriately expand the evidence base for children and women through means designed to complement the nationally representative household survey. For example, UNICEF should promote greater use of and build capacity in routine health and other information systems.
(4) Consider a move toward a permanently established information and coverage support center, including other data collection and analysis strategies focused on the situation of children and women in addition to the support of future MICS “rounds.”
These recommendations can assist UNICEF to capitalize on its investments in the MICS program to date. Looking ahead, the recommendations aim to create an important, lasting resource through which UNICEF supports countries to develop and fully use an evidence base to improve the situation of children and women. Such a resource would serve countries well—both in the medium term (e.g., with Countdown 2015 and MDGs reporting) and beyond.
Full report in PDF
PDF files require Acrobat Reader.