2009 Global: Global Evaluation of DevInfo
“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”
The Development Information System (DevInfo) is a database system for monitoring human development. It is a tool for organizing, storing and presenting data in a uniform way to facilitate data sharing at the country, regional and global levels across government departments, United Nations (UN) organizations, civil society organizations and development partners. DevInfo has features that produce tables, graphs and maps for inclusion in reports, presentations and advocacy materials. The software supports both standard indicators, including indicators for the Millennium Development Goals (MDGs) and user-defined indicators. DevInfo can operate both as a desktop and a web-based application.
DevInfo originated in ChildInfo, a database system for monitoring child survival, growth and development managed by UNICEF. In 2004, the Executive Director of UNICEF offered to share UNICEF’s experiences with using ChildInfo to collect data and report on indicators with other UN organizations. DevInfo is currently a global programme funded by eight UN organizations and managed by the DevInfo Global Administrator, who is a staff member of UNICEF. The main oversight body is the DevInfo Inter-agency Advisory Committee, which reports to the Working Group on Programming Issues within the United Nations Development Group (UNDG). The implementing partner for DevInfo support is the Community Systems Foundation, a not-for-profit foundation. Some support is also provided by the United Nations Development Operations Coordination Office (DOCO) and through regional support structures.
The objectives of the DevInfo initiative are: to anchor DevInfo implementation in a wider national effort to collect, analyse and disseminate information on human development to ensure long-term sustainability and usage of the database system; to strengthen the capacity of governments to adapt DevInfo to their national monitoring strategies and thereby effectively use DevInfo technology to monitor progress in human development; and to strengthen the capacity of UN country teams and national partners to effectively use DevInfo in monitoring progress on national and international priorities, such as the MDGs.
Purpose and objectives of the evaluation
The purpose of the evaluation is “to inform UNICEF senior management in their decision making and the DevInfo Inter-agency Advisory Committee concerning the way forward for the initiative, notably as far as the future management and implementation structure of the initiative and its sustainability and possible expansion are concerned.” This evaluation is expected to help senior management explore ways in which DevInfo might develop and possibly scale up and accelerate the distribution and use of DevInfo.
The evaluation addresses three objectives: Objective 1—Assess relevance, effectiveness and impact of DevInfo at national and sub-national levels by governments and other partners in programme countries as well as by the UN system; Objective 2—Assess efficiency in terms of oversight, management and implementation of DevInfo; and Objective 3—Explore options for institutional sustainability and possible opportunities for expansion of DevInfo.
The timing of this evaluation was prompted by: the need to take stock and plan ahead at the end of DevInfo’s first five-year cycle (2004-2009); a request by UNICEF Headquarters’ Contracts Review Committee to assess the impact of DevInfo prior to the next contract reviews with the Community Systems Foundation in early 2009; and planning for the UNICEF biennium 2010-2011. The evaluation starting point was taken as the launch of DevInfo in 2004.
The overall approach for this evaluation was to adopt a broad strategic view of DevInfo and then focus on critical factors affecting its performance and perceived usefulness. Five research methods were used: desk review of key documents; interviews with stakeholders; six country case studies (of Egypt, India, Liberia, Malawi, Serbia and Thailand); UNICEF regional office visits and consultations (with Bangkok, Thailand and Panama City, Panama) and telephone interviews (with Amman, Jordan; Dakar, Senegal; and Geneva, Switzerland); and an online global user survey using a sample frame from the DevInfo Support Group trainee database. Information obtained was triangulated.
The evaluation faced a number of limitations. Only six countries could be visited by the team, but an effort was made to have a reasonable geographical spread, although French and Spanish speaking countries were not visited. The user survey had to use a limited sampling frame, which meant that the findings of the online user survey were not representative. Restriction to scalable closed questions reduced qualitative review, and restriction of the survey to English, French and Spanish may have limited the access for some respondents. The DevInfo website is in the process of redesign and so was not reviewed, nor was the latest version of the software, DevInfo 6.0.
A general assessment was made of developing industry trends in data presentation and display, but the evaluation did not explore in depth how such tools might evolve in the next three to five years and what this might mean for DevInfo.
Key findings and conclusions
Relevance. There has been growing demand during the past decade from policy and decision makers around the world for sound and accessible statistical data, notably data related to national development goals and the agendas of Internationally Agreed Development Goals, including the MDGs. DevInfo is a relevant initiative that contributes to the standardization of statistical data that are indispensable for sound policy and decision making related to human development within and among countries. Its value is anchored in the normative and analytical mandates of the UN system, and this represents its comparative advantage in an environment of rapidly evolving and competitive innovations in information technology.
Display functionalities of DevInfo have proven to be effective tools for raising interest of policy and decision makers in statistical evidence. The relevance of DevInfo adaptations will depend on both their additionality and complementarity to existing and nascent systems as well as on their quality and timeliness of data.
Effectiveness. The three key performance monitoring indicators for the DevInfo initiative were set out in 2004: number of countries using DevInfo; number of adaptations of DevInfo; and number of trained users. As of March 2009, more than 100 developing countries were using DevInfo, more than 140 versions had been adapted, and more than 12,500 people had reportedly been trained on the system. These three indicators address effectiveness mostly in terms of outputs and, to a limited extent, address outcomes and impact. There is evidence that national DevInfo databases are being used to improve the capacity of results-based planning and monitoring of national development priorities. In the user survey, 83 percent of respondents indicated that they agreed or strongly agreed that DevInfo had been used for presenting statistical data to decision makers.
This evaluation sought to assess the actual use of DevInfo within the UN system and, more importantly, by national partners. However, evidence concerning results at the outcome and impact level was limited, as there is no systematic system for monitoring and reporting on such developments. Such evidence would depict institutional or behavioural changes that could be attributed to or associated with DevInfo. While the current approach of training a large number of people on DevInfo (a reported 12,500 since 2004), many of whom will not actually use it, and providing support to users upon request has produced some results at the outcome and impact levels, it has major limitations and does not allow DevInfo to be used to its full potential.
The proprietary software and closed development model currently used by DevInfo is challenged by the open source model, which is defined as computer software for which the source code and certain other rights normally reserved for copyright holders are provided under a software license that meets the Open Source Definition or is in the public domain. User needs and demands have required periodic adjustments and three updated releases of DevInfo since 2004.
DevInfo’s use to improve the capacity of results-based planning and monitoring national development priorities is due, in part, to the general DevInfo template, which comes preset for global MDG indicators. National adaptations of DevInfo are structured around the global and, in some cases, nationally adapted MDG indicators. In some countries, MDG reports are prepared using data from national adaptations, and some national reports use the mapping and presentation features of DevInfo to produce graphics and presentation materials. However, the use of DevInfo data for MDG reporting is not universal. While the MDG networks advocate use for reporting, agencies do not systematically support it. Thus the full potential of DevInfo for MDG reporting is not reached.
Oversight and management. The DevInfo initiative is a global programme managed by UNICEF with support from other UN organizations and DOCO. The main oversight body is the DevInfo Inter-agency Advisory Committee, which is organizationally located within the UNDG Working Group on Programming Issues. The group aims to contribute to UN reform by striving for greater coherence and harmonization of approaches within the UN system.
The oversight and management structure has its limitations and may not be appropriate if DevInfo is to evolve to its full potential. It is questionable whether the DevInfo Global Administrator should chair the DevInfo Inter-agency Advisory Committee, which could result in a potential conflict of interest. Membership in the Advisory Committee is mostly at a technical level and does not involve senior levels of management. The link between the mandate of the DevInfo Inter-agency Advisory Committee and the overall policy framework of the UNDG Working Group on Programming Issues is relatively weak. The present oversight and management structure does not allow DevInfo to respond to the expectation that it support the linkage between monitoring of country-level national development results and monitoring of the United Nations Development Assistance Framework. The DevInfo Inter-agency Advisory Committee is currently not capable of proposing a strategy to achieve these objectives.
At global, regional and country levels, DevInfo is still strongly perceived to be a programme sponsored, managed and implemented by UNICEF. For support to member countries, UNICEF depends to a large extent on a single service provider—the Community Systems Foundation, a non-profit organization based in the United States, with the development work being carried out by Avalon Information Systems Pvt. Ltd., a company based in India. While the commitment and high quality of services provided by the Community Systems Foundation are acknowledged, this evaluation endorses the recent observations of the UNICEF Office of Internal Audit that drew attention to the risks involved in the contractual relationship with a single service provider. However, the risks should be considered in the inter-agency context of the UNDG. Any exploration of alternative arrangements should also take into account the possibility of certain services being provided by regional centres. The experience of the Latin America and Caribbean region is instructive in this regard.
Funding. DevInfo coordination and support has so far relied exclusively on financial resources from within the UN system, with UNICEF covering a major share, which is, however, decreasing in relative terms. While there are no indications that UNICEF or other participating UN organizations would withdraw or decrease their support to DevInfo, there is currently no institutional framework or organizational set-up to guarantee sustainability of the programme funding beyond 2009.
Overall conclusion. The DevInfo business model has streamlined its procedures as it has evolved. It has also been successful in terms of mobilization of internal resources and more equitable burden-sharing within the UN system. However, it is still unnecessarily burdensome for both UNICEF and the Community Systems Foundation. Most important, it has not been conducive to developing a medium- and long-term vision in response to the challenges related to a more systemic and comprehensive strengthening of monitoring and evaluation capacities in developing countries.
DevInfo as part of monitoring and evaluation capacity development. DevInfo should be more strategically embedded in monitoring and evaluation capacity development activities by the UN system at the country level. Specifically, this should involve developing statistical capacities to monitor human development indicators (including MDGs) of national statistical offices and other institutions involved in data processing and dissemination. Database and display functionalities of DevInfo allow it to make a major contribution to such efforts. DevInfo needs to demonstrate its added value and comparative advantage in relation to other tools for MDG monitoring and measurement of development effectiveness. This needs to happen globally and at the country level.
More systematic UN support to monitoring and evaluation capacity development at the country level will require an explicit demand from national partners to ensure national ownership and leadership in the process and a high degree of responsiveness from the Resident Coordinator and the UN country team. Such capacity development should ideally be part of the Common Country Assessment/United Nations Development Assistance Framework process and possibly be articulated as a joint programme of the UN system at the country level. Given its monitoring and evaluation capacity in country offices, UNICEF could assume the role of lead agency or administrative agent. Such a demand-based and integrated approach should be piloted in a limited number of countries that show interest and commitment.
If monitoring and evaluation capacity development is recognized as an important part of the role and contribution of the UN development system at the country level, this may require a substantive revision of the Common Country Assessment/United Nations Development Assistance Framework guidance issued by the UNDG Working Group on Programming Issues with DOCO support. A renewed commitment by UNDG may entail strengthening the role of DOCO in the administration of DevInfo.
For DevInfo to become a truly joint programme of the UN development system at the global level, there is a need for senior management of the UN system, specifically UNDG, to renew their commitment to the system. The Global Administrator, in consultation with the DevInfo Inter-agency Advisory Committee, should prepare a strategy document that would outline a renewed vision and objectives for DevInfo in the coming 5 to 10 years. The document should be submitted for approval by the Working Group on Programming Issues or the UNDG senior management meeting.
A consultative process with member countries and UN country teams on the draft strategy document will be required to ensure that DevInfo remains, to the greatest possible extent, demand-driven and responsive to needs and priorities at the country level. Such consultations could be ensured, for example, through the UN Statistical Commission of the General Assembly (which recently launched CensusInfo) and possibly the Economic and Social Council (ECOSOC). Another consultative process should be launched with Resident Coordinators and UN country teams, especially in countries that have mature national adaptations of DevInfo.
DevInfo software development and technical support. There are an increasing number of applications being built using open source technologies. However, the important debate is not about open source versus proprietary software but about how to increase interoperability and to work within open standards. The development of DevInfo started with proprietary software development tools, but it has been increasingly using open software development tools. The DevInfo initiative has made a significant contribution to interoperability through its leadership role in the implementation of emerging international standards for socio-economic data (including SDMX, DDI and the UN Geographic Information Working Group). DevInfo could now shift to allow DevInfo technology to be available to software developers who could then take the currently available DevInfo 6.0 Software Development Toolkit even further.
Such an approach would be highly compatible with principles of national ownership and leadership in development and the specific role of the UN system in capacity development, including facilitating exchanges of innovations among countries and reducing dependence on commercial service providers whenever feasible and advantageous. DevInfo is also not at risk of losing its unique role and contribution, which is related to data dissemination and visualization, as the DevInfo database technology is not limited to one software development model or another, as long as the DevInfo database model is compliant with the agreed international standards. Aware of the growing interest in open software development platforms, DevInfo management has decided to commission an open source solution for DevInfo in 2009 in addition to the current platform. This open source version will provide developers with the source code, technical documentation and a governance mechanism for software change management procedures for the official reference version. This will also include strengthening technical oversight and support as part of the governing and guidance structure, with the intent that DevInfo technical innovations will emerge from an exchange among developers in different countries facilitated by the DevInfo Support Group and other DevInfo support structures.
Funding and sustainability. The inter-agency funding mode of DevInfo should be pursued and equitable burden-sharing among UN organizations should be ensured. Given donors’ and other funders’ interest in quality data in developing countries, there is a chance that they may be willing to invest in systems that are conducive to improvements and national capacity development in this regard. The establishment of a UN Trust Fund, with funds pooled by major stakeholders, is a model that has worked for other initiatives.
Conditions for a possible expansion of DevInfo. DevInfo has been a relevant and potentially effective tool to generate and disseminate statistical evidence related to human development in support of policy and decision making. With a presence in more than 100 countries and a proliferation of adaptations and themes, the question is whether DevInfo should expand further or consolidate and become more coherent and focused.
The current business model is still rather dependent on individual ‘champions’ both at the global level and in regions and countries. In addition, a longer term vision and possible expansion of DevInfo is limited by the facts that fund-raising is limited to internal sources within UNICEF and the UN system and that work planning and contractual agreements are with a single service provider on an annual basis.
Possible expansion would have to be considered in the context of broader UNDG engagement in a comprehensive and multi-year capacity development programme concerning monitoring and evaluation, in which DevInfo could play a major role. Placing DevInfo in such a broader framework would entail major changes in fund-raising modalities (including the establishment of trust funds) and institutional arrangements (including a strengthened role for DOCO), and considering changes in the DevInfo support role in an environment evolving to open source models.
Full report in PDF
PDF files require Acrobat Reader.