2012 India: Evaluation of Knowledge Community on Children in India Internship Programme
Author: Sonal Zaveri
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 2' of the report."
The purpose of the evaluation is to answer the question ‘whether the IP is an appropriate approach to knowledge management for UNICEF India which continues to move upstream and is increasingly expected to play a role as a policy organization’. Started in 2005 and into its eighth year, the IP has not been evaluated and as UNICEF India moves into its next programming cycle 2013-2017, the evaluation will address whether the original objectives of the IP were met and to what extent it contributed to the knowledge management results of UNICEF India. The IP program invests about USD150,000 annually (In 2009, the program cost was USD 217,000 for about 80 students and in 2011 it was USD 142,069 for 41 students). [... more in report]
The India Summer Internship program was conceived as part of a broader initiative to build a Knowledge Community on Children in India (KCCI), UNICEF India’s first knowledge management (KM) efforts initiated in 2005. The importance of KM to UNICEF’s work in India has been increasingly recognized and one of the key strategies for achieving the results of the 2008-12 GOI-UNICEF Country Program was “providing technical assistance and support to improve KM systems and sharing of lessons learned.”1 The ultimate goal of KCCI is to learn lessons from the field in order to improve programming and eventually influence policies related to children.2 The KCCI includes four components: 1) Summer Internship Program 2) Publications 3) Website and 4) Resource Center. The purpose of this evaluation is to evaluate the Summer Internship Program. The current UNICEF India Five Year Country Programme is from 2008 to 2012 and the evaluation of the IP follows the same timeline i.e. from 2008-2011
In the IP program, UNICEF India partners with research and academic institutions to encourage young students and scholars from around the world to engage in development issues pertaining to India's children and women. The assignment involves a combination of desk and fieldwork at the district or village level and/or at state and union level, with interns being grouped into teams that are based in field offices and work under the supervision of selected research institutions/non-governmental organizations. Every year between 40 to100 interns Indian and international, selected through a rigorous selection process, representing over 40 countries and with a Masters or PhD in development related disciplines participate in a ten-week long IP usually in the months late May to early August. The UNICEF India Internship program is not an individual assignment; teams of four work on different assignments in different states; national and international interns are included in each team providing opportunities for learning and sharing at multiple levels – personal and professional. The internship program is a planned study with a case study (print and other media) as an expected output. Further, case studies are reviewed and only those that pass the quality criteria (adherence to TOR, good understanding of broader context, describes value added to current programming practice, process and results describe actual changes in physical condition/behavior/attitude of project stakeholders, quality and source of quantitative and qualitative evidence, strategy and implementation describes successes and potential for replication) are published and disseminated. The case studies are available in print as well as on the KCCI website.
The Social Policy, Planning, Monitoring and Evaluation (SPPME) team manage the Summer Internship Program. It includes the Knowledge Management Officer3, who spends the majority of her time (70%) managing the Internship program assisted by hired consultants during phases of the internship such as selection of interns and managing the ten-week internship program. A Research and Evaluation Specialist who provides 10% of her time assists the PO to supervise the program4.
The original objectives of the IP as formulated in 2005 were to:
a) Produce high quality documentation of innovations in service delivery, what works (and what does not), why, and under what conditions, so as to use this to influence and inform the development discourse and disseminate evidence based research amongst government functionaries, development practitioners and policy-makers
b) Collect field-work based data and information relating to children across India which will offer comparable snapshots into progress made in selected issues relating to children’s development, as well as complement UNICEF’s documentation needs.
c) Support and develop a cadre of young research and development professionals with interest, commitment and skills relating to children’s development in India
d) Develop partnerships with leading research institutes in India and promote greater interest in social development research.
e) Promote awareness about UNICEF activities among graduate students in India and
The evaluation was conducted from May to September 2012 for a period of 30 days. The sample was 109, including 67 interns, 15 host organizations, 16 UNICEF managers, 2 from IP secretariat, 3 from KCCI management and 7 key decision makers. Data was collected through online survey, which was piloted and pre-tested using Zoomerang, and remote interviews. The government representative was not able to confirm an interview appointment.
Findings and Conclusions:
The report presents findings according to relevance, efficiency, effectiveness, sustainability and possible impact.
1) Relevance: The IP program is to a great extent relevant to and able to respond to the KM needs as outlined in the various documents. The IP program contributed a small but important niche towards achieving India’s KM results especially in knowledge generation and to some extent in knowledge use. There is a mismatch regarding UNICEF and host organizations’ perception of the relevance of different IP objectives reflecting a lack of coherence about IP. The IP partially contributed to the knowledge generation of other IRs; however it is difficult to track use. There is a rigorous process of selecting the best interns, nationally and internationally, use of criteria to ensure quality of the KM product (intern’s case study) and strategy for dissemination (print and web). Use is dependent on many factors of which knowledge generation is just one. The IP relevance to the new country program is clear in terms of its contribution to knowledge management on issues of importance to children. In terms of the structure and design, the IP is eminently suited for KM but its design and management to a small extent only encourages use.
2) Effectiveness: There were limitations in the ability of the host institutions to support the interns in the knowledge and skills needed for the development of the case study. Host organizations need to be selected according to their research capacity, content knowledge and ability to have a committed mentor for the interns. The host should facilitate and guide and provide the space for interns to complete their research. The quality of the case study was dependent on the clarity of the TOR, strength of the research design and timely support of both UNICEF state offices as well as the host. Limited participation of government and host organization in developing the TOR was seen as a gap. UNICEF’s regular follow up during data collection and feedback on interns’ draft case study was considered important to ensure quality and additional quality criteria were identified. Streamlining host-UNICEF-intern relationship regarding management of interns’ study, clear and consistent communication about study objectives amongst all stakeholders, use of knowledge product and providing the space to interns to independently express their findings were important for the effectiveness of the IP. Dissemination and advocacy for use of case study has not targeted key decision makers, the government or community.
3) Efficiency: The IP program may be considered as efficient as the cost of implementing the IP program has remained more or less consistent over the years. Stakeholders such as interns and host organizations contribute to the IP resources. The interns particularly the international interns fund their own travel to India. The host organizations receive modest support to house the interns and for related expenses of fieldwork. Case studies, a KM product, are cost effective considering the short time and skill available to produce them by a team of young professionals using original field data.
4) Sustainability: The IP program is sustainable since its systems are in place and has involved a variety of stakeholders at different levels, country, state and local. The IP program of UNICEF ICO has a unique design of cross-cultural teams working at grassroots to produce a knowledge product – however, there is little evidence regarding how the case studies were used. UNICEF ICO can continue the program as is. The evaluation could not explore if the program could be ’adopted’ by others since these decisions are dependent on many other factors. Currently, some parts of the IP are outsourced but fully outsourcing selection of interns, teams, matching with organizations, and reviewing case studies requires dedicated staff and expertise wherever it is housed. The evaluator can only opine that maintaining quality standards is important and that the outsourcing must weigh efficiency considerations with quality maintenance. Whether government can take it over is a moot point since the many pieces that make the IP program work are unique and difficult to replicate to the quality and flexibility needed in a bureaucratic environment.
5) Possible Impact: Overall, the IP program has contributed to interns’ awareness of development issues with field experience, ability to develop a knowledge product such as the case study using primary data collection and unexpectedly learned how to work in multi-cultural and multi-disciplinary teams. The case studies were disseminated but lacked an explicit strategy for use. Expecting that interns’ case studies can influence policy is unrealistic even when the KM products are vetted for quality. Overwhelmingly, the interns, host organizations and UNICEF have suggested that the benefits of the IP justify its continuation.
The IP program is relevant with knowledge management strategies and has by and large fulfilled the objectives of the IP program. Due to the effective management and structured design of the IP regarding linking a KM result with developing a cadre of young professionals, there has been a continuous demand for internship over the years even though the numbers accepted have decreased. Also the number of products – case studies and videos (in earlier years) has steadily increased with acceptance for publishing also going up. The IP program promoted multi-cultural and multi-disciplinary understanding and resonated with the international mandate of UNICEF.
Systematic, transparent and substantive communication has emerged as a key factor for the successful implementation of the IP. This includes the importance for regular communication a) between UNICEF state offices, government and host organization not just about logistics but of more substantive issues b) between UNICEF state offices, host organizations and government to select the most appropriate areas of study considering the two month period available c) between UNICEF ICO and the state offices regarding quality of research and guidelines for production of a quality product by interns and not just about contracting host organizations or managing logistics d) between interns and UNICEF enabling transparent reporting of findings and how they can be represented in the case study e) between UNICEF ICO, state offices, sector specialists and leadership for the more substantive issues of the IP such as how to use the product (also depends on what is selected for research and its quality) and finally f) communication with interns both in the short term before their arrival ( and connecting with other interns past and present, proposed state offices and host organizations); immediately after the internship to communicate the status of publication (or not); information regarding use of the case study; and in the long term after they move on to catalyze on their India experience. Interns’ keen desire to continue to contribute to the IP program needs to be actively and strategically exploited – having a Facebook page is only one step in the process.
UNICEF ICO, state offices, host organizations, government and interns’ collaboration and shared expectations affected the case study quality and use. If any of these stakeholders have different expectations, then it is impossible for the interns to produce a case study fit for publication. Managing expectations was important including the host and UNICEF state offices ability to provide the interns the ‘space’ for honest data collection and reporting (and that one can learn from what does not work well) and interns for representing data in the context in which it is collected.
A network of research organizations has been created over the years and it is important to nurture this relationship as well as to continually add new organizations to the pool. The IP lesson is that in short term relationships where the benefit is perceived in only one direction (for the interns and UNICEF), the engagement is likely to be limited and be compromised in quality. To liaise with the best research organizations available – academic or otherwise – requires a long term relationship perceived to be of mutual benefit.
Impact of the IP on Interns, host organizations and UNICEF and recognition of the benefits is substantiated by the overwhelming recommendation for the program to continue. The IP has enriched interns in many ways – being better prepared to enter the development field, understanding children and development needs in India, able to network and make useful contacts and being able to apply their research skills. As one intern mentioned ‘one of the great things about the program is the creative freedom within a framework of scientific rigor’. UNICEF recognizes the role in its own knowledge building and although the request for hosting interns is now voluntary, the demand has remained constant from the UNICEF state offices indicating they do value the IP. Host organizations have added to their own knowledge building through the IP program and believe that their relationship with UNICEF is stronger. Although government is consulted and information shared, their engagement is limited for various reasons.
Sustainability of the IP program whether by UNICEF or others will depend on many factors including how UNICEF positions the IP as a program of value for knowledge management. It will depend, besides recommendations for improving the program, upon how much UNICEF leaders, staff at country and state level believes in the interns’ ability to conduct research, host organization’s to mentor them, government to support the IP and the overall quality of the case studies that encourages not only dissemination but a clear strategy for use.
The eleven recommendations based on lessons learned are clustered around three broad categories: strategic, programming and operational issues based on the analysis of findings according to relevance, effectiveness, efficiency, sustainability and impact. The responsibilities for the implementation of recommendations have also been mentioned. The recommendations (R) are as follows:
Related to Strategic Issues: R1) All stakeholders to strategically disseminate case studies and decision makers, especially in UNICEF, to advocate for use: Dissemination is not use and strategies are needed to create pathways for the creative use of the publications by all stakeholders including interns themselves and the academic institutions they come from. A product by interns over a two month period has to have realistic expectations; it is unlikely to influence policy (as expected in the IP objective). R2) UNICEF to revise and reorder objectives of IP to realistically reflect what it can achieve and to manage multiple expectations: Some of the objectives of the IP and their order of importance need to be reviewed. R3) UNICEF ICO and state offices to brand and advocate the IP nationally and internationally: Champions in UNICEF at leadership levels at ICO and the state would energize and bring value to the IP in India. Internationally, the unique design of the IP – the selection process, multi-cultural teams of interns, the collaboration with research organizations at state level, the field work opportunity and a concrete KM product vetted for quality – needs be branded and disseminated
Related to Programming Issues: R4) UNICEF to support high quality research skills and provide timely technical backstopping: Research capability and inputs by all stakeholders in the IP is important – interns, UNICEF ICO, state offices and host organization – to produce a quality document. This may mean additional research skill building workshops or recruiting personnel with research skills. R5) UNICEF to extend the reach of IP recruitment in India and abroad: The selection of interns has been streamlined and outsourced to a consultant that ensures transparency of selection from a variety of applicants from India and abroad. However, catchment areas from India and internationally have remained the same over the years for various reasons. R6) All stakeholders to make creative use of the short timeline through early introductions, understanding the context and guidelines for quality documentation: The time period of two months is very short for interns to understand and adjust to the context in India so the process can start early and will mean tweaking of the logistics. R7) UNICEF to collaborate with key stakeholders to develop good quality TORs: The research topic must be an expressed need of relevant stakeholders especially government and well articulated so that government, host organizations, interns and UNICEF clearly understand what data is to be collected, why and how it is to be used.
Related to Operational Issues: R8) UNICEF ICO and state offices to be proactive in identifying and building relationships with quality research institutes to support the IP as well as other research needs: Selection and motivation of quality host institution is key to the success of the IP as they will be the mentors for interns and their first point of contact. R9) UNICEF ICO and states to develop/strengthen strategies for advocacy and use of case studies (and other knowledge products) with government: The government is informed and supports field collection, but they need to be actively involved in the TOR development and findings need to be disseminated to them in a succinct manner. Only then is use possible. R10) UNICEF ICO to strengthen IP ownership by UNICEF and other stakeholders at state level: The IP is currently viewed as a UNICEF ICO program driven by SPPME, although participation of state UNICEF offices is voluntary and the IP is decentralized in operation. R11) UNICEF to actively build alumni networks: UNICEF has not engaged actively in the over 400 alumni network created over many years missing out an opportunity to build knowledge and partnership networks and advocates
Full report in PDF
PDF files require Acrobat Reader.