We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2010 Nepal: Evaluation of UNICEF Nepal Mine Action Activities: Victim-Activated Explosion Injury Surveillance and Mine Risk Education



Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:

Ten years of civil conflict between Nepali government and Maoist forces resulted in explosive device (ED) contamination in all 75 districts of Nepal. Systematic clearance operations began following the signing of the Comprehensive Peace Agreement in 2006, but significant contamination remains. As in other conflict and post-conflict settings, civilians have suffered disproportionately from ED contamination. In recent years, non-state armed groups (NSAGs) seeking autonomy have become active in the southern Terai region, leading to further ED proliferation.

The United Nations Children’s Fund (UNICEF) in Nepal works with partner organizations to diminish the effects of ED injury among civilians. While UNICEF conducts a wide range of mine action (MA) activities, most of its resources support two tasks:
1) maintenance of the victim-activated explosion (VAE) injury surveillance system run by the national non-governmental organization Informal Sector Service Centre (INSEC); and
2) coordination and support of mine risk education (MRE) activities in partnership with over 20 national and international organizations.
In November 2009, UNICEF Nepal initiated an external evaluation in order to learn how to improve its MA activities and to fulfill obligations outlined in its annual work plan and donor proposals.

Purpose/Objective:

The goal of the evaluation, according to the Terms of Reference, is “to contribute to the review, planning and improvement of the prevention strategies which have been implemented thus far in Nepal for the purposes of minimizing the number of victim-activated explosions.” The objective is “to assess the relevance, effectiveness, efficiency, impact and sustainability of the support provided by UNICEF in addressing the mine action needs in Nepal in…MRE and surveillance.” The intended audience for the evaluation report includes UNICEF Nepal and all other stakeholders in UNICEF MA activities.

Methodology:

The evaluation was conducted by members of the International Emergency and Refugee Health Branch at the United States Centers for Disease Control and Prevention (CDC). It was carried out in consultation with MA experts at UNICEF Headquarters in New York City and the United Nations Development Programme (UNDP), with technical and logistical support from the Child Protection Section at UNICEF Nepal.

The evaluation team conducted a field visit to Nepal in November 2009. Prior to the field visit, the evaluation team reviewed relevant literature and consulted with MA specialists at CDC, UNDP and UNICEF Headquarters. During the field visit, the evaluation team conducted individual meetings with key informants working in organizations that conduct MA activities in Nepal. Follow-up interviews were conducted as needed by telephone and e-mail. Preliminary findings and recommendations were presented at a meeting held in Kathmandu in November 2009 and attended by MA stakeholders in Nepal; a follow-up presentation was held in May 2010. An evaluation report was completed in August 2010.

Surveillance activities were evaluated according to CDC’s Updated Guidelines for Evaluating Public Health Surveillance Systems, developed by CDC’s Guidelines Working Group and published in 2001. MRE methods were evaluated according to the International Mine Action Standards Mine Risk Education Best Practice Guidebook 8 on evaluation.

Findings and Conclusions:


Recommendations:

The evaluation team made the following recommendations (Section names in parentheses indicate where explanations of recommendations can be found throughout the report):

A. Recommendations for the VAE injury surveillance system

1) Improve data collection
· Rename the narrative field currently labeled “Other injuries” on the data collection form as “Description of injuries” to better distinguish between this field and the diagram with the same name. Change the name of this field from “Others” to “Description of injuries” in the database. (Section V.B.2.c)
· Ensure that INSEC district representatives (DRs) understand the difference between the two diagrams and can read and understand the English labels. (Section V.B.2.c)
· Ensure that DRs understand how to select checkboxes in the “Other injuries” diagram. “Head/neck”, for example, should be selected for all injuries to the eyes, ears, face, etc.; “arm” should be selected for all injuries to the fingers, wrists, hands, elbows, etc. (Section V.B.2.c)
· Encourage DRs to select checkboxes on the two diagrams whenever possible and to describe all injuries in the narrative “other injuries” field. In this manner, the accuracy of data entered in the two diagrams can be validated using the narrative field. Any descriptions in the narrative field should include locations of injuries. (Section V.B.2.c)
· Train data entry personnel who enter records into the database and regional- and central-level INSEC personnel who conduct quality checks to understand the concepts mentioned in the preceding three recommendations and to thoroughly check each record for errors. In the future these personnel should be able to recognize coding errors, recode them correctly in the database and communicate with DRs to prevent future errors. (Section V.B.2.c)
· Review past records in which injuries were described in the narrative “Other injuries” field but checkboxes in the “Loss of” and “Other injuries” diagram fields were not selected sufficiently. Wherever possible, select appropriate “Loss of” and “Other injuries” diagram columns in the database based on the descriptions available in the narrative “Other injuries” field. (Section V.B.2.c)
· Consider completion of a “quick guide” for all data collection, entry and quality assurance personnel to draw attention to data entry concerns and recent changes to the data collection form and the database. This guide could be included in the data collection form or printed as a laminated card or other separate document. (Section V.B.2.c)
· Remove the “Occupation at the time of accident” question from the “Victim Description” table and list it as a separate question with checkboxes for answer options, in the same format as “Activity at the time of accident”. (Section V.B.2.c)
· Create three data source categories on the data collection form and in the database:
o “Source name” with a narrative field;
o “Source address” with a narrative field (may be omitted from the database if not considered necessary); and
o “Relationship of source to victim” with the following answer choices: victim (self); witness family member; witness non-family member; non-witness family member; and non-witness non-family member. (Section V.B.2.c)
· Ensure that all fields in the data collection form and database are labeled and arranged consistently:
o Change the order of SN (record number assigned by DR) and CSN (casualty number) on the data collection form to match the order in the database and consider adding ISN (incident number) and VSN (casualty number per incident) to the form. Completion of ISN and VSN fields on the data collection form by data entry personnel could facilitate matching of data collection forms with database rows for necessary follow-up.
o Change “Prepared by” on the data collection form to “DR name” to match the database.
o Place the three questions related to incident circumstances (“Who activated the ED?”, “How did the ED explode?” and “Activity at time of accident”) together on the data collection form to match the database. (Section V.B.2.c)
· Ensure that all fields on the Nepali and English data collection forms are consistent. (Section V.B.2.c)
· Ensure that all DRs understand and code answers to the following questions in a consistent manner:
o “Was area marked as dangerous?”: this refers to official markings (such as a sign posted by the Nepalese Army (NA)) and not unofficial markings (such as a piece of cloth tied to a fence by a community member);
o “Literacy”: an injured individual is considered literate if he/she can read and write simple sentences in his/her mother tongue;
o “Family size”: family members are defined as those who have been using the same kitchen and sitting and eating together in the same home for the past six months;
o “Number of dependents”: dependents are family members who rely on the injured individual for food, education, and other resources);
Steps to ensure that these definitions are clear may include additional training of DRs, provision of a “quick guide” drawing attention to data quality concerns and/or additional clarification on the data collection form. (Section V.B.2.c)
· Consider modifying definitions of economic status and MRE to increase clarity:
o Current options listed under “Economic status” are “lowest (family members must work to feed themselves)”, “lower (family has food surplus that will last six months)”, “middle (family has food surplus that will last one year)” and “higher (family has food surplus that will last more than one year)”. The evaluation team suggests the following alternative options: “No food surplus”; “Food surplus for 1-6 months”; “Food surplus for 7-12 months”; and “Food surplus for > 12 months”.
o It may be desirable to differentiate between formal MRE (such as attendance at a session conducted by UNICEF or a partner organization) and informal MRE (such as looking at a leaflet passed along by a family member); some partners may wish to limit this data element to formal MRE to study the coverage of MRE programming). (Section V.B.2.c)
· Adjust types of ED listed on the form and in the database to reflect past and current injury trends. Remove answer choices that have never been chosen: “bucket”, “pressure cooker”, “flag”, “banner”, “roadblock”, “cartridge”, “claymore type” and “anti-vehicle mine” for “Type of ED” and “health worker”, “social worker”, “political worker” for “Occupation at the time of accident”. Add some answer choices (such as “detonator” for “Type of ED” and “housewife”, “police” and “child” for “Occupation at the time of accident”) that have been entered frequently in “Other” fields in the past. Please note that “removing roadblock/flag/banner” should be removed from the answer choices following “Activity at time of accident” if these ED types are removed from the answer choices for “Type of ED” question. Such activities should be coded as one of the “Handling ED” choices. (Section V.B.2.c)
· If partners are trying to understand the incidence of known risk behaviors (intentionally striking/throwing ED, handling ED, etc.), the data collection form may be changed to ensure that “Did victim know activity was dangerous?” is asked only of individuals who report high-risk behaviors at the time of the incident. If partners prefer to continue asking this question of all respondents, they may wish to disaggregate responses by activity at time of accident during analysis. (Section V.B.2.c)
· Revisit the exclusion of members of NSAGs from the case definition, especially given the recent decision to include members of government security forces. This may encourage all injured individuals to seek victim assistance (VA) services, regardless of political affiliation, and may identify previously unknown areas of ED contamination. (Section V.B.2.c)
· Ensure that a mechanism has been created to incorporate NA and other government security forces injury data into the data collection process, in accordance with the recent decision to include members of government security forces in the case definition. (Section V.B.2.c)
· Ensure that DRs continue to apply the case definition of VAE rigorously, particularly as the inclusion criteria expand to include combatants. (Section V.B.2.c)
· Revisit the question about the last grade passed with DRs; this question resulted in the lowest response rate (82.9%) in 2009. (Section V.B.2.c)
· Ensure that a mechanism exists to differentiate between unintentionally skipped and intentionally unanswered questions in the data collection form and database. This mechanism can be as simple as using “x” to denote one and “?” the other, but should be appropriate for a Nepali data collection form. DRs and data entry personnel should be trained in the new mechanism. (Section V.B.2.c)
· Clarify all case definitions and ensure that they are applied consistently in all districts. (Section V.B.2.c)
· Investigate the possibility of conducting a “capture-recapture” comparison of INSEC data with those of other sources (including Ban Landmines Campaign Nepal (NCBL), the Nepal Police and HimRights) to reveal more about the sensitivity of the system. (Section V.B.2.c)

2) Ensure efficient maintenance of the database
· Clarify the naming and order of Microsoft Excel database worksheets. (Section V.B.2.c)
· Consider storage of the database on a secure online data storage system such as Dropbox Additional information about Dropbox is available at http://www.getdropbox.com/. or a designated File Transfer Protocol (FTP) site. Storing the database on such a system would allow the INSEC focal point for MA to control access to the file and save time currently spent saving the database in multiple locations. (Section V.B.2.a)
· Address discrepancies between the data collection form and the database. (Section V.B.2.a)
· Reconsider the idea of regional data entry; allowing more people access to the database may jeopardize data quality. If the primary reason for considering this idea is to allow regional personnel to analyze their own data, it may be possible for the central office to send them the data separately. On a related note, regional personnel should be cautious when interpreting data at the regional level, at which sample sizes are so small that conclusions may not be statistically significant. (Section V.B.2.c)

3) Encourage use of data collected
· Consider expanding the surveillance system objectives to link data collection with all pillars of MA. (Section V.B.1)
· Clarify the naming of reports posted on the INSEConline website; this will increase the usability of the reports and encourage more users to return to the site. (Section V.B.1)
· Consider making reports available in some other format, as individuals without access to the Internet do not have access to reports online. If UNICEF and INSEC do not wish to publish reports elsewhere, they may consider using other media to inform the public about the availability of the online reports. (Section V.B.1)
· Collaborate with NA personnel who prioritize areas for demining and clearance to ensure that injury data are being considered during the selection of priority areas. (Section V.B.1)
· Encourage the development of a standardized response mechanism by which VAE injury surveillance system data can be used to clear newly found ED in a timely manner. At this time, INSEC data reach the NA via UNICEF and the United Nations Mine Action Team; this new mechanism would accelerate the flow of data by sending updates directly from INSEC to the NA. (Section V.B.1)
· Encourage all MA stakeholders to use INSEC surveillance data, explaining the benefits of using better data to drive advocacy and MRE activities. (Section V.B.2.d)

4) Encourage and protect participation in the surveillance system
· Ensure that adequate measures are in place to protect the security and confidentiality of informants and DRs. (Section V.B.2.d)
· Address fears of repercussions for reporting among communities through public campaigns or other appropriate means. (Section V.B.2.e)
· Improve the accessibility of reports for surveillance system informants and DRs. Steps may include translating reports into Nepali, renaming the reports posted online for better clarity, or providing access to printed reports. Special attention may be required to ensure that results reach communities who speak languages other than Nepali or have low literacy levels. (Section V.B.2.d)

5) Ensure sustainability for necessary surveillance system functions in the future
· Start to develop an “exit strategy” for the surveillance system. Given that so few VAE injuries are occurring every year, it may become increasingly difficult to justify the expense of running a separate surveillance system. UNICEF and INSEC may wish to establish a threshold at which they will discontinue system activities; this may be the number of injuries or affected districts. Given that intentional explosions pose an ongoing threat in Terai districts, it may be possible to combine the VAE surveillance system with monitoring of intentional explosions. (Sections III.D.3 and V.A.2.h)
· Reconsider the current level of resource expenditure on VAE injury surveillance, especially if VAE injuries continue to decrease in incidence. It may be possible to collaborate with the Ministry of Health, World Health Organization or other partners to establish an all-cause injury surveillance system, which could result in better use of limited resources. Support already exists among some Nepali medical personnel for the establishment of such a system. (Section V.B.1)
· Ensure that DRs are paid enough to cover their travel expenses; this may involve paying a travel per diem that varies with duration of travel rather than the current fixed sum. (Section V.B.2.i)
· Develop a full understanding of what expertise and institutional memory would remain if INSEC discontinued participation in the surveillance system, and assess how plausible it would be for another organization to take ownership of the surveillance system. (Section V.B.2.i)
· Collaborate with INSEC to maintain more thorough records on surveillance system expenditures. (Section V.B.2.i)
· Ensure that contingency plans are in place to address problems of delayed access due to transportation and communication problems; the INSEC central office may be able to find additional methods by consulting with DRs and regional staff. (Section V.B.2.h)
· Establish contingency plans to address computer and telephone problems. (Section V.B.2.i)

B. Recommendations for UNICEF MRE activities

1) Consider integration of MRE activities into broader health and education initiatives.
· Continue to evaluate whether systematic MRE coverage is adequate and consider scaling up activities to reach more students. This may be feasible through incorporation of MRE messages into all school curricula and/or integration with a broader injury prevention program (see the following two recommendations below). (Section VI.C.4)
· Consider incorporation of MRE and broader injury prevention messages into the curricula of all government education programs. Provision of MRE to all recipients of government education activities could reduce the need for other more expensive forms of MRE dissemination. (Section VI.C.4)
· Consider incorporating MRE into a broader injury prevention program. Integration of MRE into a health education program could serve to alleviate the burden of injury on the population; better justify program expenditures to donors; and remove some of the political tension associated with ED use and contamination. UNICEF could continue to work with many of the same partner organizations and seek additional collaboration with the World Health Organization, Ministry of Health, Ministry of Education, and other relevant partners. If partners did not consider ED injury a risk in all areas, ED injury prevention education could be included as an optional module for use only in schools in at-risk areas. (Section VI.C.4)
· Promote MRE among marginalized populations by integrating MRE messages into non-formal education activities such as those for adult literacy and for children not attending school. Inclusion of MRE in non-traditional programs such as the UNICEF Education Section’s Life Skills Project could ensure better coverage of vulnerable populations. (Section VI.C.1)

2) Adopt new measures to improve financial sustainability and management, particularly in preparation for handover of MRE activities to the National Mine Action Authority (NMAA)
· Examine costs of MRE activities and consider ways to address financial sustainability of MRE programs. (Section VI.C.5)
· Maintain more thorough records on the production and dissemination of all MRE materials (leaflets, user guides, flipcharts, posters, and MRE emergency kits). These should include the following data elements for each type of material: cost per unit; number made each year; number distributed each year; recipients; cost of distribution; and number remaining. (Section VI.C.3)
· Consider coordinating with MRE partners to ensure that MRE focal points are receiving equal pay and/or reimbursement for equal work. (Section VI.C.2)
· Consider conducting a formal cost-effectiveness study of MRE materials and/or activities. (Section VI.C.3)
· Find faster delivery methods for MRE materials, sharing transportation with partner organizations when possible. (Section VI.C.2)

3) Plan for a UNICEF “exit strategy” and the eventual handover of MRE activities to the NMAA
· Build capacity for all MRE activities, including development, production and distribution of MRE materials; training; coordination of media campaigns; and monitoring. (Section III.C.1)

4) Improve the strength and focus of MRE messaging
· Include more photographs and life stories of children, adults and families who have been affected by VAE injury. Current MRE materials contain drawings of risk behaviors and explosions, but real photographs and life stories may encourage session participants to think more deeply about the consequences of injury. This inclusion was recommended by the 2007 KAP survey and several stakeholders during the evaluation field visit; NCBL has already included pictures of injured individuals in some outreach materials (see Appendix B.1.g). (Section VI.B.1 and Appendix B.3)
· Work with the government security forces, if possible, to establish some kind of assurance that individuals will not be punished for reporting the presence of ED, and educate the public about this assurance. (Section VI.C.4)
· Continue to monitor whether economic necessity is a risk factor for ED injury. If it becomes more of a problem, provide viable alternatives to high-risk behaviors. If this does not seem possible given limited resources and the sensitivity of this topic, train MRE focal points to encourage affected communities to suggest alternatives to high-risk behaviors. Qualitative study could be used to clarify reasons for and solutions to high-risk behaviors. (Section VI.C.4)
· Pay close attention to the changing ED situation in the country; the shift from civil conflict between government and Maoist forces to violent acts committed by emerging NSAGs has implications for ED contamination and, as a result, MRE messaging. (Section VI.C.5)
· Ensure that all MRE symbols and messages selected or designed for use in the future are easily understood by the target audience. (Section VI.C.2)

5) Reevaluate and adjust assessment activities
· Conduct an impact survey in schools receiving systematic MRE using a tool similar to the “Risk Education Individual Test” (see Appendix B.2.e). (Section VI.C.4)
· Reconsider the utility of certain MRE monitoring activities such as district-level mapping. If an activity does not provide benefit commensurate with the human and other resources used, decrease or stop the activity. Time may be better spent on other necessary MRE activities. (Section VI.C.3)

6) Ensure that targeting is carried out appropriately
· Ensure that all emergency, regular and systematic MRE activities are targeted according to a specific and rigorous set of criteria, including VAE injury surveillance system data; recent VAE incidents; situations of obvious risk; and community needs. UNICEF may wish to undertake a global review to assess priorities and gaps in service provision. (Section VI.C.4)
· Provide police constables (as well as representatives of the NA, Armed Police Force and Maoists) with MRE materials and focal point training sessions, enabling them to conduct their own MRE activities. (Section VI.C.1)
· Reach out to the Communist Party of Nepal/Maoist and other Maoist groups at the central, regional, district and local levels. Framing the issue of ED injury as a child health issue – rather than a political issue - may encourage their participation. (Section VI.C.1)
· Encourage individuals in ED-affected communities to design some interventions that are executable at the local level. This may include suggesting local alternatives to high-risk behaviors as mentioned under “Impact” above, or promoting community-based methods of spreading MRE messages. (Section VI.C.5)
· Consider disseminating information through saving and credit and/or forest users’ groups, as per the recommendations of the 2007 KAP survey. (Appendix B.3)



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search