We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2010 Sri Lanka: Evaluation of the Mine Risk Education in Sri Lanka



Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:

The Democratic Socialist Republic of Sri Lanka is an island nation in South Asia with a population of approximately 21.5 million. The majority of Sri Lankans are Sinhalese, followed by significant populations of Tamils and Sri Lankan Moors and smaller numbers of Burghers and Malays. The smallest administrative unit in Sri Lanka is a Grama Niladhari (GN) or Grama Sevaka (GS) Division. The 14,008 GN (GS) Divisions in the country are grouped into 324 Divisional Secretary (DS) Divisions. DS Divisions are grouped into 25 Districts, which are further clustered into nine provinces. Map 2 in Appendix A shows the nine provinces and 25 district headquarters
1983 is widely acknowledged as the start of the civil conflict between the Government of Sri Lanka and the Liberation Tigers of Tamil Eelam (LTTE), which sought to establish a separate Tamil state in the northern and eastern parts of the country. Levels of violence fluctuated throughout the three decades of conflict, with a marked increase in fighting in 2001 and 2002 and establishment of a formal ceasefire in February 2002. Hostilities began anew in late 2005 and early 2006 and escalated in 2007 and 2008. Government forces made significant military gains in 2008 and 2009, seizing all territories under LTTE control. The government declared an end to the conflict on 16 May 2009, and the LTTE released a statement acknowledging defeat on the following day.

Both the Sri Lanka Army (SLA) and the LTTE used landmines and other explosive devices extensively throughout the conflict. The SLA used antipersonnel mines, anti-tank mines, command-detonated Claymore directional fragmentation mines, and tripwire improvised explosive devices (IEDs), while the LTTE used antipersonnel mines, anti-tank mines, IEDs, Claymore-type directional fragmentation mines, anti-tank mines, and booby traps. The Indian Peace Keeping Force, present in Sri Lanka between 1987 and 1990, also used landmines.

In 1999, the government estimated that 20,000 to 25,000 landmines had been planted during the conflict; this estimate had risen to 1.5 million landmines by 2004. In 2010, the SLA estimated that a total of 1.6 million landmines were laid during the conflict.

According to Landmine Monitor, 17 agencies and organizations have been involved in demining and clearance operations since 1999. Nine of these are active at this time: the SLA’s Humanitarian Demining Unit (HDU); two national non-governmental organizations (NGOs) (Delvon Assistance for Social Harmony Sri Lanka and the Milinda Moragoda Institute for People’s Empowerment); and six international NGOs (Danish Demining Group, the HALO Trust, Horizon, Mines Advisory Group (MAG), Sarvatra, and the Swiss Foundation for Mine Action). Demining groups operating during the conflict faced numerous setbacks, including staff abductions and resignations, restrictions on movement and resources, suspension of demining operations, and continuing use of landmines.

In 2002 the United Nations (UN) reported that both parties to the conflict had been using landmines “mainly in accordance with conventional military tactics” and that “neither [g]overnment forces nor [the] LTTE seem to use landmines to target civilian populations or to deny civilian population access to particular areas.” However, significant contamination with antipersonnel and anti-tank landmines and other explosive remnants of war (ERW) (including unexploded and abandoned explosive ordnance, or UXO and AXO) remained in ten districts following the conflict. According to Landmine Monitor, the Northern and Eastern provinces—particularly in Jaffna and the Vanni—were heavily affected.
It is difficult to determine exact levels of current contamination. According to the SLA, 366,870 of the total 1.6 million landmines laid in Sri Lanka were cleared by September 2010, leaving a remaining 1.23 million landmines. As of 31 August 2010, an estimated 552 square kilometers of land remained contaminated by landmines and ERW.
In addition to marked minefields, contaminated physical locations include living areas such as houses and livelihood areas such as gardens and agricultural and pastoral land. Reports also confirm contamination of urban areas, water sources, and roads. During and after the conflict, contamination with landmines and ERW has posed a major obstacle to resettlement and economic reconstruction in the northern and eastern parts of the country 

Purpose/Objective:

In early 2010, MA personnel at the UNICEF Sri Lanka central office in Colombo asked members of the International Emergency and Refugee Health Branch (IERHB) at the United States Centers for Disease Control and Prevention (CDC) to conduct an evaluation of UNICEF’s MRE activities in Sri Lanka. According to the Terms of Reference (TOR) agreed upon by UNICEF and CDC, the goal of the evaluation was “to contribute to the review, planning and improvement of the prevention strategies which have been implemented thus far in Sri Lanka for the purpose of minimizing the number of victim-activated explosions” and the objective was “to assess the relevance and effectiveness of the support provided by UNICEF in addressing the MA needs in Sri Lanka in …MRE.” The TOR also specified the following major tasks for the evaluation:
• “Understand the context (political, humanitarian, financial, accessibility, partnership, etc.), the history, and the current implementation of the project.
• Analyze the activities and strategies developed by UNICEF in MRE.
• Analyze the previous findings of evaluations, surveys and reports conducted on these areas.
• Develop and implement methods to assess the impact of MRE on reducing injuries and/or risk-taking behavior.
• Propose a summary of strengths and areas for development of the MRE project and propose how best UNICEF can help build sustainable capacity in this area.
• Present the final results with recommendations to UNICEF.”
The evaluation team conducted the tasks stated above with one exception: it did not develop or implement methods to assess the impact of MRE on reducing injuries and/or risk-taking behavior because it was not possible to do so at this time. The evaluation team did, however, examine the possibility of conducting a future impact evaluation; these findings and recommendations are discussed

Methodology:

Members of IERHB evaluated UNICEF Sri Lanka’s MRE activities according to the UN Evaluation Group’s Norms and Standards for Evaluation and the International Mine Action Standards (IMAS) Mine Risk Education Best Practice Guidebook 8 on evaluation, developed in 2005 by UNICEF and the Geneva International Centre for Humanitarian Demining. Additional interview questions and qualitative methods were developed based on the evaluation team’s previous MRE evaluation experience and conversations with key stakeholders in Sri Lanka. The evaluation team used three primary methods for the evaluation: document review; qualitative data collection via interviews, group discussions and meetings; and observation of community-based MRE activities. Evaluation questions focused primarily on the current time, although participants often mentioned past activities and events.
The evaluation team reviewed the following documents:
• the UN Evaluation Group’s Norms and Standards for Evaluation;
• the IMAS MRE Best Practice Guidebooks 1-12, published in 2005, especially Guidebook 1 (Introduction to Mine Risk Education), Guidebook 2 (Data Collection and Needs Assessment), Guidebook 3 (Planning), Guidebook 4 (Public Information Dissemination), Guidebook 5 (Education and Training), Guidebook 6 (Community Mine Action Liaison), Guidebook 7 (Monitoring), Guidebook 8 (Evaluation), and Guidebook 10 (Coordination);
• Landmine Monitor reports published between 1999 and 2009;
• national strategy and MRE standard documents;
• data collection forms and incident and injury data from the SLMAD;
• a database of MA images compiled by UNICEF for the NMAC;
• previous evaluation reports by Ms. Anusha Fonseka, Prof. Kalinga Tudor Silva and Prof. Asoka Jayasena (2004) and Mr. Marco Ramazzotti (2007) ; and
• reports, presentations and other documents provided by UNICEF and partners.
Field work comprised 75 key informant interviews, group discussions and meetings in Colombo and eight additional districts, Field work was carried out during five weeks in September and October 2010 with technical and logistical support from the Child Protection Sections at UNICEF in Batticaloa, Colombo, Jaffna, Trincomalee, and Vavuniya. Of the 75 interviews, group discussions and meetings, 54 were conducted outside Colombo. 

Findings and Conclusions:

The establishment of the NMAC, development and implementation of national MA standards, and finalization of a national MA strategy demonstrate the government’s dedication to advancing MA activities.
• UNICEF central and zonal personnel are knowledgeable, experienced, and dedicated to their work.
• UNICEF has fostered a very positive relationship with the NMAC.
• Good relationships between UNICEF and some SLA field commanders facilitate the SLA permission process. UNICEF and NGO personnel maintain strong relationships with the HDU, government agents (district administrative heads, or GAs), and other local partners, who often intervene when MRE field personnel are stopped at checkpoints.
• UNICEF has fostered positive relationships with government education personnel at the central, provincial, district, and local levels. Government education personnel who met with the evaluation team understood and discussed the importance of MRE for school-aged children and seemed supportive, enthusiastic, and well-organized.
• NGO MRE trainers are very committed to their work. NGO MRE community session trainers appeared confident, knowledgeable, and engaging during observed MRE sessions. Several NGOs make an effort to work through existing government and village-level structures rather than create new ones.
• HDU personnel are dedicated to providing high-quality MRE activities to communities. Many HDU personnel cultivate positive relationships in the communities they serve; they sometimes play with child and youth clubs and have collaborated with local NGO chapters to organize sports clubs for local youth.
• NGO and HDU personnel support each other’s contributions to MRE and coordinate activities together in several districts. Positive relationships between NGO and HDU personnel improve MRE programming and build peace and trust within communities.
• NGO field workers believe that the methods they learn during trainings are appropriate for the communities they target, and report feeling knowledgeable and confident to perform their field duties following UNICEF trainings.
• The joint Mines Advisory Group (MAG)-UNICEF capacity building project objectively reviews the activities of NGOs and targets project areas in need of strengthening.
• MRE and demining teams exchange information that is beneficial to each other’s programming.
• Strong links between MRE and victim assistance stakeholders connect survivors and their families with needed resources and services.
• NGOs invite landmine and ERW survivors to participate in MRE activities.
• MRE partners utilize integration of MRE with child protection and other development concerns to strengthen skills, connect people with needed resources and trainings, and conserve UNICEF and NGO resources.
• Selection of communities and schools for MRE is conducted based on injury and contamination data and in collaboration with Grama Sevakas (GSs) and DMAO and UNICEF personnel. Special efforts are made to address the needs of high-risk populations.
• Integration of MRE into the national school curriculum will reach all children who pass through government schools, and these children will be able to spread MRE messages to others.
• When asked about messages used in the field, NGO MRE trainers consistently answered that they use the curriculum of core messages provided by UNICEF. Use of the same core messages by all MRE operators guarantees the consistency of MRE messages delivered.
• UNICEF and partner NGOs have updated messages according to changing contamination, injury trends, and newly developing risk behaviors and misunderstandings.
• Partners have used a wide variety of MRE settings, methods and techniques, often developing different approaches to reach specific communities and populations.
• NGOs value the review and revision of materials with other MRE stakeholders, and some have conducted formal field-testing. NGOs have made several modifications to improve MRE materials following field tests.
• UNICEF and NGO partners have improved the cost-effectiveness of community-based MRE.
• Some challenges, such as removal of landmine/ERW signs, markings, and stakes from demining sites, have been resolved via CL.
• Creation and use of community maps have encouraged village participation in MRE.
• Evaluation participants believe that risk reduction initiatives such as the Safer Village project have reduced engagement in high-risk behaviors in some areas.
• Overall, community views of MRE heard by the evaluation team were overwhelmingly positive.


KEY CHALLENGES
• Some donors fear that the external—rather than internal—impetus and significant external funding for the establishment of the NMAC questions its ability to function in the future without significant external support.
• The implementation of MRE activities has been obstructed by the need to obtain Ministry of Defence clearance, a lack of support from some government personnel (see Section VI.A.2.a), and—until recently—required approval by the Presidential Task Force for Resettlement, Development and Security (see Section VI.A.4.c.ii).
• The placement of UNICEF MA focal points (in UNICEF zonal offices or DMAOs) has not been resolved, leading to confusion and frustration among several stakeholders.
• Some UNICEF MA focal points seem overwhelmed with duties and unable to complete all of their tasks.
• Combining MRE with child protection issues may have endangered government support of MRE activities.
• MRE partners do not always have sufficient knowledge of MRE implementing partners and activities in specific areas.
• NGO MRE teams are often stopped by local SLA personnel, leading to delays and cancellations of MRE programs.
• The review process for proposals and contracts within and between UNICEF and NGOs is inefficient.
• Internal UNICEF bureaucracy at the central level is slowing the pace at which contracts and proposals are approved.
• A combination of access, approval, and contract delays has effectively shut down civilian capacity for programming in some areas, notably in Jaffna District.
• Feedback on MRE training sessions does not appear to be solicited in a consistent or effective manner.
• HDU teams do not include women or representatives of several ethnic and religious communities.
• Most HDU personnel do not speak Tamil and cannot communicate adequately with participants in community-based MRE.
• At this time no clear uniform policy is enforced during HDU MRE activities.
• Some Village Child Rights Monitoring Committees, child clubs, and other village-based partners are not currently functioning.
• NGO and HDU partners experience considerable transportation shortages in the field.
• Some local partners are having difficulty conducting programs with limited resources.
• Many evaluation participants believe that school-oriented MRE has been inadequate.
• A UNICEF risk reduction consultant found several obstacles to school-based MRE, including poor prioritization of MRE relative to other subjects and “lukewarm enthusiasm” among teachers.
• Little or no quality assurance (QA) is conducted on the Sri Lanka Mined Area Database (SLMAD). SLMAD contents are not adequately analyzed and regular reports based on these contents are not created or used sufficiently during the planning of MA activities.
• Many MRE partners conduct knowledge, attitudes, and practices (KAP) surveys but do not always analyze survey results or use them for program planning. Because NGOs do not use the same sampling methods, data collected in different communities and at different times cannot be compared to determine trends.
• Low levels of injury give people a false sense of security.
• People living in landmine- and ERW-affected areas do not have adequate access to alternative livelihood projects.
• No formal process or standard technical procedures are in place to clear wells. As a result, people try to clear wells themselves or hire others to clear them.
• Targeting of foreign returnees, Sri Lankan tourists, religious pilgrims and workers from unaffected areas is inadequate.
• Evaluation participants believe that certain more accessible communities have been targeted multiple times while some less accessible communities have not been targeted at all.
• Some demining agency personnel are bridging MRE coverage gaps with their own MRE activities. Demining agencies do not conduct activities using messages and methods agreed upon by UNICEF and partners.
• Misinformation and misunderstandings related to landmines and ERW prevail among target audiences.
• MA partners are spreading inconsistent messages about community markings.
• The presence of uniformed military personnel at MRE sessions conducted by NGOs may intimidate civilian participants or dissuade them from participating during MRE sessions.
• Some materials have contained mistranslations between Tamil and Sinhala.
• Materials have not been field-tested in a consistent manner.
• Demining agencies never received CL support personnel pledged by UNICEF.
• Some communities receive very little information about demining progress or future demining plans.
• MRE monitoring criteria currently used by DMAO personnel are superficial and do not measure the quality of MRE activities. SLMAD MRE monitoring information is not being adequately used in planning of MRE.
• Some UNICEF and NGO personnel have been asked to leave communities upon arrival to conduct MRE activities.
• Challenges unrelated to landmines and ERW trouble communities and prevent uptake of MRE messages.
• Little action has been taken to address trap gun proliferation and injury.

Recommendations:

National coordination activities
• Utilize UNICEF’s strong relationships with senior government and military personnel to secure permission to allow MRE activities to continue as planned. Engage the assistance of government-affiliated personnel (such as representatives of the HDU and DMAOs), non-HDU deminers, and advocates from disabled servicemen’s organizations and/or other victim assistance and advocacy groups who are supportive of and knowledgeable about community-based MRE activities
• Encourage the hiring of NMAC and DMAO personnel who have the skills, experiences, and competencies recommended by IMAS. The NMAC and DMAOs may wish to consider hiring NGO workers with significant field experience for such positions.
District and zonal coordination activities
• Finalize the decision regarding placement of UNICEF MA focal points in UNICEF zonal offices or DMAOs. The evaluation team recommends placement in the DMAOs, which would improve government MRE capacity and enhance coordination with other MA activities. If stakeholders decide that the UNICEF MA focal points should split their time between the two locations, adherence to a regular weekly schedule with equal time in each office will improve time management and communication with partners in both offices.
• Assess the workloads of UNICEF MA focal points and reassign non-MRE child protection responsibilities to other UNICEF staff in locations in which MRE is urgently needed at this time—especially in areas with increased resettlement and insufficient MRE coverage.
Training and capacity building of MRE partners
• Establish refresher trainings for all NGO and HDU MRE trainers on a regularly scheduled basis.
• Collect training feedback in a consistent and confidential manner, preferably on paper, and use both positive and negative feedback to guide future trainings. Discuss feedback immediately following trainings, while memories are fresh and trainers are not occupied with other activities.
• Compare indicators measured in the ongoing MAG-UNICEF capacity building project with the indicators recommended for training needs assessments by IMAS and edit ongoing and future project indicators as necessary.
Coordination of community-based MRE
• Create a clear record—via a chart, map, or diagram—of MRE implementing partners and activities in specific areas (“who is doing what where”) to find areas of low coverage and avoid duplication of MRE activities for specific populations.
• Strengthen community-based volunteer networks to increase MRE coverage. Community-based MRE creates a powerful “leave behind legacy” with local networks of community-based volunteers trained in advocacy and communication skills. Community-based volunteers can provide valuable assistance—not only for MRE but for other health and social interventions. Consider the “barefoot doctor”—or “barefoot health educator—model to increase MRE coverage in rural areas.
• Discuss physical access problems with senior SLA personnel. Acknowledge that this is a problem of communication rather than intent, and avoid blaming lower-level SLA personnel who are conducting their jobs as they have been instructed. Educate lower-level SLA personnel about MRE activities to increase their familiarity and comfort with MRE teams and materials.
• Assess the extent to which MRE field officers are at risk of occupational injury and reduce these risks wherever possible.
• Urge senior HDU personnel to hire women, representatives of more ethnic and religious communities, and fluent—and preferably native—Tamil speakers. If they are unable to hire additional personnel they may be able to second experienced NGO personnel for MRE activities.
Encourage the HDU leadership to finalize a clear dress code for HDU personnel conducting MRE in communities, ideally with civilian clothes or with the blue Ministry of Economic Development (MoED) uniform as a second choice. HDU—not UNICEF—personnel should monitor adherence to the uniform policy.
• Continue to coordinate joint NGO and HDU MRE activities and discuss their benefits during high-level advocacy activities with relevant government personnel. Language and cultural barriers and the history of the conflict will not allow HDU personnel to work as well in communities, but they have much to offer when partnering with NGOs.
• Coordinate MRE activities conducted by HDU personnel for Sinhala-speaking people traveling to landmine- and ERW-affected areas from unaffected parts of the country.

Coordination of school-based MRE
• Study the recommendations made by the risk reduction consultant for education authorities, school principals, and UNICEF MA personnel—as well as the additions and edits of other UNICEF personnel—and implement recommendations as appropriate to improve coordination of school-based MRE.
• Provide adequate instruction and materials to ensure that all district, zonal, and local education personnel have adequate knowledge and materials to incorporate MRE into their routine teaching activities.
Data collection and use for planning and implementation of MRE
• Develop standardized QA procedures to improve management of the SLMAD.
• Develop procedures to guarantee regular analysis, reporting, and use of a minimum set of key SLMAD variables by DMAOs and other stakeholders. Explore the possibility of automated data analysis. EpiInfo, for example, can be programmed to allow people with little epidemiologic or database training to generate basic reports with a simple series of keystrokes.
• Ensure that all fields are consistent on the SLMAD data collection forms and in the database.
• Require NGOs to maintain well-organized records of KAP results. Collect results in a national repository and use both aggregate and location-specific data to guide programming. Include training in KAP survey administration and submission of KAP data to the DMAOs in the NGO accreditation criteria
• Collaborate with representatives of all NGOs to create standardized KAP sampling methods and ensure that all NGO personnel use them.

Integration of MRE with other programming
• Investigate the possibility that combining MRE and child protection programming has hindered implementation of MRE activities, particularly in relation to government approvals. If this is found to be a significant problem, reconsider the decision to combine MRE and child protection activities.
Injury risk and MRE targeting, coverage, and messaging
• Increase MRE coverage in Jaffna with media campaigns and targeting of specific groups, including school children, high-risk workers, and tourists and religious pilgrims coming from unaffected parts of Sri Lanka.
• Collaborate with demining agencies to find gaps in MRE coverage and work with DMAOs and other MRE partners to bridge these gaps. Discourage demining agencies from conducting MRE activities unless they have been accredited by the NMAC and agree to disseminate messages consistent with those used by UNICEF and partner NGOs.
• Target returning populations via government and NGO service providers and GAs, Divisional Secretaries, and GSs.
• Target working populations via transportation and construction companies, the Ministry of Transport, and employment agencies.
• Partner with the MoED, Sri Lanka Tourism Development Authority, travel guide writers, travel bureaus and companies, other tourism partners, and temple personnel to target tourists and religious pilgrims.
• Target child surrendees/separatees through UN and other programs.
• Collaborate with the DMAOs to assign responsibility for particular communities to each MRE implementing partner, either during meetings or by other methods of communication (telephone or e-mail). Establish a strong external monitoring system that is inclusive of all MRE partners. Monitor districts where different partners are conducting MRE activities to ensure that communities are being targeted appropriately.
• Determine the frequency of certain high-risk behaviors (such as removing stakes and using ERW as household items) and use risk reduction measures to address situations in which people continue these behaviors despite understanding the risk of injury.
• Urge the DMAOs and demining partners to develop standard procedures to clear wells and inform communities about these procedures. In the absence of such procedures, conduct risk reduction activities to decrease the risk of injury.
• Work with all relevant stakeholders to develop one unified national policy for community marking of landmines and ERW and ensure that all partners are committed to this policy. This policy may decree marking by no civilians, marking by some civilians (such as NGO personnel), or marking by all civilians. If some or all civilians are expected to conduct community marking, they should be trained to mark objects and areas safely. • Address high-risk practices among military personnel with their superiors and explain why these behaviors—though possibly safe for military personnel—may be setting an unsafe example for civilians. Request that military authorities create policies to dissuade their personnel from engaging in such behaviors, particularly in the presence of civilians.
MRE methods, techniques, and materials
• Encourage military members who are present at community-based MRE activities to wear non-military clothing and take a more active—rather than observatory—role in trainings. Increased engagement of HDU personnel may also dissuade other military personnel from feeling that it is necessary to observe MRE sessions. These actions may prevent civilian participants from feeling anxious, frightened, or otherwise uncomfortable during MRE sessions.
• Collaborate with MRE partners to determine which materials contain mistranslations. Create a professionally translated set of core messages in both Tamil and Sinhala and ensure that materials created in the future are translated and back-translated by separate professional translators. Consider the use of more wordless materials, such as a well-designed leaflet with core messages in the form of illustrations
• Create a centrally-produced, high-quality, colorful and attractive leaflet that contains images of all core MRE messages without words .
• Create a new set of laminated training flashcards that includes all core MRE messages; risk and safe behaviors for which photographs are not available can be illustrated with drawings. Consider binding the flashcards into a flipchart to ensure proper order of messages, keep images flat (to improve visibility), and increase grip and usability.
Community liaison
• Hire and train CL personnel promised to demining agencies and coordinate their integration with demining teams.
• Strengthen the risk reduction capacity of MA partners by increasing funding, training, and other resources. Continue and increase support of alternative livelihood activities for people living in landmine- and ERW-affected areas.
• Improve communication between DMAOs and GSs, who can share information about demining priorities and schedules with community members.
Monitoring and evaluation of MRE activities
• Collaborate with MRE stakeholders to create a new MRE monitoring checklist that is IMAS-based, simple to use, and useful for program planning. According to IMAS, monitoring should track and monitor program activities and how these capabilities are being applied as well as changes in the landmine/ERW threat and operating environment (including changes in priorities, target groups, behavior, victim surveillance, victim assistance, and humanitarian demining). Use IMAS Guidebook 7 for guidance, exclude information that will not be used, and provide results to all relevant stakeholders.
• Use geographic information system (GIS) technology to overlay population or resettlement data with MRE activities conducted. This will allow partners to study and improve MRE targeting and coverage. SLMAD personnel at the DMAOs are able to create maps by combining database contents with GIS technology.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search