2002 CBD: External Evaluation of the Pilot Project of Community Based Mine Risk Reduction
Author: Hughes, B.; Tonh, M.; Sambath, C.
In early 2001, CMAC, Handicap International Belgium (HI-B), UNICEF and other members of the Mine Awareness Working Group (MAWG) designed a strategy for a more sustainable and community-oriented approach to mine awareness in Cambodia. The overall goal was to facilitate a reduction in the number of mine and UXO casualties by enabling people to live safely in contaminated environments through a multi-disciplinary approach to mine action, enhanced with community liaison. These actors developed a final project framework, which evolved into the CMAC Community Based Mine/UXO Risk Reduction (CBMRR) project.
The evaluation of this pilot phase was conducted to look at the future development of the CBMRR project. The evaluation focused specifically on its design, set-up and the first months of implementation, its collaboration with mine action, and its links to disability assistance and community development initiatives.
The evaluation has the following objectives:
- To assess the progress of the pilot project towards reaching the original goal and objectives of the project, and the effectiveness of the project activities undertaken to-date, measured by the extent to which the indicators have been met, or are in the process of being met.
- To assess the coherence and appropriateness of the project design, strategies and procedures developed during the pilot.
- To assess the effectiveness of the relationships among project staff, project participants, and other stakeholders and partners.
- To assess the longer-term sustainability of the project in terms of the capacity of CMAC to continue to manage and implement the project efficiently, and in terms of the perceived suitability and appropriateness of the project in the Cambodian context.
Five of the six districts of the pilot phase of the project were selected for the evaluation: Pailin and Sala Krao in Pailin Municipality, and Kamreang, Sampov Loun and Phnom Proek districts in Battambang Province. As all CBMRR project areas are considered to be high risk, the selection of the districts, communes and villages for evaluation purposes was primarily a logistical one, given the limited time available for fieldwork and accessibility to these areas due to seasonal factors.
The evaluation took place in October and November 2002 and was conducted through:
- Existing documentation was reviewed and preliminary consultations were conducted with CBMRR project management and the project technical assistant.
- Stakeholder interviews were conducted using question guides developed by the evaluation team, and informal interviews. Interviews were carried out using focus group discussions and individual interviews. Those interviewed included: villagers, MUC (village level), MUC (commune level), MUC (district level), and CMAC personnel and representatives from other agencies.
- A workshop was held to get a clearer understanding of how the project was progressing in terms of the participants' level of understanding of the goal, objectives and strategies of CBMRR. It was also important to take the opportunity to assess whether future dialogue between stakeholders was likely and to assess how others viewed the project in relation to their own role. It also provided another avenue to clarify initial findings and to better define necessary, future steps.
Findings and Conclusions:
Objective 1 - To assess the progress of the pilot project towards reaching the original goal and objectives of the project and the effectiveness of the project activities undertaken to-date, measured by the extent to which the indicators have been met, or are in the process of being met.
The original goals and the project objectives are suitable to the Cambodian context and in line with other agencies working from a "bottom-up" perspective. Whether it is achievable or not will be very much dependent on future support from CMAC and sponsors such as UNICEF, HI-B and NPA.
The progress made to-date, however, has been encouraging in particular areas. It is too early at this stage to gauge whether there have been any real changes to the reduction of risks taken in mine fields and UXO tampering. There were some positive signs that the messages were reaching the target audience. There appears to be an increased awareness of mines/UXO risk, however, the fundamental factor still remains one of livelihood. Even if people are aware of the risk of carrying out a particular activity, they do not see that there are other alternatives. It remains a larger economic issue and one where the broader CBMRR concept is appropriate. Given the length of time the programme has been operative, the statistics from the CMVIS relating to accidents are not conclusive as an indicator of the success of the pilot.
The project has contributed towards building community capacity to participate in the prioritisation and planning of mine action in a fairly limited way. There is insufficient understanding of the planning and prioritising of needs at the village level, which results more in the creation of a "wish list" than realistic planning. The DFP have been very active in assisting the community to make realistic requests.
The progress of the project with respect to facilitating access of mine/UXO-affected communities to appropriate mine action activities, victim assistance programmes and community development responses is limited. This area has not progressed as far as other aspects of the project due primarily to the lack of understanding at all levels of the CBMRR project. The PCs demonstrated a good awareness of this area and were active in developing relationships with relevant agencies. The public information campaign at village level is still very much word-of-mouth as the MUC representatives do not have the time or resources for more organised activities. The commune and district levels are more organised but complain of the lack of financial support for their activities.
Due to the nature of the project, the assessment of its progress in terms of reaching the original goal and objectives may be somewhat subjective as it is difficult, if not impossible, to quantify. What was evident from the initial interviews was that only those who received direct training fully understood the CBMRR project. Despite obvious support for the project in general, a number of those interviewed did not have a clear understanding of the CBMRR project goal and objectives. At the community level, there was little distinction made between CBMRR and CMAC as a whole.
Objective 2 - To assess the coherence and appropriateness of project design, strategies and procedures developed during the pilot.
The organisational structure of the project is suitable for the community-based approach of the project, however, there was insufficient time allowed for the implementation of the project with technical support. It is not possible to fully implement a community-orientated programme within the time allowed for the pilot, particularly when the TA (or equivalent) is not having a continued association with the project in the next phase.
The organisational structure allows for easy replication in other high-priority areas and for future sustainability, if implemented fully and carefully. The project coverage has created considerable debate both within CMAC and with other agencies. The areas for immediate extension should be selected not only on the statistical areas of highest priority but also in consultation with other agencies working within the region. This will ensure that the limited resources are focused in the areas of greatest need. It is an area where CMAC should learn from previous misunderstandings with other agencies. Dialogue with other agencies also ensures that complementary programmes are instituted in selected areas.
The staff recruitment and evaluation process has been effective. The most resounding strength of the design of the project is the selection and usage of District Focal Points (DFP). It is evident that they are the pivotal points of the programme. The evaluation found that the decision to recruit the DFP from the community was a well-considered one. Although the individuals selected have very different styles of service, they are well-accepted by their respective communities.
The training curriculum designed for the CBMRR staff appears comprehensive and suitable, and there was good indication that those who attended the initial training had a good understanding of the basic concepts of the project. There were, however, problems encountered by the MUC representatives at village level. The three-day training given by the TMO in each district was too short for them to get a sufficient understanding of the project. It was sufficient for the commune and district levels, as they were at least familiar with "community-oriented" programmes. The concept was not as familiar at village level.
The CBMRR monitoring framework is not fully comprehensive and is an area that needs immediate attention. There currently are no TMOs recruited. The reporting practices and ability of key participants to report is of concern, and has been identified by CBMRR staff as an area in need of improvement. The CBMRR Standard Operations Procedures (SOP) is a comprehensive document that can be practically implemented in the field. To date, there is no Khmer translation that is problematic for easy access to the information for non-English readers. Monitoring of the CBMRR project staff must continue to ensure that they fully understand their roles and responsibilities. Other CMAC staff whose role interrelates with the CBMRR project should also receive training regarding CBMRR to enhance the effectiveness of its strategies. A particular strength in the design of the project is the suitability of the key personnel and an effective appraisal system to monitor performance. This was evident by the non-renewal of contracts, when it was apparent that the personnel did not demonstrate the required attributes for such a position. At this stage of the implementation of the project, it is essential that this diligence continue.
Objective 3 - To assess the effectiveness of the relationships among project staff, project participants and other stakeholders and partners.
The role and responsibilities of the MUC at district, commune and village level are clear and appropriate, as the design of the project also reflects the national policy on decentralisation. This brings it effectively into line with other established institutional frameworks. With the newly created role for the Commune Council, there are the inherent uncertainties associated with the roles and responsibilities of the various administrative levels. This is not a CBMRR design fault but one that is indicative of the anomalies of the framework established by law.
The level of community involvement in the CBMRR project is still limited, primarily due to the livelihood commitments of the MUC representatives. In general, the community did not distinguish between the CBMRR project and other CMAC activities and, accordingly, their expectations of the CBMRR project were not distinguishable.
The level of integration between the CBMRR project and other CMAC units, such as EOD and CMMT at the provincial DU level, is very encouraging. However, this level of integration is not evident at HQ level. There is a need for CBMRR to be promoted at HQ level to ensure inclusion in the entire CMAC operational work plans. These relationships could be improved significantly by a greater understanding of the CBMRR goal and objectives within CMAC. CBMRR has made reasonable links with other mine-action organisations or agencies. The success of these links has again been restricted by the lack of understanding of the project by these organisations and agencies, in particular how CBMRR fits within the CMAC structure and how it is distinguished from other CMAC operations.
Objective 4 - To assess the longer-term sustainability of the project in terms of the capacity of CMAC to continue to manage and implement the project efficiently and effectively, and in terms of the perceived suitability and appropriateness of the project in the Cambodian context.
To assess the long-term sustainability of the project in terms of the capacity of CMAC to continue to manage and implement the project efficiently is somewhat speculative. Community-oriented projects are, by nature, both slow to implement and difficult to quantify in terms of outputs. The flow on effects of a project such as CBMRR is unlimited. What is apparent from interviews is that there is a widely-held view that the linkage to outputs (clearance), whether real or perceived, becomes fundamental to the sustainability of the project. If CMAC does not allocate sufficient resources to attend to the prioritised needs of the community, the community will lose confidence in the project. The training of CBMRR staff should emphasise that promises, which are unrealistic, should not be made, creating expectations that are not achievable in the mind of the community. They will become quickly disheartened if there are lengthy delays in the response to requests or if the reasons for the delays are not communicated convincingly. CMAC must make the necessary commitment to the project to ensure that there are sufficient resources to meet the reasonable expectations of the community.
Full report in PDF
PDF files require Acrobat Reader.