Base de datos de evaluación

Evaluation report

2005 Global: Evaluation of UNICEF learning strategy to strengthen staff competencies for humanitarian response 2000-2004



Author: Le Groupe-conseil Baastel ltée

Executive summary

Background:
In 1998, UNICEF decided to enhance its preparedness for emergencies and was able to obtain funding from DFID to support this process. The first phase of the DFID-funded programme sought to apply international resolutions on children in crisis situations to UNICEF emergency response. Phase II had as an overall goal to improve the capacity of UNICEF to respond effectively, reliably and in a timely manner to the needs of children and women in unstable situations. Of the eight goals, Goal 4 addressed the learning needs of the organization by improving staff competencies for advocacy, programmatic and operational support in emergencies, reflecting the human rights based approach to programming and the normative framework of UNICEF.

Purpose/Objective:

The objective of the evaluation implemented by the Baastel team was to provide an assessment of UNICEF efforts to strengthen staff competencies for humanitarian response through learning during this phase.

Methodology:

A number of tools were designed to meet the information needs of the evaluation team. In the core survey, an internet-based data collection strategy was implemented and informants could fill in the questionnaires directly. A total of close to 1500 individuals were invited to participate. Qualitative information produced in the online and the emailed questionnaires confirm a very significant number of opinions expressed by key informants in topics such as follow-up training, the advantages of staff exchange, and the urgent need to review Performance Evaluation Reports (PERs) in order to factor in participation in emergency learning opportunities.

This evaluation has focused on three sets of assessment criteria. The first set of criteria relates to how the different learning strategies that UNICEF has used have contributed to the mainstreaming process. The second set of criteria assesses whether these learning strategies were in line with the best practices outlined by the United Nations System Staff College (UNSSC) to facilitate the development of a learning culture that leads to improved management capacity and organizational performance. The third set relates to the actual impact of the learning strategies used on UNICEF’s performance and its capacity to plan for and respond to humanitarian crises.

Findings and Conclusions

The learning strategies developed through the UNICEF-DFID Programme of Cooperation have focused as much on changing staff attitudes towards emergency as they have on developing specific skills. The primary learning strategy used by UNICEF in this project has been direct training and a significant portion of the project’s resources have been allocated to funding workshops on diverse emergency-related topics. The two main training programmes developed through the project were the Emergency Preparedness and Response Training (EPRT) and “A Principled Approach to Humanitarian Action” (PATH). These are the two sets of direct training that were offered to UNICEF on a fairly global basis.

UNICEF has offered direct training related to emergency in three different ways. The first has been to set up a Training of Trainers (ToT) or cascade system of training in which a group of UNICEF staff are trained in both the subject area and on facilitation skills. These trainees were then expected to go out and train others at the RO or CO level. This approach is relatively cost effective, can potentially generate a wider reach, and is better rooted in the local context. It does, however, require greater follow-up, may result in a dilution of information during the subsequent transfer, and add to the staff workload. UNICEF staff who become trainers are expected to carry out their regular tasks in addition to following through on their new training responsibilities.

The second was to bring in professional trainers or sector experts from headquarters, the region or from external organizations and to have them deliver training to participants at the regional and country levels. Regional training generally involved the RO asking each CO in the region to send one participant to take part in the training sessions – which were usually held once a year or sometimes once every other year. Those participants were then expected to share this knowledge with their colleagues upon their return, although in a more informal way than was the case for the ToT participants. The major constraint to this approach, however, remains its cost. UNICEF has been unable to maintain at least one of its sectoral emergency programmes to date (health and nutrition) in part because of this cost factor. For the external facilitator model to work then, UNICEF would need to significantly increase the percentage of funds it commits to training from its regular operating budget and to enhance its recourse to local institutions.The bulk of the DFID funding has been devoted to direct training in the form of workshops. These workshops have covered a wide range of topics related to emergencies.

In general, UNICEF appears to be developing a more coordinated response to emergency preparedness and response and an important part of this has been developing closer relationships with other agencies. Staff indicated that given the growing need for them to coordinate emergency activities, they would like to see a component on coordination skills included in the EPRT and other emergency-related training. UNICEF needs to consider developing a global policy that would support increased inter-agency training. Currently most initiatives are done on an ad-hoc basis, as it is not part of UNICEF strategy yet, so much is dependent upon the individual relationships that specific officers or representatives have developed in the countries or regions in which they work.

Computer-based learning has also been allocated significant programme funds. In many cases it has been perceived as a viable option to direct learning strategies. Nevertheless, informants have mentioned circumstances that (1). Impede the adoption of one or the other technological innovations; or (2). Prevent them or colleagues from taking full advantage of the many digitalized services that the programme has offered them. In most cases the major view was that there was no time to either learn new computer-based tools or to go through a web-or-CD-based course. From an organizational perspectives, the lack of monitoring tools (with the notable exception of the security CD-Rom) and the limitations of UNICEF’s IT system to evaluate the comparative advantages of direct versus computer-based learning has been a challenge.

All staff interviewed indicated that they have found on-the-job training to be effective. The participants of the three control focus groups the evaluation team conducted for staff who had not undergone the EPR training all demonstrated that they had been able to find diverse ways of learning what they needed to know about emergencies on the job. However, they showed considerably less confidence with regard to knowing exactly what their specific roles should be and in their ability to handle emergencies if they arose. Focus group participants in general indicated that experience in an emergency was the best learning strategy. Senior management also repeatedly stated that having prior emergency experience was an important factor in staff’s ability to handle emergencies effectively. For this reason, despite the many constraints that appear to exist, they were also quite supportive of increasing the different types of opportunities for on-the-job training related to emergency work for staff.

Although UNICEF staff are quite interested in taking part in staff exchanges or deployments to other offices and despite the fact that senior managers see the value in this type of experience, there are also constraints to the establishment of an institution-wide on-the-job training programme within UNICEF. These constraints include:

  • Representatives beingare reluctant to provide release time to the staff person concerned since there is no one to replace them during their absence.
  • It is an expensive form of staff training as it involves both travel expenses and the payment of a DSA while the staff member is living in the exchange or deployment country.
  • Some staff members are reluctanthesitate to participate as it will mean that either their colleagues have to pick up their work during their absence or else the work will not get done and they will have an even heavier workload upon their return as they will have to catch up.
  • In general, staff exchanges and deployments are a fairly strategic learning opportunity in that it buildsthey build staff capacity in key areas where it is anticipated there will be future needs, uses a shared responsibility approach, fosters the view that learning is a part of UNICEF’s culture and offersculture, and offer a viable alternative to direct training.

UNICEF identified a clear need to support its staff in the development of emergency preparedness and response plans and through the DFID project made it possible for COs to have access to external facilitation of this process. The participatory and very country-specific nature of the planning process is both appreciated by staff and an effective way to introduce the key issues and processes to be considered. However, the follow-up in each country has been inconsistent and dependent to some extent upon the interest and leadership of the Representatives. However, given the prevalence of this lack of follow-up, it is not possible to say that the problem does not lie with the process. The planning process is a dynamic one, but UNICEF clearly needs to build in a more effective follow-up process and accountability system to accompany the contingency planning process.

While UNICEF senior managers generally agree that it would be beneficial for them to establish mentoring and coaching programmes for some of their staff, they still need access to simple guidelines to assist them with this process. The one-on-one attention that more junior staff would receive from their managers in this type of relationship would certainly foster learning and skills and capacity development. It also may be that the provision of guidelines on mentoring and coaching with clear expectations, tips and lines of accountability would be sufficient to get a more systematic coaching and mentoring programme established within UNICEF. This process would support future capacity-building related to emergency mainstreaming if the guidelines also provided some insight and tips on key areas in which staff need to be coached related to working within the context of a humanitarian crisis.

In general, “lessons learned” activities – such as evaluations, field reviews and exercises – foster the view that learning is a part of UNICEF’s culture, represent a strategic learning opportunity in that they tap the actual experiences of individuals and country programmes within UNICEF, use a shared responsibility approach (when they are participatory), and offer a viable alternative to direct training. UNICEF would need to do more in terms of improving the quality of lessons learned activities, enhance its dissemination strategies and strengthen its ownership on this data.

UNICEF has made progress in the institutionalization of learning within the organization in that at least half of its COs now submit annual learning plans. OLDS has also recently elaborated a “Learning Road Map” which should be helpful for career-oriented and motivated staff who are interested in advancing within UNICEF to know what skills and competencies are required for positions at different levels of the organization. Nonetheless, there still needs to be greater pressure from the top levels of the organization to ensure that the remaining COs start to take the learning plan process seriously. The learning plan system also still puts too much onus on individual initiative. Given that staff feedback to the evaluation team indicates that they often feel guilty if they take “time off work” for training or other forms of learning, it is clear that the view that learning is an essential part of their work has not yet permeated the organization.

A fully institutionalized learning plan system could be highly strategic as it would help significantly in the allocation of resources for different types of learning throughout UNICEF. It also makes learning a shared responsibility as it is up to the COs to come up with a plan that meets their particular learning needs facilitating the ability of ROs and headquarters to respond to these needs. This process is also an important measure to help establish learning as a part of UNICEF’s culture.

The evaluation team and their UNICEF counterparts discovered significant challenges in the DFID-UNICEF Programme’s M & E system:

  • The most basic information from the programme’s workshops (i.e. the list of participants) was not given a standardized format where important variables, consensually designated, would have been identified, inserted by organizing stakeholders, controlled by a responsible party and placed in a central database, with the potential to be then shared throughout the organization.
  • The inhibited fluidity of information circulation presented some surprising situations: key informants would ask the evaluation team for data on workshops in their sectors. Causal factors were often attributed to decentralization practices (most key informants used this as a leitmotiv) and to a relative lack of accountability tradition. All of these factors could have been addressed though a consensual, light, form-sheet based monitoring system conveyed by a flexible communication protocol.
  • With the noted exception of the EPR and PATH workshops, the beneficiary information was not processed or analyzed. To the team’s knowledge, the only analysis produced by OLDS for its own purposes was targeted to the building of a network of suitable trainers in the core sectors. Not only does this limitation weaken the accountability process, both internally and externally, but it shrinks the foundation of all decision-making processes dramatically.

Recommendations:
General Recommendations on Learning Strategies

Direct Training


Direct training is still the most effective way to train most UNICEF staff and it is appropriate for UNICEF and DFID to continue to allocate the majority of their capacity-building funds to this learning strategy. To make direct training more effective, UNICEF needs to:

1. Allocate sufficient resources to key areas of training required so that the regional offices can organize the number and types of workshops actually needed in any given year to develop and maintain core competencies in humanitarian assistance at the CO level.
2. Establish a core group of regional staff and external trainers with relevant experience for the EPRT course and for PATH . For the staff trainers, this work should be a fundamental part of their job description as opposed to an add-on for which they would need to seek release time. National staff should also be considered as candidates to become regional staff trainers depending upon their expertise and experience, with preference being given to national staff who are fairly senior and who have a training background and strong emergency knowledge.
3. experience.Recognize the critical role of the Regional Emergency Officers in the success of the training process by including selection criteria related to their training experience and ability, and include accountability of REOs (and other regional officers) for capacity building related to humanitarian response.
4. In the long term, establish a small training support unit in each of the regional offices that would provide both training support and leadership in key areas as well as take on the monitoring and evaluation of regional learning strategies related to humanitarian response.
5. Invest in building the capacity of external regional trainers who can supplement the work and expertise of the staff trainers. The role of OLDS in this process would be to work with the ROs to help them identify, select and develop the core group of external regional trainers and to integrate the external and staff training teams.
6. Focus future training initiatives at the CO level to increase the numbers of staff who can access the training, with a particular emphasis on making different kinds of humanitarian response training accessible to national staff.
7. Call upon regional specialists in the ROs to serve as resource people only for selected trainings and do not rely on them to play the role of lead facilitator in multiple training sessions in any given year.
8. Provide additional resources to the HQ-based Programme Learning Group to help strengthen their capacity to coordinate UNICEF’s various programme and emergency learning initiatives and to systematize UNICEF’s approach to learning.

Training of Trainers (ToT)/Cascade Model of Training

9. Instead of developing a ToT system in which selected international staff are trained to serve as co-facilitators alongside the regional specialists, establish a core group of regional staff and external trainers as outlined in the Direct Training recommendations above. This would require UNICEF’s making a commitment to establish a core team of external trainers and developing a longer-term institutional or contractual relationship with them as well as strengthening their understanding of how UNICEF systems and programmes operate. It would also mean developing and working with a core group of UNICEF staff trainers who have access to senior management and part of whose role would be to become advocates of the training programme. Ideally, in any given training programme, there would be an internal UNICEF staff trainer and an external trainer with advanced training experience serving jointly as co-facilitators.

The rationale for this recommendation is that despite its great success in the ESARO region, the ToT model of training is not a particularly workable model for most of the other UNICEF regions. The evaluation process identified the following challenges with the ToT model:

There is considerable wastage with only between 50 to 70 % of the ToT trainees going on to provide training to their colleagues for a variety of reasons.
Staff trainers often have difficulty getting release time to lead training sessions outside of their CO and accommodations are not made so that this responsibility does not add to their workload. 
It takes a long time and a lot of coaching before staff trainers can function effectively as lead facilitators on their own. 
There are concerns about the dilution of the training materials by staff trainers who are not sector experts. 
Staff trainers often do not have the same degree of credibility and stature as a Regional or external specialist.
UNICEF does not have an effective monitoring and evaluation system in place to follow-up on the ToTs.
Staff trainers do not necessarily get career recognition for their additional training role.
A ToT system requires a major commitment on the part of the regional specialists to serve as the lead facilitator and to provide the necessary follow-up support required for the staff trainers. While this contributes greatly to staff capacity-building in the region, it also means that the regional specialist has considerable less time to address other priority areas and can lead to regional staff burnout.

On-the-job Training

On-the-job and other types of personnel field training such as staff exchanges, mentoring and coaching suffer from the very variable positions taken by country representatives. Therefore, there is a need for UNICEF to:

10. Address the diverse constraints that exist to these types of staff training by encouraging Representatives to both ask for help and to lend their staff to other countries in emergency situations, as has been done in the TACRO region. This encouragement could include acknowledgement and recognition in the PERs of Representatives who foster and promote increased staff exchanges.
11. Increase the resources and staff available to nurture and support greater implementation of on-the-job training strategies.
12. Have the DHR develop and disseminate tip sheets and guidelines on coaching and mentoring from the perspective of both the managers doing the coaching and mentoring and of the staff being coached.
13. Ask each sector/unit to develop and disseminate rosters of specialist coaches in emergency-related work as the Evaluation Office has done with their M & E “coaches” in crisis and unstable situations roster.

Lessons Learned

Recommendations concerning lessons learned activities from visits to the field and from a review of core documents include:

14. Develop an evaluation report format that is user-friendly and short to make them more accessible to field staff and establish standards/benchmarks to ensure quality control of evaluations to maximize their learning potential.
15. Have the HQ Evaluation Office and the ROs circulate these evaluation reports widely by email or in hard copy format in addition to the current practice of just posting them on the Internet.intranet.
16. Continue preparing the evaluation briefs called “Hot Off the Press: Lessons from Evaluation” andto sponsor “lessons learned exercises” as in Istanbul in October 2003 to review UNICEF’s emergency preparedness in Iraq.
17. Promote “communities of practice” amongst staff on topics related to emergencies so that they can exchange experiences, learn from one another, and share information amongst themselves on a regular, informal basis. This can be at the regional level or across regions on operational or sectoral issues such as financing procedures, water and sanitation, education, etc. and could be included as a part of the 5% of staff time that they are supposed to allocate to training and learning with the support of their Representatives.

Computer-based Training

18. In order to make computer-based training more effective, make the most important cross-cutting issues in emergencies such as PATH, HIV/AIDS, and Child Protection mandatory for all with a set protocol for going through the training and receiving recognition/certification, and strengthen the PER process so as to motivate staff more.
19. Increase collaboration between IT and OLDS to support the roll-out of e-learning courses.

Learning Plan Systems

UNICEF must achieve a greater level of institutionalization of its learning plan system. To do this, the organization needs to:

20. Ensure that there is a systematic review of each staff member’s learning goals each year as part of their performance review and supervisory process. Develop and implement HR policy and practice that would support and recognize these efforts.
21. Ensure that emergency learning needs are considered and integrated into CO learning plans everywhere.
22. Ask OLDS to enforce the annual submission of CO learning plans by all countries and give them the authority to do so.

Inter-agency Training

23. Develop a global policy that would support increased inter-agency training on EPRT, PATH, child protection, HIV/AIDS and M & E within emergency contexts.

Leadership

24. Foster increased leadership and vision in capacity building related to humanitarian response by UNICEF senior management at the field level by establishing clear institutional rewards and incentives through HR policies and practices and through senior management at the HQ and ROs levels, making it clear that this is an institutional priority and commitment.
25. priority.Make the institution and senior management, in particular, more accountable for building staff capacity in emergencies by establishing a system that holds.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Date:
2005

Region:
Global

Country:

Type:
Evaluation

Theme:
Emergency - Preparedness

Partners:
Le Groupe-conseil Baastel ltée 

PIDB:

Follow Up:

Language:
English

Sequence Number: 
2005/803

Búsqueda