Evaluation database

Evaluation report

2010 Global: Review of the Global Education Cluster Co-leadership Arrangement



Author: Stuart Reid and Darian Stibbe (The Partnering Initiative); Catherine Lowery (Independent Evaluation Consultant)

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:

The creation of the Global Education Cluster co-lead arrangement was a bold attempt to bring something new to the cluster approach to humanitarian response. It was – and remains – a unique organisation of agency resources predicated on the expectation that a UN/NGO partnership might add value to the work of a cluster. As one of the last in a series of clusters to be created – and one which was controversial at the time – the Global Education Cluster offered an appropriate and timely vehicle for this pioneering experiment.

The purpose of this review is to help inform the forthcoming Global Education Cluster evaluation, and to suggest improvements in the co-leadership arrangement’s management and operations in their own right.
The approach taken by The Partnering Initiative (TPI), as the commissioned consultants for the review, emphasises an analysis of the co-lead arrangement as an actual or potential cross-sector partnership which can be further developed and refined in order to benefit the Global Education Cluster.

The review team from TPI employed a multi-strategy approach to generating and analysing data on the co-lead arrangement. The main source of data was a series of interviews with 32 key participants in the Global Education Cluster. This was supported by access to a substantial range of documentation relating to the Cluster and by surveying of views from 34 individuals working in regional and country level offices whose responsibilities included some work for the Cluster. In addition, a final phase follow-up questionnaire was sent to 8 key Cluster personnel to draw additional data on the issues of the Cluster management structure as a whole and on the mainstreaming of cluster responsibilities within the agencies.

As a specialist organisation in the field of cross-sector partnership, TPI has also drawn on its accumulated expertise, published documents and comparable research to provide insights into the partnership under review.

The Terms of Reference (ToR) for the review specified a series of questions that would need to be addressed. For convenience, these questions are presented in Table 1 at the end of this Executive Summary, with reference to the sections of the review that address each particular question. A brief summary of the findings and recommendations is presented below.

Key successes
“The added value is tremendous. With all the difficulties we still see tremendous potential in this colead arrangement. It brings new positive dimensions to the work, great degree of accountability and
most of all the co-lead provides credibility to the cluster.” (On-line survey respondent, Question 8)

Although the review was explicitly not an impact assessment, it identified the following factors that indicate that the co-lead arrangement is contributing well to the achievement of the Education Cluster’s objectives:
 Co-leadership gives the Global Education Cluster a distinct character and sends a strong signal on partnership to both the UN and NGO communities.
 The structure of the Global Education Cluster has been set up in accordance to the MoU.
 The Education Cluster Working Group (ECWG) has established a wide and inclusive network of agencies and the Global Education Cluster has a breadth of agenda that reflects this inclusive membership.
 The ECWG has established active Task Teams and Thematic Groups to meet Cluster objectives.
 The ECWG and the Education Cluster Support Unit (ECU) has been well supported by strongly committed individuals in both co-lead agencies.
 Both agencies have achieved significant progress towards the mainstreaming of Cluster support within the organisations.
 Guidance and standard-setting materials have been provided to the Cluster at country level, and training for cluster co-ordinators has been rolled out extensively during 2010.
 The comparative advantages that each agency brings are recognised (if not yet fully realised) at global and country level.
 Co-leadership is beginning to offer an inclusive and high quality response to humanitarian crisis.

Key challenges
The Global Education Cluster required both the setting up of a completely new structure and the adoption of a level of partnership working which was relatively unfamiliar to both agencies. As such, it is inevitable that the operationalization of the new structure would face challenges and much of the detail would need to be developed as it was rolled out.

The review identified the findings below that indicate that revision and further development of the co-lead arrangement need to take place for it to achieve its full potential.

Set-up
 The decision to establish a co-lead arrangement for the Cluster was more the result of innovative leadership than of a specific needs-based rationale.
 The practical implementation of the new concepts of co-leadership specifically, and partnership more generally, between a UN Agency and an NGO, were not sufficiently clarified before implementation. The nature of shared leadership within the partnership is still not well understood and the procedures for managing the co-lead arrangement – including on decision-making and accountability – have not been adequately defined, leading to challenges in implementation.
 Although now significantly more symmetric, institutional commitment to the principle of co-leadership was initially quite unbalanced, with UNICEF more sceptical over working in partnership with Save the Children than vice-versa.Final Report Draft Version 2.01
 There are continued concerns over the reliance on a single source of funding for the Cluster and how it is administered through UNICEF.

Overall management structure
 Although the overall architecture adopted by the Cluster would appear to be fit-for-purpose, its effective operationalization is being held back by a lack of clarity over the roles, responsibilities and accountabilities of each of the elements (ECU, ECWG plus sub-groups, Steering Group, mainstreamed agency roles).
 Although now improving, the Steering Group has not played a sufficiently strong role in its oversight of the ECU, leading to insufficient guidance and action to overcome problems holding back more effective operations.
 There is a need for stronger buy-in and commitment of agencies in the ECWG to extend a sense of collective responsibility for the success or otherwise of the Global Cluster. Balanced with this is the need for stronger oversight to ensure focussed and effective delivery of projects in accordance with the Cluster objectives.
 The ECWG is not sufficiently well linked into the country level to ensure that the global activities respond to the needs on the ground.
 There is insufficient coordination and communication between the mainstream agency roles and difficulties occur due to different structures of roles within the two agencies.
 A lack of agreed clear process and formal responsibilities to which the agencies could be held accountable has put significantly more responsibility on the individuals involved – and their relationships across agencies – to make it work.
 There remains significant confusion at the country level over how the Global Cluster operates and how the co-leadership arrangement should play out at the country level.

Performance of the ECU
 Although the situation has been improving, UNICEF is seen as not delivering sufficiently on its
commitments and responsibilities within the ECU2.
 The ECU has not been operating effectively as a team across the two organisations. Other elements such as the Education Cluster Working Group and the dedicated Task Teams have been working much more successfully.
 The structuring of the roles, responsibilities and lines of reporting within the ECU exacerbates difficulties in working together as a team.
 Communication between the partners within the ECU has not been sufficiently effective.
 Although a plan was drafted, there is no agreed management plan in place to guide the operation of the ECU.
 The physical separation of the ECU UNICEF and Save the Children staff inhibits effective team work between the organisations.
                                                           
2 Note: at the time of writing of this report, the situation within the ECU was changing with the appointment of a new Cluster Coordinator. This finding predates that appointment.REVIEW OF

External aspects
 Support to country-level staff is seen as uneven. Country level staff would like to see stronger advocacy and more focus on country, rather than global, concerns.
 The Cluster has not yet achieved an identity as a single entity. A balance between individuals representing their organisation’s interests and representing the Cluster’s interests has not yet been achieved.

Relationship
 UNICEF’s unilateral decisions on the appointment of the first two Co-ordinators were a serious breach of the partnership protocol and spirit, and continue to have repercussions for the partnership relationship.
 The comparative advantages of the two organisations are being utilised, if not yet to their full potential, but their different operating cultures can be a constraint to effective co-leadership, for example in the creation of a joint deployment roster.
 Equity between the partners – as felt within the partnership, and as perceived from the outside – has not yet been achieved.

Monitoring
 There is no system currently in place to monitor the effectiveness of the overall management of the Global Cluster or of the health of the partnership and effectiveness of the work relationship between the lead agencies.

Purpose/ Objectives:

The creation of the Global Education Cluster was approved by the Inter-Agency Standing Committee (IASC) in 2006 as an additional sector in the IASC’s overall cluster approach. In contrast to all other cluster sectors, the Global Education Cluster was established with a co-leadership arrangement whereby leadership of the cluster is shared by UNICEF and Save the Children. Given this pairing of two organisations of very different types, the Memorandum of Understanding (MoU) between the co-lead agencies stipulates that a review should take place
within two years of the cluster’s creation. The current exercise, led by The Partnering Initiative, constitutes this review and seeks to offer an independent, impartial analysis of the Global Education Cluster co-leadership arrangement. The Terms of Reference for this exercise set out the review’s purpose, objectives and preferred mode of procedure. The ToR states that

“The purpose of this exercise is.....not merely to help inform the forthcoming Global Education Cluster evaluation, but rather also to suggest improvements in the co-leadership arrangement’s management and operations in ther own right.” (UNICEF/Save the Children 2010:1)

The TPI review team notes that the current review takes place within the context of a planned review of the Global Education Cluster as a whole and needs to provide insights on the co-lead arrangement that will inform that wider exercise. Second, that the review should combine an analysis of the structure and operation of the colead arrangement with an assessment of evaluability indicators that might feasibly be used to measure the performance of the co-lead arrangement. Finally, that the review should be “formative and forward-looking” (UNICEF/Save the Children 2010: 3) and focus on identifying ways of improving the performance of the co-lead
arrangement.

In line with the origin and overall purpose of the co-lead review described in 1.1 above, The Partnering Initiative has organized the review process in order to achieve the main objectives set out in the Terms of Reference i.e. to determine:
 A sense of whether the elements in place to manage the co-leadership arrangement are adequate, both with regard to its architecture (Set-up) and its partners’ mode of working together (Operations and Relationship); and
 The indicators an evaluation of the Global Education Cluster co-leadership arrangement might use to measure the co-leadership arrangement’s performance (Evaluability).

However, it has done so in such a way as to emphasise the central role of partnership values in the relationship between the co-lead agencies and to encourage those agencies to address the question of how they develop their co-lead arrangement as a true partnership.

Methodology:

Research methods used in the review process
The broad methodological approach to gathering data for the management review and evaluability component followed what was set out in the Inception Report. This responded to the requirements of the Terms of Reference, which stated that the review should utilise a “mixed-method approach” including a review of documentation relating to the Cluster; a wider review of other relevant literature and documentation relating to partnership practice; semi-structured one-to-one interviews and focus groups with key stakeholders; and direct observation of formal and informal meetings.

In response, the Inception Report set out a methodological approach which consisted of four main stages: i) literature review; ii) one-to-one interviews; iii) remote/on-line survey; and iv) a review workshop. The primary aim of this approach was to generate sufficient data, primarily of a qualitative nature, to “complete an analysis of the partnership as it stands” and “to provide recommendations for how it could be improved” (TPI 2010: 9).

As mentioned previously, it was decided not to conduct the review workshop. Hence the contents of the report are based on the first three stages of that four-stage process: drawing data from the analysis of documentary evidence, one-to-one interviews and an on-line survey of staff in country offices. In addition, as the other methods of data capture did not draw out sufficient information on certain issues, a written questionnaire was sent out to a limited number of people in the final phase of the review. Each of these methods is described in greater detail below.

The review team would strongly recommend that the co-lead agencies consider, as originally intended, the added value of a further review workshop element as a next step to this process, in which evidence and recommendations can be processed jointly with a view to deciding actions to take the co-lead arrangement forward.

Analysis of documentary evidence
Internal documentation
The TPI review team was provided with a very substantial quantity of documentation both from the review management team and from senior individuals in each of the partner agencies. Although this was accompanied by guidance on what were “must-see” documents, it also presented some difficulties as there was duplication of documents from different sources, occasionally complicated by differing titles being used by each organisation.  Additional material was provided by some individual respondents to the one-to-one interviews, where documentation existed which were referred to in interviews or, in general, thought to be of relevance to the colead review. In total, the team was provided with 80 documents, an amount that presented a rich resource but one that was difficult to do full justice to within the timeframe of the review process.

External documentation
In compiling this draft report, the TPI review team has drawn on the substantial resource of material produced by TPI itself in the course of its work as a pioneer in the research, standard-setting, capacity building and evaluation of cross-sector partnerships. This resource includes tool books, case studies and previous evaluation exercises.  The team has also accessed insights from the published literature on partnering including both academic and practitioner material. Works explicitly cited in this report are included in Section 7 (References); documents accessed in the review process but not cited are listed in Section 8 (Bibliography).

One-to-one interviews
Telephone interviews during pre-Inception phase
Telephone interviews were conducted with a small number of personnel closely associated with the Global Education Cluster as a preliminary to the production of the Inception Report. These interviews were relatively informal and structured by a short questionnaire consisting of seven questions as a stimulus to wider discussion. A copy of the questions used in this survey is included in Appendix 1. Seven individuals were interviewed during this phase, four from UNICEF and three from Save the Children, including the Co-ordinator and Deputy Co-ordinator of the Education Cluster Unit. Data from these interviews were used to shape the Inception Report and to identify other individuals who might be interviewed during the main phase of the review.

Questionnaire survey used in face-to-face and telephone interviews
After producing the Inception Report, the TPI team worked closely with the review management team to identify a wider group of individuals engaged in work relating to the Global Education Cluster and, specifically, to the operation of the co-lead arrangement between Save the Children and UNICEF. This list was further expanded using suggestions from senior members of the partner agencies and from the Co-ordinator and Deputy Coordinator of the Education Cluster Unit. A few additional names were added during the early stages of the interviewing process as respondents identified colleagues whom they thought would prove of interest to the TPI review team.

A list of 39 individuals was compiled, which would act as a sampling frame for the main interview process. The list comprised individuals not just from UNICEF and Save the Children but also from UNESCO, UNHCR, OCHA, INEE, IIEP and Plan International. The aim was to interview as wide a cross-section as possible so that responses would be gathered from representatives of the various constituent organisations within the Global Education Cluster – the Steering Group, the Working Group, the Education Cluster Unit and the themed Task Teams (e.g. Capacity Development; Knowledge Management). Because the interview process was the main data-gathering tool for both the management review and the evaluability component, the list of interviewees included a number of people specifically identified by the consultant developing the evaluability indicators. These included several interviewees working (or recently working) in regional or country offices and the two Monitoring and Evaluation experts from UNICEF and Save the Children who were managing the co-lead review on behalf of their respective agencies.

The interviewing was carried out in two broad, if overlapping, phases. In the first, two of the TPI team visited Geneva to carry out face-to-face interviews with all the members of the Education Cluster Unit, those directly associated with the Unit and members of the Education Cluster Working Group based in Geneva. Over two visits to Geneva, 12 individuals were interviewed, representing all but one of those originally identified.

In the second phase, telephone interviews were conducted with Education Cluster participants from a range of organisations and locations. In this phase, a further 20 individuals were interviewed. A copy of the full questionnaire used in both the face-to-face and telephone interviews is included in Appendix 2. Three names on the original list were removed after further consideration; one declined to be interviewed; and three didn’t respond to our requests for interviews during the time period designated for the interviewing. Of these four nonrespondents, two had already been interviewed during the pre-Inception phase of the project so the TPI team had some data from them. In total, 32 individuals were interviewed in this phase, including the two members of the review management team. Taken together with the pre-Inception interviews, the TPI team had interview data from 34 individuals associated with the work of the Global Education Cluster, representing 82% of the original sampling frame population.

In addition, a written questionnaire was sent out to 8 individuals as a follow-up to get further information and opinions relating specifically to the overall management architecture and to the mainstreaming of Cluster roles and activities within the two agencies.

Observation of meetings
Although this method was considered – and discussed with the review management team – the opportunities available were limited and the practical arrangements not easy to achieve. In practice, one meeting was observed. Two members of the TPI review team sat in on a bi-weekly meeting of the Education Cluster Unit in Geneva. This was done informally, and with the full agreement of the participants. Some brief comments based on this observation are included in section 4.3.3.

On-line survey of country and regional offices
The Inception Report proposed a small-scale survey carried out at the country level “to gain insights into local management arrangements and, in particular, interactions with and opinions of the ECU in Geneva.” (TPI 2010:15) The sample of respondents selected for the on-line interviewing consisted of Cluster co-ordinators and other core staff in country and regional offices where the Global Education Cluster is represented. There was some discussion between the TPI team and the review management team over whether to target a selected sample of offices or to offer all personnel a chance to participate in the survey. Both for reasons of inclusivity and practicality it was agreed to make the survey generally available. This was felt to be the policy that would be most likely to ensure both a reasonable response rate (at a time when typically low rates may be even lower due to holidays) and to offer the chance of securing a spread of different local arrangements (e.g. single-lead; co-lead etc).

A list of 94 individuals was identified, covering 39 countries where the Global Education Cluster is active. In advance of the survey going live on-line, an email was sent from the review management team to Cluster coordinators alerting them to the survey and asking for their co-operation. Immediately prior to going live, the TPI team sent an email alert to a wider group of potential respondents introducing the survey and asking for responses within a two-week (10 working days) period. Mid-way through that period a reminder email was sent to encourage those who had not yet responded. A copy of the on-line survey is included in Appendix 3.

The response rate was slightly better than expectations, given the remote contact and the timing at the height of the holiday season. 39 individuals completed the on-line questionnaire, representing 41.5% of those contacted. Of those responding, 19 (48.7%) were from UNICEF, 19 (48.7%) from Save the Children and 1 (2.6%) from Plan International. Of the 39 countries contacted, responses were received from 21, with between 1 and 3 respondents from each office.

Over half of those respondents (53.8%) were acting as the local cluster co-ordinator and other respondents included those managing or supporting cluster co-ordinators; those acting as cluster co-leads; and those heading up their respective agencies in a particular country or region. Only four (10.3%) of the respondents spent 100% of their time on tasks related to the Cluster and two-thirds (n= 26) spent 30% or less of their time on Cluster work.  The majority of respondents (74.4%, n=29) described their local Cluster leadership as co-leadership between UNICEF and Save the Children; a small, and evenly divided, minority were operating in a single lead arrangement under either UNICEF (7.7%, n=3) or Save the Children (7.7%, n=3). Four respondents (10.3%) were working in colead arrangements either with different agencies involved or with the addition of a government partner to the normal UNICEF and Save the Children arrangement.

Data from the survey responses has been used to provide additional evidence in sections 4.1, 4.2 and 4.3 on the inter-relations between the Cluster at global level (and the ECU specifically) and those carrying the work of the Cluster at regional and country level.

Final phase follow-up written questionnaires
As previously mentioned, the information from the data sources focussed predominantly on the operations of the ECU. From the various comments on early drafts of this report, it became clear that there was a need to bring in more strongly an analysis of the architecture and management structure of the co-lead arrangement as a whole and around the mainstreaming of Cluster responsibilities within the two agencies.

Without sufficient data from existing sources it was decided to send out a written questionnaire focussing specifically on these issues to a limited number of people: 11 senior level Save the Children and UNICEF staff and 2 people from external agencies strongly involved with Cluster.

Seven forms were completed and returned within the limited time (10 days) available for completion (response rate: 54%) with 4 from Save the Children, 2 from UNICEF and 1 from an external agency.

Information received from the questionnaires has mainly been used within the findings section 4.2.5 and corresponding recommendations.

Findings:

The co-lead management arrangement has been set up in accordance with the MoU and is successfully delivering many elements of its mandate. However, the partnership itself has been suffering significant ‘teething pains’ and is not operating as effectively as it should in a number of areas. These partnering issues, along with management structure issues, are preventing the co-lead arrangement from achieving its full potential.

Recommendations:

Informed by the knowledge and experience gained to date and the findings and specific recommendations of this review, the co-lead agencies should institute a collaborative process to take a step back and re-build the partnership, including adapting or instigating new management structures and procedures as necessary, to ensure the co-leadership is fit-for-purpose to deliver on the Cluster’s remit.

The collaborative process could involve:
 Developing a clear vision for the partnership, including an agreed definition of what partnership and coleadership mean in practice and what ‘success’ would look like from each partner’s point of view.
 Developing clear objectives for the partnership (including explicitly what benefits are expected from colead rather than single-lead arrangement) and ensure transparency over individual objectives of the agencies.
 Developing new and revised structures and procedures to ensure the most efficient and effective delivery, and an operational plan which will best utilize the resources and competitive advantages of each agency to achieve the Cluster objectives.
 Taking a ‘clean sheet’ approach, determining the roles and competencies, and hence job descriptions, for the portfolio of staff required for effective delivery of the revised plan.
 The MoU between the agencies should be updated to reflect the new understanding and expectations of the partnership that come out of the re-building process, based on best practice guidelines for partnering agreements.

As a steer for the recommended collaborative process for moving forward, there are a number of specific recommendations which should be considered:Final Report Draft Version 2.01

Overall management architecture
 There needs to be a more active role in the oversight of the Cluster as a whole and, as the central coordination mechanism, the ECU in particular. To help achieve this, the agencies should consider splitting the Steering Group roles into two separate entities: A UNICEF / Save the Children Management Group overseeing the co-leadership arrangement (including line management of the ECU; the partnership between the agencies; institutionalizing / mainstreaming the work of the Cluster within each organisation); and a newly formed expanded Steering Group to include two or three other agency partners and UNICEF and Save the Children implementing staff (including regional and country level) which would provide guidance to, and be accountable for, the full Cluster’s strategy and activities.
 The (potentially expanded) Steering Group should more strongly oversee the work of the ECWG by increasing the resources of the ECU to play a more active role in coordinating and supporting the ECWG sub-groups and projects. The ECU should also facilitate stronger connections between the ECWG and country level clusters.
 Stronger coordination mechanisms should be put in place to ensure maximum benefits from the Cluster work mainstreamed into agencies’ staff roles. For example, there could be a regular forum between the agencies to exchange information, coordinate an approach and identify new opportunities.
 Once agreed, clear guidance should be created/updated and disseminated to explain the whole Global Cluster architecture including:
o Purpose and responsibility of each element of the Global Cluster.
o Roles and responsibilities of each agency (including external agencies) within each element.
o Definition of lines of reporting / accountability.
o Decision-making and discrepancy resolution mechanisms (where relevant).

Finance
 Strong efforts should be made to diversify the resource base for the Global Cluster beyond the Dutch government, in particular with a focus on funding that can come through Save the Children. This will firstly reduce the reliance on a single donor and secondly, if through Save the Children, will help to promote equity between the partners.
 If feasible, a trust fund (or other instrument) should be dedicated as a central pot for the Cluster to administer funding for the ECU and certain Cluster activities, with joint decision-making on allocation (i.e. it becomes ‘Cluster money’ as opposed to ‘UNICEF money used for the Cluster’).

Performance of the ECU
 The roles, job descriptions and reporting lines of the Co-ordinator and Deputy Co-ordinator be reviewed to engender more effective management and ensure stronger accountability.
 The Steering Group (or new Management Group) ensures that the ECU establishes, and adheres to, consistent and transparent processes of internal decision-making.
 The ECU Co-ordinator, in consultation with the Deputy Co-ordinator produces a twelve-month
Management Plan to demonstrate to the Steering Group how the unit will operate in order to achieve its work-plan objectives.
 Appointment of future new senior staff must fulfil the agreed procedure and be based on consensus of the partners irrespective of the employing agency.3
 The whole ECU team should be physically sited together to promote better teamwork. Given the small size of the Save the Children operations in Geneva, the UNICEF office would be preferable to maximize linkage within the partner agencies.

Monitoring and review
 The Steering Group should put in place systems and procedures for the monitoring of the efficiency and effectiveness of the Cluster management to ensure issues and problems are highlighted early. These will include the development of targets both on outputs and on process (for example on responsiveness and communication).
 A regular, informal review meeting between the two agencies (both ECU and regional/HQ staff) should take place to bring up and talk through any issues around the partnership.

3 It is noted that this has been done in the recent (September 2010) appointment of the new Coordinator.

 

 

 



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search