Evaluation database

Evaluation report

2012 India: Evaluation of District Planning and Monitoring Unit



Author: Deloitte India

Executive summary

"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is "Outstanding, Best Practice", "Highly Satisfactory", "Mostly Satisfactory" or "Unsatisfactory" before using it. You will find the link to the quality rating below, labeled as 'Part 3' of the report."

Background:

The District Planning and Monitoring Unit (DPMU) was established in the context of a policy environment which was inclined towards enhancing decentralization of planning systems. The DPMU was set up in Koraput due to its unique situation and existing socio-political vulnerabilities. The analytical cell in the DPMU of Koraput was set up by UNICEF (as a part of its integrated district approach) in partnership with the district administration to institutionalize an effective decentralized planning and community monitoring process, to improve the planning and monitoring of flagship programmes; and to strengthen evaluation and research at the district level. UNICEF engaged Deloitte Touche Tohmatsu India Private Limited (Deloitte) to undertake a detailed evaluation of this analytical cell.

Purpose/Objective:

The rationale for this evaluation is derived from two main areas:
UNICEF‟s internal assessment of the outcomes and impact of its intervention in Koraput, especially to inform the process of its next phase of strategic planning for the country (2013-18).
Eliciting learning from the current Koraput experience for scale-up and replication in Odisha and in other states, in the context of the proposed state-wide scale-up plan of the Government of Odisha to set up analytical cells in all the 30 existing DPMUs.

The specific objectives of the evaluation include assessment of the following parameters, as detailed in the Request for Proposal issued by UNICEF:
Relevance - whether the design and interventions of the DPMU-AC were in line with the requirements
Effectiveness - the extent to which the objectives of DPMU-AC have been achieved
Impact - the extent to which the DPMU-AC has been able to change the planning process and service delivery in the district
Efficiency - whether resources have been used economically and within the specified timeframe
Sustainability - whether the institutional mechanisms put in place will ensure the continuation of the DPMU-AC

Methodology:

The evaluation followed a qualitative methodology, which implies that the focus was on participation of all relevant stakeholders, and not randomization; on inferences based on divergence and experience, and not on generalizations; and on changes manifested through processes, and not on quantifiable indicators. The three primary methods used were desk research, document review and participatory qualitative research.
The desk research helped to set the policy context in the country and in Odisha, the socio-political context of Koraput, and the history and evolution of the analytical cell.
The document review was used to understand the details of activities and their results for chronological and thematic analyses; information about human resources; organizational structure; and evolution and change over time. This also formed the basis for forming the lines of inquiry in designing tools for the participatory research.
The participatory qualitative research involved in-depth interviews and group interviews with stakeholders. This was used to understand stakeholder perceptions about the analytical cell and the processes of change catalysed by it.

The sampling plan for the evaluation was developed to achieve maximum representation from all relevant stakeholders. This involved two phases – first, the selection of the comparison district (Nabarangpur) on the basis of a-priori matching criteria and existing intervention context, and the selection of two blocks each within the focus district Koraput and comparison district Nabarangpur (following the principle of maximizing divergence); second, the selection of stakeholders from these geographies to ensure representation from the government (at state, district and block levels), the civil society (at the district level), and UNICEF (at state and district levels). The total number of stakeholders covered through interviews was 38 across all categories.

Data collection tools - interview guides were developed based on the lines of inquiry informed by the desk research and document review. The collected data was collated and both chronological and thematic analyses were undertaken for the lines of inquiry under the five parameters of this evaluation – relevance, effectiveness, impact, efficiency, and sustainability.

Findings and Conclusions:

Relevance: The evaluation aimed to analyse the concept and design of the analytical cell as a complement to the existing government structure of planning processes and monitoring systems of on-going flagship human development programmes. The main findings and inferences related to the relevance of the analytical cell are as follows:
The stated objectives in the original design of the analytical cell were relevant and strategically aimed to address the areas of weakness that were identified in programme planning and monitoring.
The modalities followed for the operationalization and functioning of the analytical cell, including its structure and reporting relationships, have played an important role in helping the analytical cell meet its objectives.
The design of the analytical cell and its stated objectives complement the planning and monitoring processes.

Effectiveness: The study involved an evaluation of the extent to which the analytical cell achieved its stated objectives through its activities; the nature of its partnership with government departments at the district level, and the mutual contributions of the analytical cell and the government departments/district administration towards the overall outcomes of DPMU. The main findings and inferences are as follows:
The analytical cell has been involved in a wide range of activities over the years that have cut across multiple functional and sectoral areas to effectively meet its stated objectives.
The objectives and activities have been evolving through the years and the focus has changed from strengthening decentralized planning to an increased focus on monitoring government programmes and schemes.
These activities of the analytical cell have helped it to achieve its stated objectives to a reasonable extent.
Despite an evident evolution of the objectives of the analytical cell, as reflected in its prioritization of activities and profile of human resources, the same has not been officially documented and neither has the original programme document been revisited during the existence of the institution for any strategic review.
The analytical cell has been working closely with government officials in a collaborative manner. It is uniquely positioned with linkages with the District Collector, District Planning Office, DRDA, DPC, Zilla Parishad and with Line Departments. This strategic positioning has brought about convergence of activities and efforts towards better decentralized planning and monitoring in the district.
While it has maintained strong relationships with most departments in the district, these relationships have not been formally documented through legal memoranda.

Impact: While there have been early results, the analytical cell is still at a nascent stage and improvements in output indicators or quantifiable results are still to emerge. Evidence in the form of quantifiable data or causality for improvements in education, health, nutrition, rural development indicators is beyond the scope of this study. Further systemic changes take time to result in impact. Instead, change is manifested in the processes and systems being followed in the district as well as stakeholder perceptions. The analytical cell‟s impact is indicative in the conceptualization and implementation of innovations for strengthening decentralized planning and monitoring. Apart from unique innovations, the DPMU has also strengthened and improved existing systems and processes in place. The main findings and inferences are:
While the analytical cell has been impactful in many of its activities related to its objectives, there is lack of conclusive evidence yet about the impact of the institution as a whole within the public system. The evidence of such impact is complex to establish, and often requires long term sustainability of interventions.
The analytical cell has catalysed a number of innovations in Koraput, especially in the area of monitoring flagship programmes and undertaking research studies. Besides innovations, the analytical cell‟s impact is evident in strengthening existing systems and processes related to decentralized planning and monitoring viz. improving district information flow within and between departments and with the district administration, availability of quality data leading to better usage of data, strengthening monitoring mechanisms, filling any gaps felt by the district administration.
In the area of decentralized planning, the impact of the analytical cell has been more indirect and facilitative through the provision of pertinent data that aids planning decisions at different stages – needs assessments, feasibility analyses, resource allocation and analysis support.
In the area of monitoring, the impact of the analytical cell has been through the conceptualization and implementation of innovative monitoring mechanisms and initiatives across different flagship programmes. These activities, in turn, have had impact in terms of corrective action. However, these activities/monitoring initiatives have not been followed up or regularized into systems – which are needed for sustained impact.
Data generation and analysis emerges as the most significant role and area of impact of the analytical cell. However, the generation of data is determined to a large extent by requests from the respective departments, rather than a selected list of indicators that are published with pre-determined regularity and on the basis of standardized data sources.
A significant area of impact of the analytical cell has been the process of bringing about convergence amongst different departments through the creation of forums like the Technical Sub-Committee Meetings and District Core Committee Meetings, as well as through initiating joint action on monitoring systems.

Efficiency: This study evaluates the efficiency in resource management, financial reporting, monitoring and review mechanisms of the analytical cell. The main findings and inferences are:
- T
he analytical cell is fully funded by UNICEF till December 2012. The current budget allocation is reasonable for the activities carried out by the analytical cell indicating good financial planning. However, its funding has been decreasing over the years, keeping in line with its redefined role as a catalytic supporter.
Routing the budget disbursal through the district administration is aimed towards increasing ownership and control of the analytical cell as a part of the government system.
For financial reporting, the analytical cell maintains budget and expenditure details and these are shared with UNICEF on a quarterly basis. This is a strong and efficient system of financial management.
While internal reporting and monitoring mechanisms are in place and working efficiently, they have tended to become more informal over the years.
There is a differentiation of programme budgets of the analytical cell from its organizational budgets. It will be important for the government to have visibility into the overall cost including the spending on programs activities at the time of takeover of the Analytical Cell of Koraput and during scale up and replication.
With its lean staffing, there is efficient utilization of resources by the analytical cell. However, whether this is optimal to attain the objectives is doubtful. With the increasing dependence of the district administration on the analytical cell, and consequently, the increasing workload on the staff members, more human resources may be required to achieve the objectives of the analytical cell while at the same time maintaining quality.

Sustainability: An important focus area of the evaluation was to assess the sustainability of the analytical cell i.e. extent to which its institutional mechanisms, its modalities and their effectiveness for operationalization and functioning are sustainable; the sustainability of partnerships with government departments at the district level; and the assessment of need for the analytical cell to continue in the future. The key findings and inferences are:
- T
he concept of the analytical cell as an institution is sustainable in the current context where the public system acutely lacks the ability to generate robust data and analyse it for corrective action, monitoring, planning and reforms.
The institutional mechanisms of the analytical cell in terms of its design, organogram and formal positioning within the government system are efficient and sustainable to continue. However the modalities followed for operationalization and functioning, changes in these modalities over time, or shifts between originally stated vision and practice has been largely informal in nature and mostly undocumented.
The analytical cell‟s relationship with the district administration and various government departments has been largely individual driven, determined by current incumbents and leadership in these administrative positions. The lack of documentation or legal memoranda of understanding with government departments raises concerns about the sustainability of its roles and contributions in their respective flagship programs.
To a large extent the performance of the analytical cell has been driven by its personnel. Given the small size of the unit, it also becomes difficult to isolate the performance of individuals from the institution. This calls for knowledge management to maintain continuity of learnings in the event of attrition.
The financial support required to meet the salary and routine operational costs of the analytical cell is not high and should not be an issue for sustainability. However modalities for supporting the programme costs need to be clearly developed.
The unanimous felt need amongst stakeholders for the analytical cell and the GoO‟s proposal for setting up analytical cells across the state are key drivers for sustainability. However strategies need to work towards ensuring a smooth transition without breaking the current momentum.
While UNICEF has worked towards the sustainability of the analytical cell through gradually withdrawing from functional modalities, withdrawal of its funding at this stage would have serious negative implications for the sustainability and continuity of many of its initiatives. An interim strategy therefore needs to be worked out.

In summary, the analysis from the assessment indicates a clear relevance of an institution like the analytical cell, with planned objectives and a defined structure, in strengthening decentralized planning and monitoring in the district. Based on the evaluation findings, some key factors have been identified for the success of an intervention such as the analytical cell. The key learning is shared below to inform the process of scale-up and replication.

The findings from the assessment indicate clearly the relevance of an institution like the analytical cell, in strengthening decentralized planning and monitoring in the district. Despite an evolving role, the DPMU-AC has been effective in planning and monitoring activities for flagship programmes of different departments, and the impact of the cell is manifested in process outcomes of these programmes. Along with the perceived value expressed by stakeholders, this establishes a sound case for the analytical cell to continue in Koraput, and for elements of the model to be replicated across the state. The fact that it is a small unit with relatively low operating cost supports this case. Having seen the positive experience in Koraput, the district administration needs to prepare a time bound action plan for taking the ownership of the analytical cell. During the interim period, UNICEF may consider an extension of support to avoid discontinuation of activities. The state needs to integrate the learning from the Koraput experience while prioritising the development of the model for scale-up of the Analytical cell.

Recommendations:

Based on the above learning, possible recommendations/ action points for future programming and strengthening of the analytical cell in Koraput and in the planned scale up in Odisha are presented below.

Recommendations for scale-up and replication:
Strategic institutional positioning of the DPMU
Objectives and role of DPMU must factor in specific needs and context of the district
Analytical Cell must be a flexible and dynamic institution
DPMU must have formal linkages with line departments and with block level administration
Analytical Cell must have strong linkages with other units of the DPMU
Regular capacity building of staff should be conducted to ensure knowledge updation

Recommendations for the Analytical Cell in Koraput:
Based on the analyses, and observations presented above about the key learning, the following main recommendations are put forth by this evaluation study in three areas – a) institutional and policy change, b) structural level, and c) operational level.

Institutional and policy level
Continue and replicate the DPMU
UNICEF should extend interim support while District Administration takes up ownership of the analytical cell in Koraput to ensure continuity of service
The objectives of the analytical cell should be fine-tuned based on contextual needs and should reflect specific needs of the district
Maintain a balance between flexibility to respond to contextual needs and focus on larger goals

Structural level
Staffing of DPMU should remain lean but optimal to work load and human resources should be organized in alignment with core objectives
Need to formalize linkages with line departments, create linkages with block level administration and strengthen linkages with other units of the DPMU

Operational level
Need to make objectives and KRAs specific and undertake strategic reviews to ensure alignment of activities to objectives and goals
Need to strengthen reporting and monitoring mechanisms and strengthen performance reviews
Need to undertake formal financial reviews to provide district administration scope for budgeting activities, and for prioritization of activities and utilization under different core areas.
Ensuring an appropriate mix of activities between routine and ad hoc activities based on requests from stakeholders
Strengthening Knowledge management at an institutional level
Ensuring regular capacity building of staff for knowledge updation
Ownership and absorption of DPMU- AC activities into the regular government system

Lessons Learned:

The key learning from the Koraput model are:
The strategic institutional positioning of the institution along with effective leadership is crucial in determining its impactful functioning and sustainability.
The structure also has to provide for effective linkages with the District Collector, District Planning Office, DRDA, DPC, Zilla Parishad and with Line Departments which also need to beinstitutionalized. Besides this, the analytical cell needs to maintain links with civil society.
Many factors may independently, or in interaction with each other, contribute to a need for shifting the focus areas of an institution like the analytical cell such as, national policies, policy context in the state, development priorities, etc. However the institution has to be flexible and dynamic to respond to such changes. In doing so, the institution needs to ensure that it remains relevant – this is possible by being cognizant of the objectives and goals and ensuring that it serves the need of the end users.
Clear statement of goals and objectives of the analytical cell is important, with institutional mechanisms to review these regularly against activities and changing policy needs.
Lean but optimal staffing with right expertise is important to ensure quality of services which in turn is imperative for driving demand for services. While the desire may be to staff the cell with many specialists, availability of quality human resources needs to be borne in mind.
Balancing increasing demand while maintain quality is essential to ensure value creation and sustainability.
Knowledge management is critical along the life cycle of the institution - In order to protect the functioning and role of the analytical cell through change in leadership, human resources etc., it is important to have official documentation of the evolution and rationale of change along the project life cycle.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search