Evaluation database

Evaluation report

2010 Montenegro: External Evaluation of the Juvenile Justice System Reform Project



Author: Don Cipriani, Ph.D.

Executive summary

Background:
1. The Project
The European Community, represented by the Delegation of the European Union to Montenegro, contributed EUR 500,000 to UNICEF in support of the “Juvenile Justice System Reform Project.” The Government of Montenegro Ministries of Justice and of Labor and Social Welfare collaborated closely with UNICEF on the 18-month project (October 2008 – April 2010), which focused broadly on policy and legal reform, professional capacity building, and community-level actions affecting children in conflict with the law. This section highlights the project background, objectives, and activities.

1.1 Background
Montenegro is a small country in the Western Balkans (see Annex III) that experienced years of socio-economic stagnancy due to war, political turmoil, and economic isolation that affected the broader region since the 1990s. Following its independence in 2006, Montenegro enjoyed rapid economic growth and cemented its place as an upper-middle-income country. From 2006 to 2008, poverty rates declined from 11.3% to 4.9% among the total estimated population of 628,804. However, children comprise approximately one quarter of the population, and are significantly more likely than average to live in poverty.

Montenegro applied for membership of the European Union in December 2008, and its potential candidacy for EU accession is the dominating economic and political priority of the country. Among the many associated political, institutional, and legislation reforms that Montenegro is pursuing, the Government approved and is implementing its 2007 – 2012 Strategy for the Reform of the Judiciary. This strategy includes several components for Juvenile Justice System Reform.

Juvenile justice in Montenegro is currently regulated under dedicated provisions within adult criminal justice legislation, which is applied in adult criminal courts by non-specialized judges. Children younger than 14 years of age do not fall under the purview of juvenile justice, as they are only subject to social welfare and child protection legislation and responses. Children from 14 years to less than 18 years are subject to criminal law, and there are important differences in the types and severity of measures that may be imposed upon children younger or older than 16. Most importantly, only those 16 and older may be sentenced to periods of imprisonment, which in practice are served in the juvenile wing of the Institute for the Execution of Criminal Sanctions in Spuž (near Podgorica). Eight children faced such sentences during 2008, but just one in 2009. Children may also face pre-trial detention in Spuž, although outside of the juvenile wing, as well as police custody following arrest. The only child-specific institution is a semi-open facility serving both children in conflict with the law and children in need of protection: the Centre for Children and Youth “Ljubovic” in Podgorica. Children placed at Ljubovic may be refused permission to leave their rooms or the premises at the staff’s discretion.

Data and statistics on juvenile justice are incomplete and often contradictory, but there are now roughly 200-400 formal juvenile cases considered in the courts per year depending on the information source. The vast majority of these related to “property” and “life and body” crimes, and result in non-custodial measures or supervision orders. Figure 1 below illustrates the upward climb to this caseload in recent years, based on one relevant measure.

Figure 1.1: Number of Juveniles Formally Charged in Court, 2001-2008

Although the absolute numbers of children in the formal justice system are not high, assessments since 2002 found all aspects of the juvenile justice system to require a complete overhaul. In brief, legislation and policy were inconsistent with regional and international standards; professionals dealing with children in the justice system had little or no specialized training about children; conditions of confinement in institutions were unacceptable; diversion measures and alternatives to detention were not available in practice; and there were effectively no systems for monitoring, accountability, or redress.

UNICEF Montenegro had worked towards juvenile justice reform in the years leading up to Montenegro’s independence. Most importantly, the Swedish International Cooperation Agency (SIDA) supported system reform in Serbia and Montenegro from January 2004 to June 2007. Although juvenile justice challenges were considerable at the project outset, they were simply not a political or social priority at the time. Through the course of the SIDA-funded activities – such as professional capacity-building and the development of regulations for the implementation of diversion measures – political leadership and commitment grew in support of juvenile justice reform. For example, as noted above, juvenile justice reform became part of the national Strategy for the Reform of the Judiciary.

With this growing base and momentum for reform, UNICEF began discussions in January 2007 with the European Agency for Reconstruction (EAR) on potential support for further juvenile justice reform programs. The phasing out of EAR, and transfer of responsibilities to the new Delegation of the European Union to Montenegro, carried these discussions through September 2008. The European Union Delegation and UNICEF signed the project agreement for “Juvenile Justice System Reform” on 8 October 2008, and promptly began project implementation.

Purpose/Objective:
2.1 Purpose and Context
The original project proposal stipulated and budgeted funding for an external evaluation to be organized in the last quarter of the project in consultation with the Project Steering Committee. The purpose is to meet the expectation, which accompanies EU financial support, to independently evaluate project performance. Informally, the evaluation will inform discussions among the European Union Delegation to Montenegro, UNICEF Montenegro, the MOJ, and the MOLSW on potential future support for juvenile justice system reform work. The final two Project Steering Committee meetings began such talks, and noted the expectation that this evaluation will be an important consideration for the design and support of future activities.

2.2 Objectives and Scope
As stipulated in the evaluation Terms of Reference (see Annex VII), the specific objectives of the evaluation are the following:
4. To provide feedback to UNICEF Montenegro office and its national counterparts on the soundness (defined as relevance, effectiveness, efficiency, sustainability) and impact of their approach in the Juvenile Justice System Reform Project in order to:
a. Reveal good practices and gaps in their approaches;
b. Evaluate Project Impact following Project Plan, Project Logframe and Description of the Action
5. Based on the experience of Juvenile Justice System Reform Project to extract general lessons learned and recommendations aimed at further enhancement of the juvenile justice system reform.
6. Provide the Delegation of European Union to Montenegro with information on impact of their specific support to Juvenile Justice System in Montenegro.

These objectives were defined before the project evaluator’s involvement, and while they are reasonably specific, achievable, and relevant, they present some measurability challenges and are not time-bound. They do not foresee the completion of cost analysis, although some relevant considerations are presented in the Findings on Efficiency. The evaluation objectives appeared to be sufficiently clear and satisfactory to all stakeholders and evaluation participants, particularly given the context of the evaluation’s scope. The intended evaluation scope comprised 20 work days: 5 days for desk review; 10 days for an evaluation mission trip in-country; and 5 days for analysis and evaluation report preparation. This scope envisaged a comprehensive examination across major stakeholders and activities nationwide, to the depth possible within a limited time frame. Time pressures due to the project’s approaching end date compressed the evaluation schedule, precluded reframing the evaluation objectives, and conditioned the methodology as described in the following section.

Methodology:
2.3 Methodology, Stakeholders’ Participation, and Limitations
Evaluation methodology is based upon the evaluation’s specific objectives noted above, with due consideration for the evaluation’s scope, and is explicitly framed around OECD/DAC evaluation criteria (relevance, effectiveness, efficiency, impact, sustainability) plus two general guiding principles for UNICEF’s work (human rights based approach and results-based management). Specifically, interviews and focus groups were driven by these criteria as described below, and this evaluation report’s findings are structured around the same criteria. The UN Interagency Panel on Juvenile Justice is in the preliminary stages of developing Juvenile Justice Evaluation Guidelines, which will become the first broad international standards as such, but these were not finalized yet for use in determining evaluation methodology/criteria or project performance standards and benchmarks.

The data collection and analysis has been essentially qualitative due to several factors. First, many activities focused on national legal and policy reform, largely targeting government decision makers whose feedback is most amenable to qualitative interviews. Second, the evaluation scope and timeframe did not allow for structured surveys/questionnaires (design, translation, distribution, collection, analysis) that would have provided scientifically relevant quantitative data on professional capacity-building activities. Finally, few children were significantly involved in the project activities, and many of them had left the Ljubovic Centre by the time of evaluation, which effectively precluded quantitative approaches to children’s views.

As such, the evaluation methodology was based on the research tools of desk review, interviews, focus groups, and direct observation of project activities. The desk review considered well over 100 documents relevant to the project, most of which were shared by UNICEF or independently-researched (See Annex V). Overall, documents spanned the following major categories: original project proposal materials and mid-term reports to the EC (including financial reports); other EC documents and national progress reports; Project Steering Committee agendas, minutes, and Action Lists; most Terms of Reference issued to contract services under the project; most print work products generated under the project (certain law drafts, technical reviews, assessments, implementation plans, brochures, training programmes and manuals, etc.); implementing partners’ reports on their activities (workshops, focus groups, technical assistance, etc.); official government documents, reports, and legislation; UNICEF Montenegro annual reports, work plans, country programmes, etc.; selected confidential reports and e-mail communications shared by UNICEF, providing history on human rights developments in Montenegro as well as project development; and documents relating to UN and COE human rights bodies’ consideration of Montenegro. In addition, some project consultants’ final reports on the trainings/workshops they conducted included results of workshop evaluation surveys, and UNICEF shared partial evaluation survey results from one judges’ training workshop.

In order to gather the largest possible range of opinions on the project’s process and outcomes, 21 semi-structured individual interviews were conducted with project stakeholders in Podgorica, Niksic, Bijelo Polje, and by telephone (see Annex II). Interviewees included EC and COE officials; government officials and program managers; UNICEF staff; project expert consultants; and representatives of NGOs contracted under the project. Their views represented a broad array of stakeholder perspectives and roles, including as donor; interested party with minimal project involvement; project decision-maker; key implementer; direct beneficiary of project activities; contributing technical expert (both independent and fully-immersed in project); and implementing partner. Questions were adapted to various contexts from the interview protocol (see Annex IV), were posed as open questions to foster free expression of opinions, and sought to determine respective experiences and views on project relevance, efficiency, effectiveness, impact, sustainability, integration of human rights, and results based management.

Thirteen semi-structured focus groups were conducted to collect stakeholder viewpoints with greater breadth and depth. Comprising a total of 37 individuals (some participating in more than one focus group), these were held with direct beneficiaries and stakeholders, including UNICEF staff; NGO implementing partners; juvenile justice institution directors and/or staff; children in conflict with the law placed at the Ljubovic Centre; and various professional groups who received training, technical assistance, and/or participated in public discussions (prosecutors, institution pedagogical staff; social workers; police officers; judges). The focus group with children included the perspectives of three children who had participated in the theatrical play “On the good and on the bad road in life,” and of two children who knew about the play but did not participate. For other focus groups, the evaluation mission trip’s time pressures forced prioritization on professionals who benefited from and participated in project activities, and it was not possible to contrast their views with those of non-participating professionals. Prompted liberally from the interview protocol described above, focus group sessions sought to determine salient project effects across different contexts through observation of small group discussions. Questions were simple and invited participants to exchange their views with limited evaluator interference, facilitating a stronger understanding of participants’ unconditioned perspectives.

In particular, the variety and extent of interviews and focus groups reflects the evaluation process’s core attention on key stakeholders’ individual and collective involvement in project appraisal. A total of 53 people participated in interviews and focus groups, which is a statistically significant sampling of the total estimated direct participants/beneficiaries of various project activities (17% of 310 total). Moreover, these methods ensured consideration of all groups and stakeholders affected by the project.

Direct observation was possible for three main project activities: via DVD recording, the premiere in December 2009 of the theatrical play “On the good and on the bad road in life”; the working lunch with judges and prosecutors in Bijelo Polje during the evaluation mission trip, hosted by MOJ Deputy Minister Ms. Lakocevic; and the final Project Steering Committee meeting in Podgorica, also held during the evaluation mission trip. These were important occasions to verify first-hand the nature and quality of child participation, project leadership and management, and engagement with and ownership of relevant professionals in the field. In addition, focus groups that were conducted at the Ljubovic Centre and the Institute for the Execution of Criminal Sanctions in Spuž permitted cursory site inspections of residential areas intended for children (but not short-term detention areas in Spuž).

This evaluation methodology suffers from a number of important limitations. As suggested above, the absence of quantitative data was unfortunate but unavoidable for evaluation purposes. Issues of gender and under-represented groups are problematic for several reasons. Evaluation methods to fully account for the status, views, and impact for girls and under-represented groups were not feasible due to the absence of disaggregated data, the limited time for in-country evaluation, and the negligible emphasis on such groups in project activities. The only relevant evaluation consideration was one teenage girl’s participation in the Ljubovic Centre focus group, which was obviously insufficient to address gender and under-represented groups, but appeared to be the only viable approach given these constraints.

The reliance upon UNICEF to provide project documentation and translation services opened the potential for evaluation bias. UNICEF took the initiative to supply the evaluator with extensive project documentation from the outset, provided access to internal and confidential documents and e-mail communications, and shared numerous English translations of Montenegrin-language documents and reports. Due to the limited availability of translation services and logistical difficulties, UNICEF project staff translated for 5 of 21 interviews and 8 of 12 focus groups. Given that no other stakeholder was in the position to transmit such voluminous information, numerous steps were undertaken to control for the potential dominance of UNICEF as an information source.

Importantly, UNICEF appeared at all times to provide information subject only to actual availability, and there was no indication whatsoever of selectivity in sharing documents/translations that would favor UNICEF’s image or role. The evaluator repeatedly made specific follow-up requests for detailed documentation to ensure a balanced perspective, and UNICEF consistently and promptly met these requests. Similarly, the evaluator proposed a broader range of interviews and focus groups than UNICEF had initially anticipated. UNICEF dedicated exceptional efforts to facilitate all such meetings, and as suggested above, a notable representative sample of stakeholders’ views was collected. In terms of interview and focus groups translations, interviewees and focus group participants frequently had some knowledge of English, and affirmed the accuracy of translations through gestures and emphasis on key points. Ultimately, the evaluator was able to contrast multiple independent information sources with regard to virtually all project activities.

As a final measure of stakeholder participation and to ensure inclusion of all major viewpoints, Annex 1 is independently dedicated to UNICEF’s and Key Partners’ Comments on the Project Evaluation, which UNICEF should coordinate and insert into this document following submission of the final evaluation report.

Findings and Conclusions:
4. Conclusions
This section presents conclusions by significant recurring and cross-cutting themes in the evaluation’s Findings.

Notable Project Successes
Although the project did not accomplish its most visible objective of a new juvenile justice law, it has achieved a model democratic drafting process, a draft law that should conform to all relevant international standards, and the strong probability of the law’s passage in the coming months. At the same time, it appears to have produced important short- and medium-term results for juvenile justice professionals: improvement in the quality of preventive services offered; updated professional practice methodologies of Ljubovic Centre staff; and greater awareness and practical knowledge on children’s rights and the use of diversion/alternatives. These results do not present likely long-term impacts in and of themselves, but potential carry-over exists if the draft law is promptly passed and extensive efforts towards full implementation are undertaken.

Relevance, Ownership, and Stakeholders
EU membership is the overriding goal for the Government of Montenegro, and related government policies and priorities ascend accordingly in importance. The MOJ Judicial Reform Strategy serves as a direct bridge between juvenile justice reform and potential EU membership, and assured MOJ commitment. Similarly, some of the most visible project champions are in the justice sector: the MOJ Deputy Minister, the President of the Supreme Court, the Supreme State Prosecutor, etc. Such a bridge does not currently exist for the MOLSW, and without it or a substitute mechanism, the MOLSW risks lagging behind rather than boldly embracing its central role in the future juvenile justice system.

At the community level, the project didn’t fully engage the complete range of relevant stakeholders, especially parents, teachers, and police officers. Some of the most effective delinquency prevention programs are conducted in schools by teachers, and reinforced at home by parents. Police officers demonstrated an intimate knowledge of Montenegro juvenile justice dynamics that was not reflected in national statistics or reports. Moreover, these stakeholders are a core constituency and source of advocacy for juvenile justice reform.

Children’s Participation
One of the most resounding project successes was the theatrical play “On the good and the bad road in life,” created and presented by children placed at the Ljubovic Centre. The play drew over 800 spectators, brought many of them to tears, empowered children with dignity and respect, breathed inspiring new life into a tired institution, and humanized with immediacy why the partners are pursuing juvenile justice reform.

Most activities fell well below this model activity, and children’s views did not significantly inform the project overall or the few other activities that immediately affected children in conflict with the law. There was no special consideration for girls or refugee, displaced, Roma, Ashkaelia, or Egyptian children.

Efficient Project Implementation
Despite limited human resources of the project staff and key partners, implementation was efficient and generally followed the ambitious project design. Divergences from the project design were managed transparently and in consultation with all key partners, and usually represented effective choices given changing conditions. Certain lapses in follow-up or program redirection appear fundamentally due to a project workload that exceeded staff and partners’ capacity.

Data and Indicators
Data collection, reporting, and analysis are problematic in Montenegro, with repercussions for the project and measurement of its achievements. Decision-makers cannot gain from currently-available data any comprehensive understanding of juvenile justice dynamics or insights into its realities for children, and the same challenges are linked to project weaknesses in child and stakeholder participation. The project would have benefited from stronger national data-related capacities (which the project fully assessed) and higher-quality and more meaningful indicators.

Recommendations:
5. Recommendations
Based primarily upon evaluation Findings and Conclusions, the following Recommendations take into consideration the potential continuation of project activities in the future.

1. Passage of the Juvenile Justice Law. Project partners should consider some linkage between the juvenile justice law’s entry into force and discussions on future project phases. The approved law should be translated and should undergo independent technical legal review to ensure full compatibility with all relevant regional and international standards.

2. The MOLSW’s Role. Project partners and other stakeholders should engage with the MOLSW for a richer common understanding of the essential links among potential EU membership, children’s rights, and the MOLSW’s institutional priorities. Ideally, the MOLSW’s vision, strategy, and responsibilities would incorporate these links; regardless, some relevant internalized mechanism is necessary to ensure that the MOLSW fully embraces its central role in juvenile justice.

3. Other Partners and Stakeholders. Future efforts should feature the formal high-level participation of the Ministry of Education and Science and the direct participation and involvement of teachers. Likewise, law enforcement officials, sector- and community-level officers, and juvenile specialist investigators must be deeply involved in future planning and implementation. Parents, especially parents of children in conflict with the law, should be included directly in project activities to the greatest extent possible.

4. Children as Key Stakeholders. Project partners should ensure children’s active and meaningful participation in all relevant future activities, including substantive consideration of children’s views in formulating future project proposals. As necessary, child participation should draw upon UNICEF’s relevant global expertise, and future activities should replicate and expand upon the success of the theatrical play “On the good and on the bad road in life” as a priority. Children in institutions should enjoy an array of similarly stimulating activities that help reintegrate them successfully into their communities, and their rights and perspectives on living conditions require greater attention. Project partners should consider and address the situation of girls and refugee, displaced, Roma, Ashkaelia, and Egyptian children.

5. Embracing a Bolder Vision. As current project results are solidified and partners/stakeholders are further engaged (i.e., Recommendations 1-4), project partners should reflect creatively and boldly about their vision for children in Montenegro. Montenegro’s unique circumstances – such as sweeping momentum towards EU membership, exemplary high-level political support, enthusiastic donor interest, full UNICEF investment, and sound project successes – present exceptionally rare strategic opportunities that should be seized, and that could be leveraged to build a model culture for children’s rights. Juvenile justice reform could play a leading role, and should be seamlessly intertwined with universal violence prevention programs, comprehensive social welfare sector reform and long-term capacity-building, and other initiatives. Future efforts should aspire to have a decidedly expanded scope and time frame.

6. Focus on Law Implementation. Under that bold vision (Recommendation 5), future objectives should focus on full implementation of the new juvenile justice law, including through the prompt development and introduction of secondary legislation (implementation guidelines). Evaluation sources suggested almost unanimously that law implementation is a greater and more difficult challenge than law creation. Despite the extremely ambitious project activities to date, future efforts toward implementation must be even greater. These should expand upon project successes, especially value-added deliberative processes, specialized professional trainings, and intensive expert technical assistance. Project shortcomings require concrete solutions to blockages, particularly to the actual use of diversion and alternatives.

7. Expanded Project Capacity. Despite limited human resource capacities of the project staff and key partners, the project demonstrated notable efficiency and transparency in executing an ambitious, complex project. Recommendations 5-6 above imply a realistic expansion of those capacities, in core project staff and support, and in contributions to long-term ministerial capacity. In this sense, project partners may consider the expertise gained through UNDP Montenegro’s major outcome area of “capacity development for public management.”

8. Data, Indicators, and Monitoring. As detailed in a project-supported analysis, data collection and reporting capacities pose deep challenges that require significant attention and collaborative solutions. Ideally, future project activities should fully incorporate the UNICEF/UNODC Juvenile Justice Indicators into those solutions, and to the practical extent possible, they should feature the same indicators as the basis for measuring project results. Related to the current inadequacy of formal data/statistics, project activities should expand NGO and Ombudsman Office capacities for alternative data gathering and monitoring. In particular, the Ombudsman’s Office must have guaranteed, unfettered access to all places of deprivation of liberty at all times, and should possess updated information at all times on the number and status of children deprived of their liberty.

Lessons Learned (Optional):
4.1 Lessons Learned
Lessons learned, in the context of program evaluation, refer to new contributions to general knowledge with relevance beyond the project, sector, and context under evaluation. As many of the project dynamics were intrinsically-linked to the unique contemporary circumstances of Montenegro, most of the important lessons that the project itself learned have limited value-added applicability beyond this context.

The one project lesson learned with such applicability is the exponential project value of meaningful child participation in creative activities, in this case of children in conflict with the law in a dramatic production based upon their own life experiences. Respect for the views of the child is a guiding principle of the CRC and a leading principle for comprehensive juvenile justice policy, thus full child participation is an already-accepted lesson. However, examples that target disadvantaged groups and achieve remarkable results are not well-known and are rarely replicated.

In summary terms, the project invested modest financial resources over 3 months to support the local NGO Proscenium, which developed a theatrical play with children placed in the semi-open Ljubovic Centre. Based on their own life experiences, over 20 children participated in the play “On the good and on the bad road in life,” and presented it 5 times in December 2009. There were approximately 830 spectators across the performances, and positive television, radio, and print media coverage was extensive. Beyond the children’s own strengths, the NGO’s experience in professional theater and drama production, and in working with disadvantaged populations (Rom, refugees, etc.), was fundamental. The children’s dramatic performances brought audiences to tears, and within the Ljubovic Centre staff and children “breathed as one for the first time.” One of the NGO’s leaders identified the most important result as the “pure joy” of the children, and called the experience “the best project in my life.”

This project activity exemplified the transformative power of meaningful child participation in ways that are rarely seen. Rather than as the exception or isolated activity, children’s rights designate such work as a core value and basis for all work with children. The theatrical play humanized with immediacy the project’s goals, empowered children with dignity and respect, advanced the realization of children’s rights.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search