Evaluation database

Evaluation report

2010 Mozambique: Assessment of UNICEF NGO -- Government Partnership Programme



Author: Minna Tuominen, Hanne Roden, Katia Herminio

Executive summary

 

“With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding”, “Good”, “Almost Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.”

Background:
The HIV/AIDS epidemic is the major contributor to the increasing numbers of orphaned and vulnerable children. According to the preliminary report of Multiple Indicator Cluster Survey (2008), 17.2% of Mozambican children between 0-17 years are orphans or vulnerable because of AIDS. The Mozambican Plan of Action on Orphans and Vulnerable Children (PACOV) recognizes that the needs of the orphaned and vulnerable children are inter-dependent and that various and coordinated interventions are required if they are to make a significant difference to the children’s lives. In this context the Action Plan identifies six areas of the most fundamental needs: i) health, ii) education, iii) nutritional and food support, iv) financial support, v) legal and vi) psycho-social support. The Government of Mozambique has decided that at least three of these basic needs must be satisfied by any given programmeme or project aiming at supporting orphaned and vulnerable children.

Since 2007, UNICEF Mozambique has been running a Partnership Program with the Government of Mozambique and non-governmental organizations (NGOs) with the objective of increasing access to basic services for orphans and vulnerable children (OVCs) so as to decrease the burden on families and communities taking care of OVCs. The Program has established partnerships with 12 NGOs who are in charge of the implementation of activities at provincial, district and community levels.

Purpose/Objective:
UNICEF believes that this is an opportune time to take stock of the UNICEF-NGO-Government Partnership Programme. From a programmatic perspective, the OVC National Action Plan is being updated, so the findings of this stock-taking exercise will be able to influence components of the Action Plan related to partnerships with civil society for reaching out to OVC. On a more prosaic note, the PCAs are due to expire at the end of this year. While UNICEF has closely monitored the work at a district level and has also held regular review meetings with the partners, they also needed a broader perspective to help inform programme planning vis-à-vis this strategy for the duration of the extension of the Country Programme (2010-2011). Therefore, in line with the 2009 Annual Work Plan, the Child Protection Section required a stock taking exercise.

Methodology:
The evaluation assignment took place between August 2009 and January 2010. The evaluation has been mainly conducted using qualitative research methods, comprising of:
• Literature review;
• Key informant interviews with NGO representatives, involved government institutions and with UNICEF representatives;
• Focus group discussions with beneficiaries of the NGO’s interventions and with other community members in selected provinces/districts
In addition, the consultant team has gathered quantitative data related to the achieved results and to the financial execution by each NGO.

Findings and Conclusions:
The Partnership Programme was designed in 2006 as a short-term response to a near emergency situation – that of the rising numbers of children orphaned due to HIV/AIDS. The goal at that time was to contribute to the objectives of the PACOV in terms of stablising these households and ensuring they had access to basic services while setting up more effective, longer term community supports. As this report has demonstrated, the Programme has generated many positive results in this context, though specific recommendations for its improvements have also been identified. Given these findings, as well as the continued impact of HIV/AIDS in Mozambique, UNICEF would do well to continue the Programme, while also improving it as suggested. As we enter not only a new decade but also potentially a new National Plan of Action for Children, the continued partnerships between the Government of Mozambique, UNICEF and civil society organizations will be key to fulfilling the rights of orphans and vulnerable children.

Relevance of Programme Interventions
In general, many efforts are made by the NGOs to ensure that the assistance is relevant to the target groups. Practically all the NGOs involve local community members or representatives of local CBOs in the process of identifying those children/families who are most in need and in defining the type of assistance needed. Through the involvement of local leaders and community members, the NGOs seek to ensure that the assistance – be it material or non-material – obtains wider acceptance among the community members and responds to the most acute needs of the most vulnerable people. In order to better understand the necessities of the local communities, some NGOs carried out a situation assessment in their target communities prior to the commencement of the activities. Some other NGOs initially identify the potential beneficiaries with the help of CBOs or local community leaders. Thereafter, they interview all the identified people so as to assess their level of vulnerability and to define their individual needs, case by case.

As stated above, 7 NGOs are involved in the provision of technical assistance to DPMAS. The nature of the technical assistance to DPMAS was initially defined jointly by UNICEF and the MMAS. Both the terms of reference and the budget related to the technical assistance to DPMAS were defined by UNICEF, although one UNICEF key informant affirms that the NGOs and DPMAS were given freedom to adapt the TORs into their local context. However considering the somewhat uniform approach to this component, it can be concluded that the NGOs did not significantly modify the TORs but started to implement them as they were. All the NGOs provide one Technical Adviser who works at the DPMAS side by side with the OVC Focal Point. The main responsibility of the Adviser jointly with the Focal Point is to create and support a provincial level Multisectoral Nucleus for OVCs (NUMCOVs) and a Technical OVC Group (GT-COV). The former is a political entity while the latter is an executive organ.

While the structure of technical assistance was to some extent defined at a national level, the importance of strengthening the provincial level coordination and boosting commitment for caring of OVC is widely recognized by both NGOs and the MMAS key informants. “I believe we have responded to the most critical needs” affirmed one NGO key informant but continued stating that much more support is still needed especially at district level where both the material working conditions and the technical knowledge of the Technicians of Social Assistance are very weak. In some cases, however, where the DPMAS has not prioritized the NUMCOV agenda and has participated very little in its management, the provision of technical assistance has become a source of frustration for the NGOs.

All the NGOs operating at community level affirm that their interventions are based on active reflection of the different needs of women and men, or girls and boys. The most common approach to gender integration appears to be prioritization of women and girls as the primary target groups. The quantitative results indicate that the Programme has indeed benefited slightly more girls than boys. However, the difference is so narrow (50.35% girls vs. 49.65% boys) that one can by no means speak of specific targeting of girls. Hence, it was concluded that the Programme as whole and the NGOs individually have failed to systematically prioritize them.

Effectiveness of Programme Interventions
The main objective of the NGO-Government Partnership Programme was to provide access to basic services and social protection to at least 165,000 OVCs and their families. This target was common both to DPMAS and the implementing partners (NGOs and CBOs).

Overall, the Programme has generated several positive results. Over the last three years, it is estimated that the NGOs have supported some 60,000 children in accessing at least three of six basic services, thereby contributing to the overarching goals of the PACOV and UNICEF in supporting OVC. The NGOs have strengthened the technical and financial capacities of community based organizations and community groups across the seven provinces. Finally, the Programme has managed to foster the creation of provincial level coordination mechanisms through the provision of technical assistance.

However, there are also a few serious shortcomings which limit its effectiveness. The overall monitoring system of the programme is not functional, with room for improvements in terms of data collection, validation and analysis. All the NGOs appear to have just one indicator in common: the number of services provided. Keeping track of the different types of support required by and provided to each child is a major challenge to all OVC stakeholders which will need to be addressed urgently. While in general the DPMAS staff are aware of the Programme, it appears that the role of central level MMAS in the context of the Programme is not quite clear. It appears that the Ministry has greater expectations in relation to its participation than what has happened so far. Although, this may not be quite appropriate, the roles and responsibilities of the Ministry should be clarified to all the parties. Moreover, the oversight of the functioning of the provincial and district level GTCOVs and NUMCOVs should be strengthened.

Efficiency of Programme Interventions
Calculating on the basis of the data provided by UNICEF, the original budget of the Programme was nearly USD 9.9 million. The average budget of each NGO was approximately USD 822,000, but there are significant variations between the NGOs. By the end of 2009, the NGO partners have executed in average 69% of the budgeted funds. While there are great differences in the level of disbursements between the NGOs, all of the NGOs who provided financial data have spent the largest proportion of their funds either on salaries and other personnel related expenses or on training costs. Considering that a large part of the NGO interventions consists of provision of technical assistance and of capacity building, it is natural that the personnel expenses are high.

With an average of USD 16.37, the NGO partners have provided over 409,000 basic services to OVCs during a period of 3 years. With this level of efficiency, it has cost in average USD 49.11 to provide one child with three basic services. Assuming that the combination of these services has mitigated the situation of vulnerability, this may be deemed a small price to pay. However, without a baseline assessing the initial level of vulnerability, it is difficult to fully validate such a conclusion.

Sustainability of Programme Interventions
Sustainability is the Achilles’ heel of the UNICEF NGO-Government Partnership. In order to avoid creating a circle of dependency, most of the NGOs are training either CBOs or community groups to find solutions and respond to the most basic needs of their communities. This is very time-consuming process. Usually, NGOs who provide direct material support produce fast results as they can easily reach their target groups. However, direct support hardly contributes to lessening the vulnerability of people. For the NGOs involved in community mobilization, the beginning is often painstakingly slow and it can easily take a year or two before any concrete results can actually be seen. However, the results of the latter group are likely to be much more sustainable than those of the former group. Ultimately, all the NGOs seem to agree that the communities will continue requiring external assistance for quite some time. Hence, more money and more time will be needed to truly lessen the vulnerability of the communities. Some NGOs, such as Africare, have funds from other donors with which they can continue supporting OVCs, albeit at a lesser scale.

Most of the NGO key informants affirmed that some results have been achieved at the level of coordination but at the same time many of them express reservations in relation to the extent to which the capacities really have improved. The local DPMAS do not show sufficient competence or commitment to carry on alone. According to one NGO representative "More time is needed for technical assistance in order for the NUMCOV to be consolidated at provincial as well as at district levels". It seems that MMAS does not supervise the provincial or district level NUMCOVs and the DPMAS are not held accountable for the process of creation or functioning of the local level coordination mechanisms.

Impact of Programme Interventions
In the context of UNICEF NGO-Government Partnership Programme, the consultant team came across with some elements that indicate that direct material support may generate also counter effects and render people passive reducing their own initiatives. Non-material support appears to have less of such negative side effects. The interviewed beneficiaries did not even mention non-material support when asked about the benefits of the NGOs’ interventions. Undoubtedly, improved access to basic services, such as education and health, is likely to have greater and longer lasting impact which may, at its best, change the level of well-being of a whole family. What comes to community mobilization, it tends to have very empowering effect as it builds people’s awareness of their rights and teaches people to help themselves.

The changes that have been achieved through technical assistance, the key informants mentioned most often i) improved knowledge base related to children’s needs and rights among the OVC Focal Points, ii) reinforced political commitment for the cause of OVCs, and iii) the creation of local level mechanisms to enhance the coordination of interventions targeting OVCs (NUMCOVs and GT-COVs). However, each of these achievements continues presenting major challenges.


Recommendations:
The UNICEF NGO Government Partnership Programme has been run for 3 years and it has generated many positive results. If the Programme were to be concluded in 2009, many of the achievements so far would most likely die out within a relatively short period of time, as the obtained results are still not solid enough. This would be the likely destination of the community self-help groups as well as many provincial NUMCOVs. The achieved changes require further consolidation. And considering the still prevailing scope of vulnerability, extension of the Programme would seem well justified. On this basis, it is found necessary to extend the Programme at least until 2011 when the current UNICEF Country Programme ends. The main conclusions and recommendations of the consultant team include the following:

1. Reinforce the Programme management
Thus far, the 12 NGOs have operated very independently and their management tools – work plans, reports or monitoring mechanisms – vary significantly. Consequently, it is very difficult to form a proper overview of the Programme and the achieved results. In the new programme cycle, UNICEF should establish a common system of programme requirements and rationalize the programme process based on the experiences of the past 3 years. During the coming 3 years, UNICEF should make improvements in terms of harmonizing and rationalizing the Programme Administration; emphasizing exit strategies in the new PCAs; and

2. Strengthen the national leadership
Despite many efforts, MMAS still operates with limited human resources and competences. The MMAS key informants acknowledge that coordination of the actors and interventions remains a critical challenge and yet MMAS as an institution needs to embrace the urgency of that challenge. In order to reinforce the national leadership, MMAS needs to be held accountable for the functioning of the provincial and district level coordination mechanisms. UNICEF together with other MMAS partners should support the revision of the PACOV, which expires in March 2010, and ensure that in the future the objectives of the next version of the National Action Plan will be truly ‘SMART’. Also, in the context of the revision of PACOV, UNICEF together with the implementing NGOs should advocate for truly multisectoral response to OVCs and identify responsibilities of the different sectors in addressing the diverse needs of vulnerable children. UNICEF should assist MMAS in identifying the most essential training needs of its staff at central level and provincial levels.

In order to be able to effectively monitor the number of children who have received three or more services, there needs to be a national OVC database that tracks down each person who receives some type of assistance. In order to avoid duplications, the coding of each case should be based on the ID number of the beneficiary. Further, such database should also include information on the actual needs of each child so as to enable measuring the extent to which the provided support has alleviated the child's vulnerability. While supporting the establishment of the provincial level OVC databases, UNICEF should ensure that they are all synchronized and based on such recording system that enables effective counting of the beneficiaries and not only of the service provision events.

3. Improve knowledge management
The interventions of the 12 NGOs have generated plenty of knowledge and experience of dealing with OVCs but that is not currently been capitalized. UNICEF runs a risk of missing a great opportunity of increasing the understanding of good practices related to OVC response in Mozambique. Steps to address this could include involving the NGOs in a process of systematic gathering of lessons learnt in different programmatic areas so as to benefit of future programmes and inform the forthcoming revision of the National OVC Action Plan.


Lessons Learned (Optional):
As stated earlier, the UNICEF supported NGO Government Partnership Programme
has produced a wealth of knowledge on dealing with the needs of the most
vulnerable children. While it is recommended that UNICEF will systematically gather
and document this knowledge, we here at the end of this report present briefly the
main lessons that were learnt during the current assignment.
• In order not to overlook any children, focus should be placed on vulnerable children,not only on those who are orphaned. Similarly, in order to have a more effective and lasting impact, focus should be placed on families rather than on individual children.
• Another important lesson is that priority should be given to children from 0-3 years as adequate care, stimulation and nutrition in early childhood can lead to positive physical, socio-economic and cognitive outcomes that can be seen well into adulthood.
• In the context where social values and norms sustain and reinforce the vulnerability of women and girls, all the interventions that seek to lessen social vulnerability should actively promote women’s rights, seek to empower them and help to overcome discriminatory barriers and gender inequalities. Also training activities, technical assistance and institutional capacity building should promote gender sensitivity so as to ensure that the provision of basic services will prioritize women and girls as the most vulnerable members of society.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

New enhanced search