We’re building a new UNICEF.org.
As we swap out old for new, pages will be in transition. Thanks for your patience – please keep coming back to see the improvements.

Evaluation database

Evaluation report

2014 Barbados: Evaluation of the Pilot of the Early Childhood Health Outreach Program in St Vincent and the Grenadines



Author: Prof. Maureen Samms-Vaughan, Dr. Charlene Coore Desai, Ms. Jody-Ann Reece and Mrs. Sydonnie Pellington

Executive summary

With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report.

Background:

St. Vincent and the Grenadines cognisant of the importance of early childhood development, and in particular development during the first three years of life, made the decision to implement programmes for vulnerable children in this age group.  The Roving Caregivers Programme was implemented in 2004 with the support of the Bernard van Leer Foundation. The RCP, while receiving some support from the government through the health sector operated primarily as a privately run organisation with its own staff. Inevitably, sustainability of the programme became a challenge.  As with other countries in the Caribbean, the option of delivering the programme through existing paraprofessional community based health staff, known as Community Health Aides was considered. The Ministry of Health, Wellness and the Environment of the government of St. Vincent and the Grenadines, with technical and/or financial support from other ministries and that of regional and international development partners, developed, planned and implemented its own delivery model, known as the Early Childhood Health Outreach (ECHO) programme. A pilot programme was decided on, the results of which would guide the national implementation of such a programme.

The programme, which was piloted in two phases in 2010 and 2011, had as its overarching objective the provision of health stimulation for at-risk children.  This was to be met through six sub-objectives:

•To offer early stimulation to young children
•To improve parenting practices in “at risk” communities
•To promote and monitor good health and early development of young children
•To train CHAs from the communities in the pilot district to assist in the delivery of parent support services
•To transfer knowledge and skills to parents and communities
•To promote healthy relationships between parent and child

Purpose/Objective:

In 2014 an evaluation of the pilot programme was commissioned, based on the need of Government partners in the MHWE in SVG to determine programme effectiveness and output in order to guide scaling up, and as an organizational requirement for UNICEF, who invested in the pilot programme.  Based on the Terms of Reference, the specific objectives of the evaluation were:
I. To assess how far the ECHO pilot programme has achieved its objectives, the results that have been achieved to date, any unintended results from the programme, as well as outputs at the individual, household, and community levels
II. To identify the opportunities and constraints the programme has faced and draw lessons and good practices from them 
III. To evaluate the operational effectiveness of the pilot and to cost its scale up in the current and projected national fiscal situation
IV. To identify the extent to which cross-cutting strategies/issues such as human rights-based approaches, results-based planning and gender equality/mainstreaming have been adopted in the planning and implementation of the programme.
V. To ascertain the requirements and implications (institutional capacity, financial implications etc.) of scaling up and implementing the ECHO programme on a national scale in the model countries, especially St. Vincent & the Grenadines.

Methodology:

A mixed method qualitative and quantitative approach was used to obtain data for project evaluation.  Qualitative approaches included review of project documents, in-depth interviews of senior administrative personnel and focus groups of stakeholders including CHAs, Rovers, the Administrative Committee, curriculum writers and trainers and district nursing supervisors.  The Quantitative approach involved collection of raw data from ECHO beneficiary parents, ECHO beneficiary children and age and gender matched controls.  Data collected included a parent beneficiary survey (questions on ECHO programme) and child development assessments of developmental status (ASQ), verbal comprehension, reading, writing and spelling, using standardised tests.  All ethical principles were followed, including confidentiality, informed consent and referral for children identified with developmental or behavioural concerns.
Major Limitations of Evaluation

Project evaluation was limited by a number of factors: delayed timing of evaluation with resultant difficulty locating documents and impaired memory recall; poor quality of documents and reports when located (undated, conflicting information); and limited parental participation as cases and age and sex matched controls.  The very few control families participating in the programme resulted in an inability to compare parenting knowledge, attitude and practice of ECHO and non-ECHO beneficiaries and child development and educational outcomes of ECHO and non-ECHO child beneficiaries.  In the absence of objective evaluation of programme outcome, parental interviews rated the programme highly.  However, home visits were less frequent and often shorter than as indicted by the protocol.

Findings and Conclusions:

Programme Achievements. In the absence of a comprehensive results based framework, analysis of available data suggested that the ECHO programme had only partially met its goal, while achieving the two objectives related to provision of early stimulation and training of staff to deliver stimulation.  The objectives of improving parental practice, promoting and monitoring child health and development and transferring knowledge to parents and community were assessed as partially achieved and that of promoting healthy parent-child relationships was unable to be assessed. The assessment was compromised by ineffective targeting of children at risk and uneven programme delivery. Additionally, there was not enough data to assess how the ECHO pilot affected children’s growth and development.

Programme Efficiency. recurrent cost was calculated at EC$2,487 (US$931.46), just over twice the cost of the RCP programme.  Programme costs were primarily due to human resources.  Analysis showed greater efficiency of the RCP model, but child development outcomes of the two programmes were not able to be assessed and compared as result of a lack of available uation data at all levels of the results chain.

Programme Strengths and Challenges. Identified ECHO programme strengths included a stable political environment; political support for the programme despite changing personnel; a cross-sectoral, inclusive monitoring body; engagement of external partners; adequate financial support provided primarily by donors; development and execution of a comprehensive training programme for CHAs, capacity building of CHAs and utilisation of opportunities for further capacity building in early stimulation by incorporating early stimulation in the training curricula of nurses.

Recommendations:

Recommendations for scaling up of the programme, based on this evaluation, included:
• Addressing programme design challenges, by including a human rights based approach
• Addressing sustainability through adequate financing
• Improving targeting of vulnerable children and families
• Developing a results based framework
• Addressing training and curriculum deficiencies by developing a home visiting curriculum, including assessment in the training programme and developing a comprehensive in-service training programme for nursing supervisors. 
• Maintaining and expanding the Administrative Committee to include other government partners and the private sector

Probably most important, however, is the need to address the human resource matter of determining the professional who will deliver the programme at the community level.  The RCP model is more cost and operationally efficient.  However, the training and engagement of CHAs has potential benefits for children.  A design that incorporates both the CHAs and the Rovers may be the best option.



Full report in PDF

PDF files require Acrobat Reader.


 

 

Report information

Year:
2014

Country:
Barbados

Region:
LACRO

Theme:
Early Childhood Development

Type:
Evaluation

Language:
English

New enhanced search