Evaluation and good practices

Programme evaluations

UNICEF Image
© UNICEF/INDA2005-02266/Khemka

Independence forms the bedrock of good evaluation practice as it is fundamental to the credibility and utility of an evaluation.  Each year the Evaluation Office commissions independent global evaluations based on the office’s Integrated Monitoring and Evaluation Framework (IMEP), which is developed in consultation with the Evaluation Committee and the Programme division and units at headquarters.  Corporate-Level Programmatic Evaluations analyse the contribution of UNICEF to global strategies. 

Evaluations Scheduled in the Integrated M&E Framework of the MTSP

Other Major Evaluations

Significant evaluations conducted by UNICEF Country Offices

WASH evaluation webpage 

==================================================================================================

Early Childhood Development

Two-Country Impact Evaluation on UNICEF’s Early Childhood Development Kit (ECD Kit)

Links to Evaluation Documents:

Questions posed by potential bidders; Answers supplied by UNICEF
[This section will be updated as questions are received.  Bidders are advised to check this space periodically.]

1. According to extensive literature reviews of ECD program in developing countries,[1] the median effect of size on cognitive skills of center-based preschool and day care programs is 0.3 standard deviations. Assuming 14 clusters per study arm, an intra-cluster correlation of 0.1, and 15 respondents per cluster (15 children per 3-, 4-, and 5-year age group), the proposed quasi-experimental study in Senegal would only have a 48 percent chance of detecting program effect of 0.3 standard deviations and a 25 percent chance of detecting a program effect of 0.2 standard deviations. Given this, is there room to increase the number of clusters (ECD centers) reached by each of the intervention arms?
Yes, bidders may propose a different number of clusters as long as there is a clear rationale for the proposed increase.

2. Could UNICEF please tell us if the Two Country Impact Evaluation (IE) for UNICEF’s Early Childhood Development Kit for Emergencies will be considered a “major thematic evaluation”?
No, it is not categorized as a major thematic evaluation.

3. Does UNICEF have a budget ceiling for this project?
It is the United Nations practice not to offer any guidance on budgets other than what is presented in the ToR and RFPS.

4. Will UNICEF help the contractor and facilitate access IDP camps in South Sudan, specifically in Malakal? Or will the contractor be responsible for gaining access?
Yes, UNICEF will facilitate access to all evaluation sites, including in Malakal. See p. 42, e. in the ToR for more information.

5. The footnote on page 8 of the solicitation states that ”the region for evaluation in South Sudan may change based on the security situation, which will be reviewed at the inception stage”.  Where else in South Sudan would UNICEF consider implementing this project if violence rules out performance in Upper Nile State?
The ECD kit programming is implemented on a national level in South Sudan. If required, the evaluation could potentially take place in any state within the country.

6. Could you please clarify the statement at page 16, 42h:  Is the support offered as guidance just a recommendation for local security contractors or is it more substantial support? More details would be appreciated to be able to both assess the security risk and for properly budgeting for contractors, if necessary.  In particular we would like to know if we could use UNICEF vehicles, drivers, and if the local UNICEF office security advisor is available for assisting us directly.
This RFPS is for institutional contracts; UNICEF rules and regulations regarding duty of care for institutions are outlined on page 16 of the RFPS, bullet #9. Contractors cannot expect to utilize UNICEF vehicles and drivers, and should budget for them in their financial proposal. The local UN security advisor can advise, but does not offer direct assistance in the sense of accompanying missions, arranging logistics, etc. 

7. Could it be clarified what “transport to Maban requires support from the World Food Program” (annex C, page 3) means?
There are no private air carriers to Maban; therefore, UNICEF will assist the evaluation team in obtaining transport to this evaluation site via the World Food Program. Round-trip airfare to Maban is estimated at $550 USD; modest lodging can be arranged by UNICEF; applicable DSA rate is $91 USD.

8. Would it be possible to share any draft report of the MICS5 report for S. Sudan?
No draft is currently available to share.

9. Do the intervention sites in Table 3 of the RFP represent the entire population of kids 3-5 receiving the ECD kits and sites in the two countries? If not, is it possible to have a more comprehensive list? Following up on the first question about statistical power in this session (http://www.unicef.org/evaluation/index_60825.html), we are wondering if including additional ECD sites could even be a possibility.  For example, we would like to know whether the program is also being implemented in Juba (South Sudan) and if conducting the study in Juba is a possibility.
No, Table 3 in the ToR represents the estimated number of intervention sites each CO has identified for inclusion in the evaluation based upon their planned programming and capacity to deliver. Table 2 on page 7 of the ToR and Annex A: ECD Kit Concept Note provide an overview of the planned ECD Kit programming for 2016-2017 in both countries. South Sudan distributes the ECD kits on a national level; while Senegal distributes the ECD kits on a regional level. National partners are currently validating data on the number of ECD settings receiving the ECD kits in both countries. A more comprehensive list of ECD kit beneficiaries will be made available during inception. Regarding the possibility of adding new ECD sites, please refer to the footnotes on page 9 in the ToR. Prior to the resurgence of violence in Juba, UNICEF was supporting one ECD centre and no CFS in this location. It is not currently known if Juba can be included in the evaluation. The selection of evaluation sites for South Sudan will be finalized during inception.

10. Footnote 14: Could it be clarified whether some UNICEF personnel could directly support the evaluation team to conduct data collection in South Sudan?  
No, UNICEF personnel (staff holding UNICEF contracts) cannot support data collection in either country. This is an independent impact evaluation, therefore UNICEF personnel are only responsible for developing and implementing the intervention. It would be a potential conflict of interest to include them as data enumerators. However, UNICEF can facilitate introductions to firms, NGOs, and government units that have conducted such exercises in the past.

11. If the bidding team finds it more appropriate to form a study team of more than 4 people, or have a secondary supporting study group, and this can be accounted for within the budget, is this acceptable to UNICEF?
On page 20, paragraph 52 of the ToR, it explains that ‘bidders reserve the right to vary the team size, with proper justification for allocation of work and cost.’

12. Would the entire study group be required to travel to both countries, or is there scope to allocate countries to country experts within the study team?
No, the entire study team is not required to travel to both countries. Information about the required evaluation team composition, profile and institutional profile are provided on page 14 in the ToR, paragraphs 35-39. Bidders should allocate level of effort for their team as they see fit to achieve the evaluation at a high quality standard.

13. To what extent, if at all, would security be provided to the research teams by UNICEF, or would research teams need to account for this in the budget?
See responses to question 6. Basically, security should be budgeted for in the financial proposal. 

14. Would UNICEF facilitate or support the application to authorise study teams to access UN camps in South Sudan?
Yes, UNICEF will facilitate access to all evaluation sites in South Sudan.

15. Can UNICEF provide information about the costs of similar evaluations?
UNICEF procurement rules prohibit disclosure of budget information to bidders.

16. Can UNICEF please confirm that there are no ECD centers or CFSs in South Sudan that could potentially serve as a comparison for the evaluation?
At this time there are no ECD centres or CFSs in South Sudan that can serve as a comparison for the evaluation. The situation will be re-assessed at inception, but bidders should anticipate 100% coverage of the ECD kits where UNICEF is responding to the conflict.

17. Does UNICEF require or expect in-country consultations between the evaluators, UNICEF, and project stakeholders during the inception phase? 
No, UNICEF does not require or expect in-country consultations during inception. However, UNICEF expects bidders to submit proposals that include an approach for consultations with relevant stakeholders throughout the duration of the evaluation process. During the inception period, that must include consulting with national stakeholders, even if not done in-country.

 


 

 

TBD

New enhanced search