Evaluation and good practices

Programme evaluations

UNICEF Image
© UNICEF/INDA2005-02266/Khemka

Independence forms the bedrock of good evaluation practice as it is fundamental to the credibility and utility of an evaluation.  Each year the Evaluation Office commissions independent global evaluations based on the office’s Integrated Monitoring and Evaluation Framework (IMEP), which is developed in consultation with the Evaluation Committee and the Programme division and units at headquarters.  Corporate-Level Programmatic Evaluations analyse the contribution of UNICEF to global strategies. 

Evaluations Scheduled in the Integrated M&E Framework of the MTSP

Other Major Evaluations

Significant evaluations conducted by UNICEF Country Offices

WASH evaluation webpage 

==================================================================================================

Early Childhood Development

Two-Country Impact Evaluation on UNICEF’s Early Childhood Development Kit (ECD Kit)

Links to Evaluation Documents:

Questions posed by potential bidders; Answers supplied by UNICEF
[This section will be updated as questions are received.  Bidders are advised to check this space periodically.]

1. According to extensive literature reviews of ECD program in developing countries,[1] the median effect of size on cognitive skills of center-based preschool and day care programs is 0.3 standard deviations. Assuming 14 clusters per study arm, an intra-cluster correlation of 0.1, and 15 respondents per cluster (15 children per 3-, 4-, and 5-year age group), the proposed quasi-experimental study in Senegal would only have a 48 percent chance of detecting program effect of 0.3 standard deviations and a 25 percent chance of detecting a program effect of 0.2 standard deviations. Given this, is there room to increase the number of clusters (ECD centers) reached by each of the intervention arms?
Yes, bidders may propose a different number of clusters as long as there is a clear rationale for the proposed increase.

2. Could UNICEF please tell us if the Two Country Impact Evaluation (IE) for UNICEF’s Early Childhood Development Kit for Emergencies will be considered a “major thematic evaluation”?
No, it is not categorized as a major thematic evaluation.

3. Does UNICEF have a budget ceiling for this project?
It is the United Nations practice not to offer any guidance on budgets other than what is presented in the ToR and RFPS.

4. Will UNICEF help the contractor and facilitate access IDP camps in South Sudan, specifically in Malakal? Or will the contractor be responsible for gaining access?
Yes, UNICEF will facilitate access to all evaluation sites, including in Malakal. See p. 42, e. in the ToR for more information.

5. The footnote on page 8 of the solicitation states that ”the region for evaluation in South Sudan may change based on the security situation, which will be reviewed at the inception stage”.  Where else in South Sudan would UNICEF consider implementing this project if violence rules out performance in Upper Nile State?
The ECD kit programming is implemented on a national level in South Sudan. If required, the evaluation could potentially take place in any state within the country.

6. Could you please clarify the statement at page 16, 42h:  Is the support offered as guidance just a recommendation for local security contractors or is it more substantial support? More details would be appreciated to be able to both assess the security risk and for properly budgeting for contractors, if necessary.  In particular we would like to know if we could use UNICEF vehicles, drivers, and if the local UNICEF office security advisor is available for assisting us directly.
This RFPS is for institutional contracts; UNICEF rules and regulations regarding duty of care for institutions are outlined on page 16 of the RFPS, bullet #9. Contractors cannot expect to utilize UNICEF vehicles and drivers, and should budget for them in their financial proposal. The local UN security advisor can advise, but does not offer direct assistance in the sense of accompanying missions, arranging logistics, etc. 

7. Could it be clarified what “transport to Maban requires support from the World Food Program” (annex C, page 3) means?
There are no private air carriers to Maban; therefore, UNICEF will assist the evaluation team in obtaining transport to this evaluation site via the World Food Program. Round-trip airfare to Maban is estimated at $550 USD; modest lodging can be arranged by UNICEF; applicable DSA rate is $91 USD.

8. Would it be possible to share any draft report of the MICS5 report for S. Sudan?
No draft is currently available to share.

9. Do the intervention sites in Table 3 of the RFP represent the entire population of kids 3-5 receiving the ECD kits and sites in the two countries? If not, is it possible to have a more comprehensive list? Following up on the first question about statistical power in this session (http://www.unicef.org/evaluation/index_60825.html), we are wondering if including additional ECD sites could even be a possibility.  For example, we would like to know whether the program is also being implemented in Juba (South Sudan) and if conducting the study in Juba is a possibility.
No, Table 3 in the ToR represents the estimated number of intervention sites each CO has identified for inclusion in the evaluation based upon their planned programming and capacity to deliver. Table 2 on page 7 of the ToR and Annex A: ECD Kit Concept Note provide an overview of the planned ECD Kit programming for 2016-2017 in both countries. South Sudan distributes the ECD kits on a national level; while Senegal distributes the ECD kits on a regional level. National partners are currently validating data on the number of ECD settings receiving the ECD kits in both countries. A more comprehensive list of ECD kit beneficiaries will be made available during inception. Regarding the possibility of adding new ECD sites, please refer to the footnotes on page 9 in the ToR. Prior to the resurgence of violence in Juba, UNICEF was supporting one ECD centre and no CFS in this location. It is not currently known if Juba can be included in the evaluation. The selection of evaluation sites for South Sudan will be finalized during inception.

10. Footnote 14: Could it be clarified whether some UNICEF personnel could directly support the evaluation team to conduct data collection in South Sudan?  
No, UNICEF personnel (staff holding UNICEF contracts) cannot support data collection in either country. This is an independent impact evaluation, therefore UNICEF personnel are only responsible for developing and implementing the intervention. It would be a potential conflict of interest to include them as data enumerators. However, UNICEF can facilitate introductions to firms, NGOs, and government units that have conducted such exercises in the past.

11. If the bidding team finds it more appropriate to form a study team of more than 4 people, or have a secondary supporting study group, and this can be accounted for within the budget, is this acceptable to UNICEF?
On page 20, paragraph 52 of the ToR, it explains that ‘bidders reserve the right to vary the team size, with proper justification for allocation of work and cost.’

12. Would the entire study group be required to travel to both countries, or is there scope to allocate countries to country experts within the study team?
No, the entire study team is not required to travel to both countries. Information about the required evaluation team composition, profile and institutional profile are provided on page 14 in the ToR, paragraphs 35-39. Bidders should allocate level of effort for their team as they see fit to achieve the evaluation at a high quality standard.

13. To what extent, if at all, would security be provided to the research teams by UNICEF, or would research teams need to account for this in the budget?
See responses to question 6. Basically, security should be budgeted for in the financial proposal. 

14. Would UNICEF facilitate or support the application to authorise study teams to access UN camps in South Sudan?
Yes, UNICEF will facilitate access to all evaluation sites in South Sudan.

15. Can UNICEF provide information about the costs of similar evaluations?
UNICEF procurement rules prohibit disclosure of budget information to bidders.

16. Can UNICEF please confirm that there are no ECD centers or CFSs in South Sudan that could potentially serve as a comparison for the evaluation?
At this time there are no ECD centres or CFSs in South Sudan that can serve as a comparison for the evaluation. The situation will be re-assessed at inception, but bidders should anticipate 100% coverage of the ECD kits where UNICEF is responding to the conflict.

17. Does UNICEF require or expect in-country consultations between the evaluators, UNICEF, and project stakeholders during the inception phase? 
No, UNICEF does not require or expect in-country consultations during inception. However, UNICEF expects bidders to submit proposals that include an approach for consultations with relevant stakeholders throughout the duration of the evaluation process. During the inception period, that must include consulting with national stakeholders, even if not done in-country.

18. In light of current security situation in South Sudan (violent crimes, ongoing fighting, and foreigners being targets of extreme violence), we would like to know whether UNICEF will consider changing the evaluation to a country alternative to South Sudan now - at the proposal stage. In footnote 3 on page 3 of the RFPS, we understand that UNICEF may consider changing the study to a different country during the inception phase and after the security situation is reviewed in South Sudan. However, given the current information about the situation in South Sudan, it is challenging to propose a cost-effective study because of the high costs associated with the establishment of a strong security platform necessary to protect research team members. 
No, the Evaluation Office is committed to re-assessing the situation during inception to determine the feasibility of the evaluation in the country, not before. UJNICEF is working right now in South Sudan to address the needs of the most vulnerable children living in conflict, including through the use of the ECD kits. While the situation is very fluid, the South Sudan Country Office has already invested a substantial amount of time preparing for this evaluation and senior leadership has expressed a desire to continue exploring the implementation of this work regardless of the current situation.
We are aware of the violence that has taken place in the country against citizens and foreigners and we are monitoring the situation very closely. However, the ECD kit was originally designed for humanitarian settings, and regardless if this evaluation is completed in South Sudan, the evaluation will take place within a context which involves risks that should be properly planned for and taken into consideration. Thus, arrangements for the security and protection of research team members should be adequately budgeted for in financial proposals.
Each bidder is in the same situation regarding the cost-effectiveness of the study. As noted on the footnote on page 22 of the ToR, ‘All costs will be fixed, except for travel to selected destinations, which will be on a cost-reimbursable basis.’ Thus, security will be reimbursed based upon the actual costs submitted to UNICEF.


 

 

TBD

New enhanced search