Perspectives from the Independent Evaluation Group Martha Ainsworth and Soniya Carvalho

1. Preparing High-Quality Implementation Completion and Results Reports Perspectives from the Independent Evaluation Group Martha Ainsworth and SoniyaCarvalho

2. Part I: Tips for preparing a high-quality ICR Martha Ainsworth Part II: Project ratings: connects and disconnects SoniyaCarvalho

3. Why is ICR quality important? • The ICR is an opportunity to learn and to improve effectiveness • It’s the official (public) report on the project, its achievements and lessons • Low quality ICRs make it difficult for IEG to rate the project accurately • Unsatisfactory ICR quality is a good predictor of ratings “disconnects” and disputes • From 7/1/2006 onward, “when insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant rating”

4. What are the IEG criteria for ICR quality? • Results-orientation (ICR should be outcome-oriented, not an implementation narrative) • Quality of evidence and analysis • Lessons based on evidence and analysis • Internal consistency • Consistency with Bank guidelines • Conciseness

5. 1. Make the ICR results-driven • Organize evidence around achievement of objectives (not implementation of components) • Document the “results chain”: Show the link between inputs, outputs, outcomes, impacts • Explain the counterfactual (what would have happened without the project), other factors operating, plausible attribution • Don’t be constrained by the official project indicators • Show trends over the whole period, before and after the project, as many observations as possible • Footnote sources of evidence, triangulate data.  Provide the evidence necessary for someone disconnected from the project to be able to rate it!

6. The Results Chain Inputs  Outputs  Outcomes  Impacts • To assess efficacy using the results chain, you need: • to be able to link these elements plausibly • to understand the “counterfactual”— what would • have happened without these activities

7. Establishing a “counterfactual” • Timeline of events • Account for trends in other determinants (For example, other donor support, weather, changes in national policies, household income, changes in prices that could affect demand or incomes) • Document trends before and after, in project and non-project areas

8. Highest ranking Distribution of Malaria in Eritrea Northern Red Sea Lowest ranking Anseba Maekel Gash barka Debub Southern Red Sea Example #1: Eritrea HIV/AIDS, Malaria, STI, and TB Control Project

9. Annual outputs of anti-malaria interventions increased Nyarango et al. (2006) (incidence and rainfall data) National Malaria Control Program (2004, 2006) (intervention data)

10. New cases of malaria declined Nyarango et al. (2006) (incidence and rainfall data) National Malaria Control Program (2004, 2006) (intervention data)

11. Rainfall also declined, but malaria continued to decline when rainfall recovered Nyarango et al. (2006) (incidence and rainfall data) National Malaria Control Program (2004, 2006) (intervention data)

12. Example #2:Russia Health Reform Pilot Project Russia’s abortion rate declined during the project

13. It declined even faster before the project

14. It declined all over Russia, without the projectand there were no data for the results chain linking project activities to the decline in abortion rate

15. Example #3: Romania Roads 2 Project /road safetyRoad fatalities declined in Romania

16. They declined faster before the Traffic Safety Campaignand there were no intermediate outcome data on safety

17. 2. Show all indicators of efficiency • ICRs for allinvestment-type projects (including TA projects) must assess efficiency, not ICRs for DPLs • Answers the question: Were costs in achieving the objectives reasonable in relation to the benefits and to recognized norms (“value for money”) • Show evidence of: • Cost-benefit, cost-effectiveness, efficient use of project resources, unit rate norms, service standards, least-cost analysis and comparisons,other efficiency indicators • Aspects of design and implementation that contributed to or reduced efficiency • Cost-benefit analysis is not sufficient to assess efficiency; other indicators must be shown • If presented, transparent discussion of assumptions

18. 3. Document safeguards • Safeguard policies not applicable to DPLs • Applicable safeguard policy, project category (A, B, C, FI), assessment/mitigation plan • Whether the mitigation plan was implemented • Findings of third party safeguard reviews • Updated assessment and revised mitigation plans implemented if physical components generating economic and social impacts were modified.

19. 4. Other quick tips • Consistency: Make sure the ratings match the text and the numbers are internally consistent • Conciseness: More evidence is good, as long as it is relevant and concise; if extensive, add an annex. • Completeness: Ensure that the sections and annexes are complete and accurate • Candor: Be candid about shortcomings, data quality • Lessons: Not too many; must be evidence-based and come from the project’s experience; seek to explain variability in outcomes GOOD LUCK!