Impact Evaluation in the Global Environment Facility: Looking back for Our Common Future

1. Impact Evaluation in the Global Environment Facility: Looking back for Our Common Future Cairo, April 2009 David M. Todd dtodd@thegef.org Lee Alexander Risby lrisby@thegef.org

2. Global Environment Facility (GEF): The 10 second Overview • Established in1991 • Funds projects undertaken by developing and transition countries to achieve environmental benefits • Funding areas: Climate change; biodiversity; international waters; ozone; persistent organic pollutants and land degradation • Over 10US$ billion in project financing since 1991 • GEF Structure: Council (Governance & Strategy); Secretariat (Project approvals & Monitoring); Evaluation Office (Independent Evaluation) • Projects implemented by MDBs and UN agencies

3. Impact Evaluation (IE) in GEF: Overview • IE launched in 2005: Demand from GEF Council (donors and recipients) • Set against background of complexity of the environment and complexity of projects • Designs employed so far: • Straightforward before/after analysis with comparison between projects (Ozone) • With/without + before/after analysis through quasi-experimental design (Biodiversity) • Theory-of-Change (TOC) (Biodiversity and Ozone) • Combination of before / after and TOC (Ozone) • “mixed design / methods” approach where possible should be pursued:

4. Ozone Depleting Substances Impact Evaluation in Countries with Economies in Transition Building and Implementing Mixed-Design and Method Evaluation

5. Ozone Depleting Substances IE in Countries with Economies in Transition (CEITs): • GEF began ODS activities in CEITs in 1992 to phase-out CFCs and other chemicals that contribute to Ozone damage (The “Ozone Hole”) • Supports Montreal Protocol compliance • 26 projects (18 countries) and approximately US$380m of financing. • Projects aimed at stopping consumption and production of Ozone Depleting Chemicals and technologies • Key stakeholders: Governments, Implementing Agencies and private sector • Main issues for the evaluation: • Impact of the GEF on production and consumption of ODS • Impact on legal/illegal trade ODS • Impact on Stockpiling of ODS • Sustainability and catalytic effects • Attribution / contribution issue • Scope: Portfolio impact – why and how? Project and data synergy

6. ODS IE: Tailoring Design to Context • Data availability assessment: • Quantitative data on production and consumption exists for CEITs early 1990s – to present. • Significant quant-qual literature on ODS phase-out, many evaluations but no impact evaluations • Academic and technical literature on ozone science and environmental / human health impacts • Availability of scientific modeling of ozone hole recovery • And also on human and animal health impacts (skin cancers) • Initial Evaluation Design assessment • ODS portfolio would provide an opportunities for mix design and method approaches: • Quasi-experimental with a TOC – responding to what worked and why …

7. ODS IE: Tailoring Design to Context • But, desk review found no viable control / counter-factual countries exist • All other countries developed and developing have taken actions to stop ODS consumption / production • Only exceptions in 2008: Vatican, San Marino and Timor • Adjusted design to reflect observed data and sample limitations: • Before & after measures of changes: • By chemical • Time • Internal comparison between GEF Bank and UN countries • External matched comparison between GEF and Multi-lateral Fund countries • + Business as usual forecast: what would have happened without the MP and the GEF project? • TOC: development of TOC based on review of project documents and available terminal evaluations

8. Impact Evaluation ODS Theory of Change Assumption Availability of new technologies Threat Reduction: Illegal trade in ODS Output: National commitment to eliminate use (production & consumption) Assumption: Recovery and recycling schemes or stockpiling meet temporary demand for ODS Assumption: New Technologies are affordable Impact:ODS tons eliminated Strategy:To protect human health and the environment by assisting countries to phase-out ODS consumption and production, while enabling alternative technologies and practices Intermediate:ODS substances phased-out Output: Coordinated Regional Policies Outcome: Phased-out supply and demand for ODS technology Impact: Ozone layer improved Output:Technology producers and distributers supported to convert Intermediate:ODS technology phased-out Assumption:Enterprises are reached (large, medium and small) Threat Reduction: Reversion to ODS technologies Assumption: Effective monitoring and reporting on ODS Output:Users supported to adopted sustainable technology Assumption: Enterprises investing are sustainable

9. Methodology • TOC provides input into ODS IE evaluation matrix: sub-questions / data sources / methods and indictors • Pre-fieldwork meta-evaluation of existing terminal evaluations • Including means-ends linkage assessment (e.g., economic and trade instruments – and ends – market transformation) • Statistical methods: • Descriptive and Regression-based analyses • Taking into account differences in time, population, GDP etc • Fieldwork (qualitative method) used to confirm and / or dis-confirm TOC, with particular focus on ‘assumptions / risks and threats’ • 4 detailed country case studies • 8 ‘lite’ country case studies: Drawing on parallel UNDP – UNEP terminal evaluation of GEF-ODS projects • Semi-structured interviews with Government Ozone Units (20%) Private Sector (60%), Customs and Excise (20%) • Questionnaire survey to validate key issues arising from qualitative data collection • Triangulation between and within countries

10. Forthcoming findings • Draft report released mid-May • International ODS impact evaluation workshop in Tashkent: June 2009 • Now over to David to finish with a delicious ROTI

11. Review of Outcomes to Impacts (ROtI)

12. Impact Evaluation Framework Assess how these effects are leading to impacts Assess whether impacts have actually occurred Assess direct effects of the project ++ + +++

13. Project stakeholders create their own theory of change

14. How the ROtI addresses a key methodological dilemma in impact evaluation: representation versus reliability Represent-ation: Number of projects per $100,000 Desk–based Rotis based on Terminal Evaluations ($400) 250 Field-based ROtIs ($10-20K) 5-10 Impact Evaluations ($100K) 1 Reliability: Data Quality

15. Steps in field ROtI

16. Results from Field Review Pilot in Seychelles showed that ROtI based on field review can produce detailed qualitative assessment (with some quantitative support) of achievement of outcomes and of progress from this stage towards achievement of impacts. Key questions could all be assessed : did assumptions hold, were intended impact drivers active and were intermediate stages achieved.

17. The role of the ROtI • The desk ROtI enables a rapid scaling up of proxy information on impacts • The field ROtI provides moderate scaling up possibilities, with higher data quality • The Impact Evaluation provides detailed and reliable data on a few projects • The three levels together can provide a much richer knowledge base on impacts than is currently available

18. Thank you