Workshop: Evaluations as a policy instrument for sustainable development How can we improve the learning process in

1. Stockholm October 18, Johan Harvard Workshop: Evaluations as a policy instrument for sustainable development How can we improve the learning process in political decision-making?

2. Workshop program

3. Stockholm October 18, Johan Harvard Presentation: The Power of Knowing: Why Improved Evaluation and Learning are Needed to Realise the Green Future

4. Summary • Lots of policy movement in the SD/GG-field, in developedcountries • Complexchallenges – complex policy • Policy is not sufficiently/appropriatelyevaluated • Lessons not learned – inefficient policy and wasted tax money • However– SD/GG policy verycomplextoevaluate • Thereby – SD/GG policy evalutionsdifficulttouse for policymakers

5. Key terminology • Evaluation vs Monitoring vsBenchmarking • Evaluating process/implementation, results, impact • Factual vsCounterfactual: Need to assess additionality

6. Green policy is pickingup speed • European Union’s Growth Strategy for 2020 • Korea’s National Strategy and Five-Year-Plan for green growth • Green development focus of China’s Five-Year-Plan • South Africa’s New Growth Path and Green Economy Accord

7. Status Policy Process: Crucial Elements Flawed • Needs highlighted • Wish to stimulate sustainable growth: push for action • Analysis preceding policy action • Ex-ante analysis • Policy Action • Decisions are made • Implementation • Execution of policy • Learning after/during implementation • Lessons learned • New policy cycle

8. What is beingdonetoday? • Local/regional level • Yes: Somemonitoring/evaluationof implementation and results • No: Impactevaluationsarevery rare • National level • Yes: Benchmarking, monitoringmacroindicators • No: Impactevaluations, system evaluationslinking policy tooutcomes – very rare

9. Consequence: Suboptimal policy • If you do not measure results, you cannot tell success from failure. • If you cannot see success, you cannot reward it. • If you cannot reward success, you are probably rewarding failure. • If you cannot see success, you cannot learn from it. • If you cannot recognize failure, you cannot correct it. • If you can demonstrate results, you can win public support Source: Adapted from Osborne & Gaebler 1992.

10. But… difficult problem toaddress • SD/GG policy is inherently complex • Policy on all levels simultaneously (local/regional/national/ international) • Very different types of policies implemented simultaneously (supply/demand, skills/capital/incentives/market) • External factors very important (economic turmoil, global market demand, competition) • Conflicting objectives (economic growth vs sustainability) • Multiple policy areas – many different actors in play • Interaction effects

12. Simple – Complicated - Complex • Simple problems - Like baking a cake. A few basic techniques but once mastered likelihood of success is high • Complicated problems - Like sending a rocket to the moon. No straightforward recipe often require multiple people and teams. Unanticipated difficulties are frequent. Timing and coordination become serious concerns • Complex problems - Like raising a child. Once you learn how to send a rocket to the moon. You can repeat and improve the process. One rocket is like another rocket. But raising one child is not like raising another. Each is unique. Once child may require a completely difference approach to the previous one. This brings up another feature of highly complex problems. Their outcomes remain highly uncertain. Source: Rogers (2008)

13. Evaluating SD/GG policy • Qualitativeevaluation • Contributionanalysis • Theory-based approach • etc • Quantitativeevaluation • Comparisongroups (deal withselection bias) • Quasi-experimental methods (PSM, Discontinuityetc) • Difference-in-difference (ifunobservablesaretime invariant) • There is no one solution  need combinations – mixed methods

14. Usefulevaluation – evaluationutilization • Currently – partial evaluationsarecarriedout – leading toinabilitytodrawclearconclusions • Difficulttousetheseevaluations Patton: • ”Evaluationsshould be judged by theirutility and actualuse; therefore, evaluatorsshouldfacilitate the evaluation process and design anyevaluationwithcarefulconsiderationofhoweverthing is done, willaffectuse” • Focus on intendeduse by intendedusers • Mutual responsibilityofusers and evaluators

15. Sweden’s Cleantech Industry Development Strategy - Evaluation • Encompassing 4x100 Million SEK over 2011-2014, currently 18 programs launchedtohelp business grow • Include supply and demand side policies: export advisory services, innovative public procurement, cleantech incubation, inward FDI, commercialization & export of public sector expertise, establishing business-research development consortia’s, matching of investors-investees, etc. • Evaluation: • Process, Results, Impacts • Top-down + Bottom-up • Quantitative + Qualitative • Utilization: Hoping for process use as well as instrumental use

16. Conclusions and Recommendations • Needof systematic improvementofevaluations – moreimpactevaluations • Authoritiesshould not underestimate the complexity and importanceofimpactevaluations • Needbetterincentives? Will costmoreinitially! • Evaluatorsshouldimprovetheiruseof best methods for impactevaluation – includinguseofquantitativemethods

17. Questions • Weneedmore and betterevaluations on all levels – howcanthis be achieved? • Incentives for impactevaluations? • Howcanwe make sure evaluationsincludeeconomic and environmental aspects – not either or? • Howcan the procurementofevaluationsimprove, tofacilitatebetterevaluations and improveevaluationuse? • Complexevaluations - difficulttouse for policymakers: howcanthis be overcome? Capacity for use? • Could/should policy be designed differently, tofacilitateevaluation? • …if a changed design wouldcompromisequality?