wifi: Nesta Guest password: flourish01 follow us on twitter @nesta_uk

1. Testing social policy innovation Wednesday 12th February 2014 wifi: Nesta Guest password: flourish01 follow us on twitter @nesta_uk official event hashtag #sparkEC

2. Testing social policy innovation Simon Flemington, Chief Executive Officer, LSE Enterprise wifi: Nesta Guest password: flourish01 follow us on twitter @nesta_uk official event hashtag #sparkEC

3. Testing social policy innovation Hélène Giacobino, Executive Director, J-PAL Europe wifi: Nesta Guest password: flourish01 follow us on twitter @nesta_uk official event hashtag #sparkEC

4. Improving Policies and Building Knowledge: the Role of Creative Experimentation Support services for social policy experimentation in the EU Hélène Giacobino Executive Director J-PAL Europe

5. The Need for Rigorous Evidence • In a context of economic downturn, it is important to optimize public expenditure • Promoting the implementation of rigorous evaluation methods to test the effects of new policies is a good way to achieve this goal • Evidence-based policy is possible and highly effective • As long as we learn from our success and mistakes, this is fine, because even a failed program helps us understand what went wrong • Without rigorous evaluation, everybody can favor their own pet project – and even lessons from successes are lost

6. Social Policy Experimentation • Social policy experimentation tests the validity of policies by collecting evidence about the real impact of interventions on people • The goal is: • to bring innovative answers to social needs, • to test impact on small scale interventions, • to scale-up the ones whose results were convincing

7. How to do SPE? • It is possible to undertake SPE through different methods: • Some of the most common non randomized methods are: • Pretest-posttest (Before and after) • Differences-in-Differences • Regression discontinuity • Statistical Matching • They always rely on strong assumptions • Randomized Controlled Trials (RCTs) give the most rigorous results (internal validity)

8. RCTs: a long history • Experimental psychology: late 19th century, • Education: early 20th century • Experimental sociology, early 20th century: • rural health education, • social effects of public housing, • recreation programs for delinquent boys • Large-scale randomized clinical trials: a norm since 1962 Drug Amendments • Went through substantial debates but today widely accepted • A boom in the 60’ in the USA, (250 RCTs done) • 90’: J-PAL introduce RCTs in development economics (today: 450 RCTs)

9. BUT … not every intervention can be evaluated • A well designed SPE should include: • an explicit and relevant policy question • a valid identification strategy • a well-powered sample • high quality data. • Evaluation is not appropriate when: • the sample size is too small • the impact to measure is a macro impact • the scaling-up of the pilot will modify the impact a lot • the beneficiaries are in a context of urgency

10. AND … not every intervention warrants an impact evaluation Investing in some key rigorous impact evaluations is essential to guide policy decisions and to responsibly allocate public resources Rigorous impact evaluations should be used to test key influential, strategically relevant or novel interventions No added-value to evaluate policies or programs that will benefit a very limited number of people, or to test a policy question that has been already rigorously evaluated in a similar context (except within explicit validation program)

11. Ethical Issues • SPE should be designed to follow the ethical principles applicable to evaluations and research projects involving human subjects (even if there are no rules in the country of experimentation) and be approved by an IRB. • SPE should include rigorous protection of individual data whichis paramount • SPE can have different designs to secures fairness, impartiality and transparency (and cost effectiveness!)

12. Financial Issues • Rigorous SPE need good quality data: this is why, if the sample is large and you don’t have existing data, it may be costly. • but financing large-scale interventions without knowing their effects can potentially result in a waste of resources! • The costs depend on the program evaluated: it is more expensive if you are dealing with long-term impact

13. The « Black Box » Issue • One common critique: “RCTs can provide whether an intervention was effective, and even measure the size of the impact. But they cannot answer the question of how or whythe impact came about.” • This is true only if final outcomes are measured. RCTs where intermediate indicators that map to the “theory of change” are collected, and where qualitative methods are also used, can help us understand the how and why.

14. Randomization is not a substitute for theory • Although randomization guarantees the internal validity of the estimate, in order to interpret the result, you need a theoretical framework: The extent to which findings do or don’t generalize beyond a specific context depend on theory: only theory tells you what is likely to matter in the context, and guide replications • Value of randomization is that you can more easily be surprised: • You cannot doubt the results when you found them: if they are surprising, instead of shelving the results, you (and others) have to think about what happened.

15. External Validity: an issue for all types of evaluation • External validity: the extent to which we can be confident that the results found in one context will generalize to other contexts • External validity is a function of the program being evaluated(where is it being implemented, how replicable is the program model, how much does program implementation depend on context or interaction with the community), not of the evaluation methodology! • Nonrandomized impact evaluations are undertaken on a specific program in a specific location, with the downside that they do not control for selection bias  weaker internal validity • A randomized evaluation tests a particular question in a specific location, at a specific time and at a specific scale. If properly conducted, and it has strong internal validity • If we cannot be confident that the evaluations measure the true impact of the program in a specific context, then we can be less confident in generalizing conclusions to another context

16. External validity: an issue for all types of evaluation • The scope of an evaluation’s true external validity depends on how the evaluation sample is determined: • if the evaluation sample is representative of the target population (randomly sampled from a larger population), then the results are generalizable to that population.

17. External validity: solutions Replication is a strategy that enables us to understand how an intervention functions in various settings SPE, as they force researchers to pay attention to context, details, and realities on the ground, allow to test broad theories in a credible way, and produce evidence that can then feed back into our theories Theory and process evaluation data can help us understand the mechanisms through which the impact was produced and therefore to scale-up programs more confidently

18. Conclusions • Social Policy Experimentation is an important tool to help improve social programs • It is important to strategically allocate evaluation resources to key influential, strategically relevant or novel interventions: • to facilitate scale-up • to encourage the replication of the policy in different contexts • to provide valuable information for future policymaking • Yet not every intervention can be evaluated • And not every intervention warrants an impact evaluation • Experimentation needs to be creative… • If we are just trying, and accept the possibility of failure, we do not need to think inside the box • This mindset could revolutionize social policy

19. Testing social policy innovation Phil Sooben, The Economic & Social Research Council Dr Simona Milio, LSE Enterprise Hélène Giacobino, J-PAL Europe Jonathan Breckon, Nesta Arnaud Vaganay, LSE Enterprise wifi: Nesta Guest password: flourish01 follow us on twitter @nesta_uk official event hashtag #sparkEC