Government Social Research Unit

1. Government Social Research Unit Randomised Controlled Trials Conference University of York, 14-15 September 2006 Why Governments Need RCTs Philip Davies PhD Government Social Research Unit HM Treasury London SW1A 2HQ www.gsr.gov.uk

2. Answer Because RCTs can help get policy initiatives right more often than wrong www.gsr.gov.uk

3. Caveats • Governments have got to want to get things right • ‘Getting things right’ is more than achieving an effect size • Governments do things other than achieving impacts • RCTs are not a panacea • RCTs are only as good as their design and implementation www.gsr.gov.uk

4. Governments Have Got to Want to Get Things Right “Social science should be at the heart of policy making. We need a revolution in the relationship between government and the social research community - we need social scientists to help determine what works and why, and what type of policy initiatives are likely to be most effective …”(David Blunkett, 2000) www.gsr.gov.uk

5. Governments Have Got to Want to Get Things Right Government policy must be: • Evidence-based • Properly evaluated • Based on best practice www.gsr.gov.uk

6. Governments Have Got to Want to Get Things Right • Policy making must be soundly based on evidence of what works • Improve departments’ capacity to make best use of evidence • Improve the accessibility of the evidence available to policy makers

7. Governments Have Got to Want to Get Things Right Professional Skills for Government (2005) Analysis and Use of Evidence • Anticipate and secure appropriate evidence • Test for deliverability of policy/practice – and evaluate • Use evidence to challenge decision making • Identify ways to improve policy/practice • Champion a variety of tools for collecting/using evidence • Ensure use of evidence is consistent with wider government requirements • Work in partnership with a wide range of experts/analysts

8. What ‘Getting It Right’ Means The ‘Experimenting Society’ (Donald T. Campbell) “…a society that would use social science methods and evaluation techniques to vigorously try out possible solutions to recurrent problems and would make hard-headed, multidimensional evaluations of outcomes, and when the evaluation of one reform showed it to have been ineffective or harmful, would move on and try other alternatives”

9. What ‘Getting It Right’ Means “A social experiment benefits society by providing better information on which to base public policy. Such information can improve policy in one of two ways: it can lead policymakers to adopt a program or policy that is found to have net social benefits, or it can lead to the termination of an existing program that is found to have a net social cost” (Orr, L., 1999).

10. Why RCTs? (Recap) Randomised Controlled Trials: • Provide most precise estimates of the likely effects of policy interventions. • Against a strong counterfactual – i.e. the net effect/impact • Reduce/account for various biases. • Establish cause and effect of policy interventions and outcomes. • Raise the quality of evidence to support policy making.

11. So, Why Don’t Governments Use RCTS More Often? • RCTs not on most policy makers’ radar • RCTs not on many government researchers’ radar • RCTS not on many academic researchers’ radar • For some (many?) RCTs are anathema: • Don’t like ‘random’ (too ‘chancey’) • Don’t like ‘control’ (no ‘wiggle room’) • Don’t like ‘trials’ (sounds inhumane / 1940s Germany) • There are things other than ‘impact’ • There are things other than ‘evidence’

12. Things Other Than Impact Evidence Experimental Quasi-Experimental Social Ethics Public Consultation Experimental Qualitative Theories of Change Impact Evidence Ethical Evidence Implementation Evidence Surveys Admin Data Comparative Qualitative Research Evidence Cost-Benefit Cost-Effectiveness Cost-Utility Economic and Econometric Evidence Descriptive Analytical Evidence Statistical Modelling Multivariate Analysis

13. Things Other Than Evidence “Research of any kind is seldom the determining factor in shaping public policy. Experimentation is no exception to this rule” (Orr, L., 1999)

14. Things Other Than Evidence Experience & Expertise Pragmatics & Contingencies Judgement Lobbyists & Pressure Groups Evidence Resources Values and Policy Context Habits & Tradition

15. Different Notions of Evidence Policy Makers’ Evidence Researchers’ Evidence • Colloquial (Contextual) • Anything that seems reasonable • Policy relevant • Timely • Clear Message • ‘Scientific’ (Context free) • Proven empirically • Theoretically driven • As long as it takes • Caveats and qualifications

16. Where Do Policy Makers Go For Evidence?

17. But Not All Is Lost Examples of Recent UK Government RCTs • Employment Demonstration Project (ERA) • Job Retraining and Rehabilitation Project (JRRP) • Restart evaluation (1990) • BA Visiting Officer Pilot • Evaluation of Employment Zones • New Deal 25+ Pilots • Evaluation of the New Deal for Young People Intensive Gateway • Evaluation of the Restorative Justice programme

18. RCTs Are Only As Good As Their Design and Implementation Common Design and Implementation Faults: • Unclear policy objectives, populations, outcomes • Insufficient time  (lack of strategic policy making) • Test condition often pre-determined • Sample sizes too small – no statistical power • Problems of randomisation, concealment, cross-over • High attrition (especially in control groups) • Intention to Treat analysis not used • Limited capacity (in the UK) to do high quality RCTs • Poor counterfactual too often the result

19. RCTs Are Only As Good As Their Design and Implementation These faults can usually be overcome! • Better strategic policy making • Better planning and protocol development • Involvement of researchers, policy makers, front-line staff • High quality piloting / pre-fieldwork testing • High quality research/project management • Careful real time data management • Use of experienced technical experts throughout • Policy Trials Service • Professional development of researchers and policy makers

20. Take Home Message • RCTs have enormous potential for governments • RCTs can help governments get things right more often • RCTs are currently used considerably to UK Govnt. • But there are still non-believers • And those who remain to be convinced • RCTs, like other types of research, must compete with other notions of evidence • And with factors other than evidence • There is a need to improve the capacity to do high quality RCTs in the UK

21. Contact philip.davies@hm-treasury.x.gsi.gov.uk Government Social Research Unit HM Treasury 1 Horse Guards Road London SW1A HQ England Tel: +44 (0)20 7270 5156 www.gsr.gov.uk