The role of RCTs in Government: The experience of DWP

1. The role of RCTs in Government: The experience of DWP Jane Hall Head of Analysis Family Poverty and Work Division Department for Work and Pensions jane.hall@dwp.gsi.gov.uk

2. Overview • Current climate/appetite for evidence • DWP’s use of RCTs • RCTs as the ‘gold standard’ in evaluation • Their applicability to developing and informing social policy

3. Current climate • Relatively supportive of RCTs • Strong emphasis on evidence-based policy making, however…. • ‘There is nothing a government hates more than to be well-informed; it makes the process of arriving at decisions much more complicated and difficult’ • John Maynard Keynes

4. Chronology of RCTs • Many in DWP • Varied in size and scale • Have become increasingly sophisticated • Restart • Various New Deals • JRRP • ERA

5. Are RCTs the gold standard in evaluation? • Attempt to address the issue of causality • Do this by addressing the counterfactual • i.e. what would have happened in the absence of the intervention • Any differences in outcome can confidently be ascribed to the intervention • Analysis relatively straightforward to perform • Results lend themselves to easy interpretation by non-technical audiences

6. … or are they just gold plated? • Issues to be aware of: • Crossovers at the point of allocation: can be non-random due to subversion/sabotage • Attrition: This may be non-random • Contamination: The control group gain access to services designed for the programme group • External validity, or the degree to which, ‘a causal relationship holds over variations in persons, settings, treatments and outcomes’ (Shadish, Cook and Campbell, 2002) • ‘Hawthorne effect’ • Compensated rivalry – control group members know their status and compensate • Performance bias – Programme administrators compensate • Resentful demoralisation – discouraged control group

7. Applicability to social policy • Need to remember the purpose of policy analysis • Government interventions are predicated on some notion of causation, i.e. doing ‘x’ leads to ‘y’, for example • Wage supplements (WTC) increase employment • Speed cameras reduce traffic accidents • Implementation: complication and resistance • Cost • Timeliness of results: Now not later! • Experiments do not explain causes • Very often this is what is required by policy makers • Senior decision makers, and ministers want causal descriptions in order to determine how to allocate scarce resources

8. On balance, are RCT’s worth it? • Yes, but… • Experiments have to be well-designed • They need to be integrated into the design of the policy • Expectations need to be managed • Experiments will show whether a causal effect is at work and will describe the size and significance of the effect, so • They need to be under-pinned by process studies to explain the causal effects • Interventions that are not evidence based are harder to evaluate • Most evaluations assume a rational model • But most policies are the outcome of a bargaining process • Finally: They are not a substitute for a poorly designed policy!