Some Questions about RCTs

1. Some Questions about RCTs Martyn Hammersley The Open University .

2. The Context: The promotion of evidence-based practice, and the demands made on research to supply the sort of evidence that is thought to be needed for this.

3. Over-claiming for RCTs • It is frequently claimed that RCTs are the best means of determining ‘what works’. • However, though RCTs have strengths, they also have serious weaknesses and limitations. • While these are sometimes acknowledged, they are frequently underplayed by RCT enthusiasts.

4. A First Question 1. Are there any requirements for the use of RCTs that often cannot be met in the context of social policy and practice? Answer: Yes.

5. 2. Can the findings of RCTs be externally valid? Answer: Yes, but it is hard, in any particular case, to know whether they are, and even harder to ensure that they are. Furthermore, achieving this will always be at the expense of internal validity.

6. 3. Can RCTs be internally valid? Answer: Yes, but this is much harder to achieve than is often suggested.

7. 4. Can RCTs overcome the serious measurement problems that plague social science? Answer: There is nothing distinctive about RCTs in this respect - they face the problems that all social research does. They also involve the common danger of a preoccupation with easily measurable outcomes.

8. A final question: 5. Can RCTs tell us what works in policy terms? Answer: No. Research alone, of any kind, cannot do this.

9. Conclusion To reiterate: like all research methods, RCTs have strengths but also limitations and weaknesses. They are no ‘gold standard’. They cannot tell policymakers ‘what works’, definitively; both because their conclusions are subject to serious threats to validity, and because no research can do this on its own.

10. I am not suggesting that these problems render RCTs useless; even less that other methods overcome these problems, or do not face other serious ones themselves. My point, rather, is that social scientists must not make excessive claims about their capacities. I believe that this has been particularly true of some advocates of RCTs.

11. Serious Dangers arising from RCT Enthusiasm For research: 1. Over-privileges RCTs, even as a method of understanding the effects of policy interventions. 2. Leads to an excessive emphasis on applied research of a particular kind.

12. Dangers for Policymaking and Practice Encourages an exaggerated conception of what research can do, amounting to a form of scientism Researchers may become complicit in a pathological form of policymaking, according to which improvements across the board can be generated by some policy lever which overrides professional knowledge and skill.