Sources of bias in RCTs

1. Sources of bias in RCTs With focus on blinding and concealment of allocation David Torgerson

2. Background • Most reported educational (and health) trials are of POOR quality. • They are too small and do not adequately report their methods.

3. All RCTs are NOT the same. • Although the RCT is rightly regarded as the premier research method, by those who know, some trials are better than others.

4. Selection Bias - A reminder • Selection bias is one of the main threats to the internal validity of an experiment. • Selection bias occurs when participants are SELECTED for an intervention on the basis of a variable that is associated with outcome. • Randomisation or other similar methods abolishes selection bias.

5. After Randomisation • Once we have randomised participants we eliminate selection bias but the validity of the experiment can be threatened by other forms of bias, which we must guard against.

6. REMEMBER • Whilst most RCTs are susceptible to a range of potential biases other study types (e.g., pre test post test, quasi experiments) are also susceptible to the same biases.

7. Blinding • The ‘gold-standard’ in health care trials is supposedly the double blind placebo controlled trial. • It is helpful to blind some people some of the time but may not be always appropriate.

8. What does blinding achieve? • If a participant, their doctor (teacher) and the researcher are unaware of treatment received. This prevents a number of potential biases including ascertainment and performance.

9. Ascertainment Bias • This occurs when the person reporting the outcome can be biased. • A particular problem when outcomes are not ‘objective’ and there is uncertainty as to whether an event has occurred. • For example, if the person marking an essay knows into what group the person belongs this might affect their mark.

10. Example. • A group of student’s essays were randomly assigned photographs purporting to be the student. The photos were of people judged to be “attractive” “average” “below average”. The average mark was significantly HIGHER for the average looking student. • Why? Markers were biased into marking higher for students whom they believed were average looking (like themselves).

11. Performance Bias • Blinding, by sham or placebo, can reduce this bias BUT is it a bias? • This is where the student because they are in a group they like ‘performs’ better than in a group they don’t like. It could be argued that if they like the intervention and perform better this is part of the intervention. Conversely, if the intervention is ineffective despite being liked better (e.g., computers) then it really is ineffective.

12. Preference effects • When students (or patients) prefer an intervention this may bias an outcome. One way of attempting to control for this bias is to ask before randomisation the student’s preference and use this in the analysis.

13. Subversion Bias • Subversion Bias occurs when a researcher or clinician manipulates participant recruitment such that groups formed at baseline are NOT equivalent. • Anecdotal, or qualitative evidence (I.e gossip), suggest that this is a widespread phenomenon. • Statistically this has been demonstrated as having occurred widely.

14. Subversion - qualitative evidence • Schulz has described, anecdotally, a number of incidents of researchers subverting allocation by looking at sealed envelopes through x-ray lights. • Researchers have confessed to breaking open filing cabinets to obtain the randomisation code. Schulz JAMA 1995;274:1456.

15. Subversion in a non health trial • Boruch describes a trial in USA where incidents of domestic violence were being randomised to a ‘caution’ or being seen in the police station. • Some evidence that police officers made ‘sure’ that offenders they knew got taken to the police station. • This WILL damage the trial.

16. Subversion and Education • I don’t know of an educational trial that has been ‘subverted’ BUT they almost certainly exist. • Educational researchers are no different from health researchers and are just as likely to subvert their trials as we do!

17. Quantitative Evidence • Trials with adequate concealed allocation show different effect sizes, which would not happen if allocation wasn’t being subverted. • Trials using simple randomisation are too equivalent for it to have occurred by chance. • Often educational trials used ‘paired’ randomisation but have UNEQUAL numbers in each group – which means they lost one some where!

18. Poor concealment • Schulz et al. Examined 250 RCTs and classified them into having adequate concealment (where subversion was difficult), unclear, or inadequate where subversion was able to take place. • They found that badly concealed allocation led to increased effect sizes – showing CHEATING by researchers.

19. Comparison of concealment Schulz et al. JAMA 1995;273:408.

20. Recent Health Trials. Hewitt et al. 2004; submitted

21. Case Study • Subversion is rarely reported for individual studies. • One study where it has been reported was for a large, multicentred surgical trial. • Participants were being randomised to 5+ centres using sealed envelopes.

22. Case-study (cont) • After several hundred participants had been allocated the study statistician noticed that there was an imbalance in age. • This age imbalance was occurring in 3 out of the 5 centres. • Independently 3 clinical researchers were subverting the allocation

23. Mean ages of groups

24. Example of Subversion

25. Using Telephone Allocation

26. Subversion - summary • Appears to be widespread in health trials almost certainly occurs in psychology trials but remains undetected. • Secure allocation usually prevents this form of bias. • Need not be too expensive. • Essential to prevent cheating.

27. Secure allocation • Can be achieved using telephone allocation from a dedicated unit. • Can be achieved using independent person to undertake allocation.

28. Attrition Bias • Usually most trials lose participants after randomisation. This can cause bias, particularly if attrition differs between groups. • If a treatment has side-effects this may make drop outs higher among the less well participants, which can make a treatment appear to be effective when it is not.

29. Attrition Bias • We can avoid some of the problems with attrition bias by using Intention to Teach (or treat) Analysis, where we keep as many of the patients in the study as possible even if they are no long ‘on treatment’.

30. Dilution Bias • This occurs when the intervention or control group get the opposite treatment. • For example, in a trial of domestic violence the judge over-rode the random assignment and put 3.5% (14) men in the intervention group. Feder & Dugan, Justice Quarterly, 2002;19:343.

31. Resentful Demoralisation • This can occur when participants are randomised to treatment they do not want. • This may lead to them reporting outcomes badly in ‘revenge’. • This can lead to bias.

32. Resentful Demoralisation • One solution is to use a patient preference design where only participants who are ‘indifferent’ to the treatment they receive are allocated. • This should remove its effects.

33. Hawthorne Effect • This is an effect that occurs by being part of the study rather than the treatment. Interventions that require more TLC than controls could show an effect due to the TLC than the drug or surgical procedure. • TLC should be given to controls as well.

34. Forms of Bias • Subversion Bias • Technical Bias • Attrition Bias • Consent Bias • Ascertainment Bias • Dilution Bias • Recruitment Bias

35. Bias (cont) • Resentful demoralisation • Delay Bias • Chance Bias • Hawthorne effect • Analytical Bias.

36. Conclusions • There are a range of biases associated with POORLY designed RCTs. • Many, if not all, can be eliminated through careful attention to methods.