Randomized Controlled Trials: Allocation Concealment

1. Randomized Controlled Trials: Allocation Concealment Kenneth F. Schulz, PhD

2. Fixation on Sequence Generation • Traditionally, many medical researchers mistakenly consider the sequence generation process as “randomization” • They frequently ignore allocation concealment • Without adequate allocation concealment, proper random sequences can be subverted • For example, a properly generated sequence posted on a bulletin board

3. Allocation Concealment • Prevents selection bias by concealing the allocation sequence from those assigning participants to intervention groups • Until the moment of assignment

4. Mechanisms for Selection Biasin RCTs • If those responsible for entering participants know, or can detect, the upcoming treatment allocations: • Decide entrance based on that knowledge • Channel participants with a better prognosis to the experimental group and those with a poorer prognosis to the control group, or vice versa • In any case, they introduce bias into the treatment comparison

5. Do Not Confuse Allocation Concealment with Blinding • Allocation concealment seeks to prevent selection bias, protects assignment sequence before and until allocation, and can always be successfully implemented • In contrast, blinding seeks to prevent ascertainment bias, protects sequence after allocation, and cannot always be successfully implemented

6. Unclearly concealed and inadequately concealed trials, compared to adequately concealed trials, exaggerated the estimates of an intervention’s effectiveness by 30% to 40%, on average Importance of Allocation Concealment Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials.JAMA 1995;273:408-412.

7. Similar Results on Allocation Concealment in a Separate Study • Replicated our methods in different subject areas (digestive & circulatory diseases and mental health) • Examined 11 meta-analyses • Inadequately concealed trials exaggerated estimates of effectiveness by 37% [Moher D, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 609-13.]

8. Allocation Concealment: Summary • Adequate allocation concealment emerges from our analyses as crucial to reducing bias • Without it, the whole point of randomisation vanishes and bias may distort results • Our results support Mosteller and his colleagues “When the randomization leaks, the trial’s guarantee of lack of bias runs down the drain”

9. Method of allocation of treatment in 208 controlled trials in head injury Dickinson K, et al., BMJ 2000;320:1308-1311. Method of allocation No. of trials

10. Actual allocation method with the method just specified as “random” • Difficult (Peto and I. Chalmers experience) • MA of OCs • David G. knew the PI who put him in contact with the statistician • Described a “great” method . . . Post-it notes! • Seemed indignant . . . We asked for clarification • Perfect also for stratification—different colors!!

11. Personal Accounts of Deciphering Assignment Sequences • Conducted over 20 epidemiological workshops for medical residents and medical junior faculty • Each included 20-25 participants • Asked how many of the participants had deciphered, or had witnessed someone else decipher, an assignment sequence

12. Personal Accounts of Deciphering Assignment Sequences (cont.) • With assurance of anonymity, more than half related at least one instance of deciphering • Not half of all trials • No accurate denominator • Nevertheless, not a rare occurrence

13. Decipherings Run the Gamut From Simple to Intricate • Simple the most frequent and usually take advantage of inadequate allocation concealment schemes: • Posting sequence on a bulletin board • Opening unsealed envelopes • Translucent envelopes to a light bulb • Opening many envelopes that were not sequentially numbered

14. Intricate Decipherings Less Frequent • Needed to circumvent more adequate allocation schemes • With sequentially numbered drug containers • Based on appearance of tablets in unsealed containers • Appearance of the label • With central allocation, obtaining the next few allocations at once • Envelopes to a “hot light” in radiology

15. Attempted to Decipher a Numbered Container Scheme • Gave up • Found attending physician rifling the P.I.’s files for the assignment sequence • Horrified? No, impressed with his brilliance and proceeded to help • Should be kept in a locked location

16. Ignorance Is Not Bliss “At the annual meeting of the Society for Clinical Trials, held in Houston, … Schulz reported data showing that the experts’ fears about ignorance are justified. Schulz examined 250 reports of clinical trials and came up with an intriguing finding…” [Rachel Nowak. Problems in clinical trials go far beyond misconduct. Science 1994; 264: 1539.]

17. Ignorance Is Not Bliss “Most physicians are not trained in basic scientific principles, let alone clinical trials” Stephen George, Duke University, Chair of the Statistics Committee for NCI, NIH Cancer Clinical Cooperative Groups “I fantasize that in the future (training) will become a requirement for running clinical trials” John Gallin, Director of NIH’s Clinical Center

18. Rigorous Trials Annoy Humans • Investigators • Certain Ps to benefit • May want the results of study to reveal the “truth” • Proper trial procedures attempt to impede human inclinations • Hence, aspects of properly conducted RCTs annoy investigators

19. Without Methodological Rigour • The challenge of deciphering may frequently become too great a temptation to resist • Even without intent to bias • “The only way to get rid of a temptation is to yield to it.” Oscar Wilde • Deciphering may just reflect human inquisitiveness and ingenuity • Does not necessarily indicate scientific malevolence

20. Without Methodological Rigour • Or deciphering may reflect deliberate acts to alter findings • Whatever the motivations, innocent or naïve or deliberate, those actions undermine the validity of the trial • Investigators must devote diligent, persistent attention to randomization

21. Minimal Standards: Common Allocation Concealment Approaches • Sequentially numbered, opaque, sealed envelopes (SNOSE) • Pharmacy control • Numbered or coded containers • Central randomization • Realistically, these standards should be exceeded

22. Envelopes for Allocation Concealment • More susceptible to manipulation through human ingenuity • Less than ideal method of allocation concealment • If used, investigators must diligently develop and monitor the process

23. Envelopes (Cont.) • SNOSE • Ensure that the envelopes are opened sequentially • only after the P’s name and other details are written on the appropriate envelope

24. Envelopes (Cont’d) • Pressure-sensitive or carbon paper inside • Supports enrolment • Encourages orderly opening • Promotes proper assignment • Creates valuable audit trail • Cardboard or aluminum foil inside envelope

25. “Pharmacy Controlled” • Compliance with randomization/allocation concealment methods questionable • Aware of gross distortions • Pharmacy ran out of one drug . . . • Alternate assignment • Should not assume pharmacists knowledge • Ensure they, and all research partners, follow proper trial procedures

26. “Numbered Containers” • Excellent – prevents foreknowledge • Must take proper precautions • Appearance – exactly the same • Weight – exactly the same • Sound when shaken – exactly the same • Lettering – exactly the same • Odor – exactly the same • Sealed • Difference between concealment and blinding

27. “Central Randomization” • Frequently telephone • Fax • Email • Internet • “Computer” • Assignments only provided after participants irrevocably enrolled • Must check procedures for leaks • Monitor process for adherence

28. RCTs: Anathema to the Human Spirit • Must acknowledge the vagaries of human nature • Must establish methodological safeguards that thwart attempts to contaminate trials with bias

29. Subversions of Sequences • Conscious or subconscious? • Direct or compensatory? • Doesn’t matter • Biases the trial

30. Participants Randomize Placebo New Oral Drug 25% Non-compliance 75% Compliance 60% Compliance 40% Non-compliance Group representing the policy of no treatment Group representing the policy of oral treatment OutcomeOutcome

31. RCT Compared the Effectiveness of Clofibrate in Preventing Cardiac Deaths in Men Who Had Survived a Myocardial Infarction • Authors state that: • One can justify almost any conclusion, dependent upon the analysis chosen • Manipulating deviates leads to severe bias • Can you ever do so?

32. Exclusions After Randomization • Can introduce bias and should be carefully scrutinized • All randomized patients should be analyzed, and analyzed as part of the group to which they were initially assigned • ITT (Intention-to-treat)

33. Exclusions of LFU Damage Internal Validity • Without outcomes from those lost to follow-up, investigators have little choice but to exclude them from the analysis • Any losses damage internal validity • However, differential rates of loss among comparison groups cause major damage • Investigators must minimize their losses to follow-up

34. Retention of Trial Participants • Minimizing losses exudes difficulties • Eliminating losses may be impossible • But investigators too frequently profess insurmountable difficulties • Most should work harder to obtain higher follow-up rates • Investigators must commit adequate attention and resources to develop and implement procedures to minimize losses

35. Innovative Twists that Cultivate High Follow-up Rates • Establish many conveniently placed FU facilities • Too often investigators expect participants to visit a single, inconvenient location • Shortening data collection form caters to the participants’ wishes • Long sessions lead them to vote with their feet • Be considerate • Foster follow-up by not overburdening • Also may engender higher quality data on the main items

36. END

37. Meta-analysis of Polyglycolic Meta-analysis of polyglycolic acid versus catgut for perineal repair on short term pain 1 adequately concealed trial … OR = .89 4 unclearly concealed trials … OR = .44 Protective effect of polyglycotic acid was 51% lower in the unclearly concealed trial (ROR = .49; 95% CI of .35 - .69)

38. Meta-analysis of Corticosteriods Meta-analysis of corticosteroids vs. none, after preterm rupture of membranes, on respiratory distress syndrome (RDS) 3 adequately concealed trials … OR = 0.72 4 unclearly concealed trials … OR = 0.53 Protective effect of corticosteroids was exaggerated by 27% in the unclearly concealed trials ROR = 0.73; 95% CI of 0.35 - 1.50

39. Randomization Process (P 138B) In the 4 OB/GYN Journals, found the 2 treatment group sizes to be much too similar

40. Separation of Executer and Generator • Person(s) who prepare the randomization scheme should not determine eligibility, administer treatment, or assess outcome • Neglected critical element • Obviously important—regardless of methodological quality, access to sequence and thus opportunity to introduce bias • Faults in this element may be the crack though which much of the bias seeps into controlled trials

41. “Not quite as random as I pretended” [Martyn C. Not quite as random as I pretended. Lancet 1996; 347:70.] “Bias in the way treatments are assigned can be a more powerful determinant of the outcome of the trial than the treatments that are being investigated.” “Trialists … often find it hard to prevent their hopes that a new treatment might prove to be beneficial from evolving into a belief that it really is so.”

42. RCT in Norway • Examined the effect of Chlorhexidine compared to placebo on postpartum infection and Strep B infections in neonates • Reported as a randomized controlled trial in the published article Insert published article

43. RCT in Norway • Midwives knew on which shelf the Chlorhexidine and placebo were kept in the cabinet and just proceeded to administer whatever they deemed appropriate • Even if they forgot the shelf, they could smell the difference • No monitoring of the allocation process

44. Letter from a Physical Therapist • A “friend of mine” conducted an RCT comparing 2 training regimens following surgical repair of the knee for anterior cruciate ligament damage • “randomization method used was alternating assignment” • Referred to one patient as “difficult” and asked the friend to which group she was assigned • Friend responded “that she was in the group that required slightly less contact and that her inclusion was by his choice”

45. “Not so blind, after all” Sometimes subverting can work against a treatment’s apparent effectiveness “Medical staff … may try to help their sickest patients by sneaking them into the treatment group instead of the control group.” “The drug would then have to be significantly better than conventional treatment just to appear equal in efficacy” [Wallich P. Not so blind after all. Scientific American; May 1996.]

46. RCTs: Anathema to the Human Spirit • Scientific need to obtain unbiased data from an inherently biased source – human beings • Many trialists are unaware of the rationale behind RCTs • And the need for trial rigor • Some may intellectually grasp the need, but once immersed in a trial, have many contradictory interests (e.g. episiotomy trial in Canada)

47. Anathema: A Psychologist’s View • ”Fundamental conflict” between scientific and human imperatives … “Many probably think they already know what treatment is best.” • Geoffrey Scobie, psychologist at the University of Glasgow; Doctors cheat because they know what’s best New Scientist, 16 December 95, p. 10 • Of course, what we think we ‘know’ is frequently wrong

48. Reporting of Exclusions, Double-blinding, and Schedule Generation Related to the Level of Allocation Concealment for 250 Trials