Uses and Abuses of (Adaptive) Randomization: An Industry Perspective

1. Uses and Abuses of (Adaptive) Randomization:An Industry Perspective Benjamin Lyons, Ph. D. and Akiko Okamoto, Sc.D. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

2. Outline • Adaptive vs. Static Randomization • Implementation Challenges • Errors by Investigators • Errors in Algorithm • Errors related to Drug Supply • Conclusion Johnson & Johnson Pharmaceutical Research and Development L.L.C.

3. Adaptive vs. Static Randomization • Static randomization requires that one randomization list is generated at the start of the trial. • Adaptive (Dynamic) randomization algorithms (e.g., Urn model) assign treatments based on patient characteristics and previous treatment assignments. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

4. Covariate Adaptive Randomization • Treatment assignment of the (n+1)st patient may depend upon the previous first n patients. • Usual mechanism is a balance function that is minimized by assigning the new patient to a certain treatment. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

5. Why use adaptive randomization? • Treatment balance “required” within each level of stratification factors. • For small trials with many stratification factors static-stratified randomization will not insure balance within each strata or overall. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

6. Why avoid adaptive randomization • May be hard to interpret using standard theory (see recent CPMP guidelines on adjustments for baseline covariates). • Many chances to make errors. • Implications of some errors on inference are not easy to understand in the context of standard theory. • Some errors may put trial validity at risk. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

7. Implementation Challenges Three types of errors: • Errors by investigators; • Errors in the algorithm; • Errors caused by a faulty drug supply method. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

8. Example 1: Site Error • Site enters the wrong strata level for a patient. • Site assigns the wrong medication kit and perhaps treatment to patient. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

9. Response • Do we update the balance function by altering the assignment weights to reflect error? • If corrected there are three categories of balance functions: • randomized before the error; • randomized after the error but before the correction; • randomized after the correction. • If not corrected there are only two. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

10. Analysis • How do you report this? • Are the pre-specified test statistics asymptotically valid? • For stratification error is there a sensitivity analysis? • How should you incorporate into a permutation or or resampling procedure? Johnson & Johnson Pharmaceutical Research and Development L.L.C.

11. Prevention • Site training. • Train sponsor staff on how to react to the error. • Giving IVRS vendor staff explicit instructions on who decides to update the algorithm. • Is it sound to alter the algorithm for a few minor errors? Johnson & Johnson Pharmaceutical Research and Development L.L.C.

12. Example 2: Algorithm Error • Specification is correct for 1:1 assignment as indicated by simulation in SAS. • Actual code to calculate assignment written in an SQL program. • Validation of SQL program did not include any simulation. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

13. Result • Error in SQL program detected after 50% enrollment. Balance is 2:1. • Program fixed so that the balance at the end of the trial is 1:1. • Probability of treatment assignment correlated with date of trial entry. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

14. Analysis • Is this trial randomized? • Are the standard test statistics asymptotically valid. • How should we account for the error in any permutation test? • Should the trial results be reported at all? • Could entry time be correlated with patient characteristics and hence outcome? Johnson & Johnson Pharmaceutical Research and Development L.L.C.

15. Prevention • Validate the actual software that produces the assignment through simulation prior to roll out. • Check balance results frequently during the trial. • Vendor must have a responsible/trained statistician who understands the issues. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

16. Example 3: Drug Supply • Supply at sites is not adequate. • Lack of study drug. • Drug not re-supplied often enough. • High enrollment in short periods. • Uneven enrollment by site. • In some cases all treatment arms are not available when a subject is randomized. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

17. Response • System provides “over rides” or “forced randomizations”: • the patient is assigned to available treatment regardless of what the algorithm says. • Adaptive algorithm is ignored for this patient. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

18. Result • Trial should be balanced if only a few occurrences. • “Forced” assignment included in the balance function. • The algorithm has not been implemented as stated in the protocol and the report. • Are subsequent randomizations that used the faulty balance function valid? Johnson & Johnson Pharmaceutical Research and Development L.L.C.

19. Analysis • Are the standard test statistics asymptotically valid? • How does a permutation test account for the ‘over rides’? • How many “forced” assignments before the entire randomization is suspect? Johnson & Johnson Pharmaceutical Research and Development L.L.C.

20. Prevention • Supply trials with dynamic randomization centrally with one kit going to each site after each randomization. OR • Have abundant supply at all sites. OR • Do not allow forced randomization, turn patients away if all arms not available. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

21. Simulation • 171 Subjects. • Two treatment Arms: A and B. • 4 Strata: Site (16) and three prognostic factors (2,2, and 4 levels). • Randomization by Biased Coin. • Entry time , stratification and response based on CNS trial. • Assignment is altered in 10,000 replications. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

22. Supply Algorithm • Each site began with 4 kits: 2 A and 2 B. • Re-supplied in 1 “day” with two kits when one arm is empty. • Patients may enter with only 1 arm available. • If arm assigned by IVRS was missing then the remaining treatment was given. • Drug supply is part of the simulation. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

23. Results • For 10000 “trial” simulations • Average of 5 “forced randomization: per trial; • T-statistic calculated for each “trial”; • Distribution similar to the theoretical. • Supply error has no effect. Johnson & Johnson Pharmaceutical Research and Development L.L.C.

24. Conclusion • Adaptive Randomization is more difficult to execute then static randomization. • There are several sources of error. • Result of errors are poorly understood. • Some errors may be “minor” errors. • Using Adaptive randomization adds costs and risk to running a trial. Johnson & Johnson Pharmaceutical Research and Development L.L.C.