Controversy? What controversy? An attempt to structure the debate on adaptive designs

1. Controversy? What controversy?An attempt to structure the debate on adaptive designs Marc Vandemeulebroecke MCP, Vienna, 11.7.2007

2. Vandemeulebroecke Controversy? What controversy? 1

3. Examples (1): Sequential analysis A. Wald (1945): „The National Defense Research Committee considered these developments sufficiently useful for the war effort to make it desirable to keep the results out of the reach of the enemy.“ F. J. Anscombe (1963): „‘Sequential analysis‘ is a hoax.“ P. Armitage (1989): „If [...] ‘sequential analysis is a hoax‘, then its proponents are amongst the most persuasive swindlers, and their clients amongst the most gullible consumers, in the history of scientific endeavour.“ Vandemeulebroecke Controversy? What controversy? 2

4. Repeated confidence intervals (C. Jennison, B. Turnbull 1989) such that for all Example (2): Group sequential designs P. Armitage: „ingenious derivation“ S. J. Pocock: „a minor change of presentation style“ Vandemeulebroecke Controversy? What controversy? 3

5. Test H0: with ; interim: Change to observe  Example (3): Adaptive/flexible designs C.-F. Burman, C. Sonesson (2006): Are flexible designs sound? „The weighted test violates inference principles [...] and may lead to unreasonable results.“ M. Proschan: „Such methods, if extremely abused, produce illogical conclusions, but that is no more a condemnation of adaptive methods than Jack the Ripper is a condemnation of cutlery.“ P. Bauer: „Is wine sound? It is the way to drink it that matters.“ Vandemeulebroecke Controversy? What controversy? 4

6. Example (4): Terminology A. J. Phillips, O. N. Keene (2006): „The terms ‘flexible design’ and ‘adaptive design’ are used interchangeably.” C. Jennison, B. Turnbull (2006): „The adjectives ‘flexible’ and ‘adaptive’ are not synonymous. Indeed adaptive designs can range from being totally inflexible to very flexible.” Vandemeulebroecke Controversy? What controversy? 5

7. Issues • Imagine a study protocol: • Double-blind multicenter trial in Crohn‘s Disease • Active vs placebo 1:1 • Endpoint: CDAI after 4 weeks • IA at half of the sample size, switch to 2:1 if encouraging • Trial conduct: • IA disappointing, but new budget available •  Continue 1:1, ad hoc increase of sample size • Final analysis correctly adjusted and significant • In stage 2, patients are more severe, and placebo is doing • much worse Vandemeulebroecke Controversy? What controversy? 6

8. Issue 1: Feasibility • Practical, logistical or operational issues regarding the conduct of the study • Speed of recruitment vs. speed of response • Availability of specialized software • „Supply chain management“: Data capture/cleaning/ processing; real-time randomization and drug supply • Training/SOPs/“charter“ • Investigate behavior of procedure by simulation, play through scenarios with decision makers / IDMC Vandemeulebroecke Controversy? What controversy? 7

9. Issue 2: Validity • Statistical/mathematical property. • Correct type I error • Point estimates available and sensible (e.g., mean/median unbiased) • Confidence intervals available, sensible and correct Vandemeulebroecke Controversy? What controversy? 8

10. Issue 3: Integrity • Hard to quantify. Concern: „operational bias“ • Consistency/homogeneity across trial stages (A. Koch: identification problem)  MCP: T. Friede, A. Faldum... • Jeopardizing integrity means jeopardizing credibility. • Measures to maintain trial integrity: • avoid unblinding if possible • control dissemination of results • control scope and amount of adaptations Vandemeulebroecke Controversy? What controversy? 9

11. Issues 2/3: Validity/Integrity V. Dragalin (2006, PhRMA): „To maintain study validity means providing correct statistical inference [...], ensuring consistency between different stages of the study, and minimizing operational bias. To maintain study integrity means providing convincing results to a broader scientific community; preplanning, as much as possible, based on intended adaptations; and maintaining the blind of interim analysis results.“ H. Schäfer (2006): internal validity - external validity Vandemeulebroecke Controversy? What controversy? 10

12. Issue 4: Interpretability • Only interpretable results are communicable. • Intregrity and interpretability have the main impact of the persuasiveness of the study results. • Endpoints meaningful • Point estimates and confidence intervals available and sensible • Jeopardizing integrity (e.g., across-stage consistency or amount of adaptations) will also reduce interpretability. Vandemeulebroecke Controversy? What controversy? 11

13. Issue 5: Efficiency • Often, adaptation comes at the cost of efficiency. • ...in terms of power / expected information • Reason: non-sufficient statistics • Any adaptive design can be outperformed by a nonadaptive design - and vice versa! • „Statistical efficiency is not always the priority“ (Koch 2006) • „Response-dependent choice of group sizes can itself be a source of improved efficiency“ (Schmitz 1993, Jennison/Turnbull 2006) Vandemeulebroecke Controversy? What controversy? 12

14. Nature of change Prespeficied rule Completely free Not dep. on trial-internal data stopping Adaptable features + stage sizes + other Issue 6: Flexibility • Two main questions: • Should there be an a priori selection of study features that may be adapted? If so, now many and which? • Should the design changes follow prespecified rules or may they be done ad hoc? Should they depend on information internal or external to the trial? Group sequential „Schmitz designs“ Spending functions p-value combinations Vandemeulebroecke Controversy? What controversy? 13

15. Summary/conclusions • Issues • Feasibility: „Only“ operational issues • Validity: „Only“ mathematical issues • Integrity: Requirements? • „Operational bias“? Homogeneity across stages? • Interpret‘y: Requirements? • Efficiency: Weigh against flexibility (sponsor‘s choice) • Flexibility: A priori selection of adaptable features? How many/which? Adaptations ad hoc or following prespecified rules? Depending on internal or external information ? Vandemeulebroecke Controversy? What controversy? 14

16. Some references Biom J 48 (2006, special issue), in particular: O‘Neil, Hung et al., Koch, and discussants: Section on „Regulatory perspectives“, pp. 559-622 Burmann, Sonesson (2006):Are flexible designs sound? Biometrics 62, pp. 664-683 (with discussion) EMEA (2006):Reflection paper on methodological issues in confirmatory clinical trials with flexible design and analysis plan. CHMP/EWP/2459/02 Fleming, DeMets (1993): Monitoring of clinical trials: Issues and Recommendations. Contr Clin Trials 14, pp. 183-197 Gallo et al. (PhRMA, 2006):Adaptive designs in clinical drug development - an executive summary of the PhRMA working group. J Biopharm Stat 16, pp. 275-283 (with discussion) Jennison, Turnbull (2006):Adaptive and nonadaptive group sequential tests. Biometrika 93, pp. 1-21 Phillips, Keene (PSI, 2006):Adaptive designs for pivotal trials: discussion points from the PSI adaptive design expert group. Pharm Stat 5, pp. 61-66 PhRMA (2006):Full white paper. DIJ 40, pp. 421-484 Posch, Bauer, Brannath (2003):Issues in designing flexible trials. Stat in Med 22, pp. 953-969 Stat in Med 25 (2006, special issue), pp. 3229-3408 ...and many thanks to Norbert Benda! Vandemeulebroecke Controversy? What controversy? 16

17. V. Dragalin (2006): “a multistage study design that uses accumulating data to decide how to modify aspects of the study without undermining the validity and integrity of the trial. [...] preplanning, as much as possible, based on intended adaptations.” Appendix: Terminology PhRMA: Definition of “adaptive design” P. Gallo et al (2006): “a clinical study design that uses accumu-lating data to decide how to modify aspects of the study as it continues, without undermining the validity and integrity of the trial. [...] In such trials, changes are made ‘by design’, and not on an ad hoc basis.” Vandemeulebroecke Controversy? What controversy? A1

18. Novartis slide deck: “Adaptive designs allow for initial uncertainties in trial design to be confirmed/adapted during the trial. The integrity of the trial is maintained and the evidence for the same hypothesis before and after the adaptation is combined.” Appendix:Terminology Other definitions of “adaptive design” EMEA (2006): “A study is called ‘adaptive’ if statistical methodology allows the modification of a design element [...] at an interim analysis with full control of the type I error.” Vandemeulebroecke Controversy? What controversy? A2

19. Appendix:Terminology • Other terms • flexible • adaptive combination • flexible group sequential • flexible adaptive • adaptive group sequential • internal pilot • self designing • seamless adaptive • design automates • .... (List stolen from J. Röhmel‘s talk at BfArM Discussion on Adaptive Designs, 2006) Vandemeulebroecke Controversy? What controversy? A3

20. Appendix:Flexibility Jennison, Turnbull (2006): „The strength of adaptive redesign lies in coping with the unexpected, in particular responding to external information that could not have been anticipated at the start of a study.“ M. Frisén (2006): „One should use planned adaptive designs when one expects that it will be necessary to adapt the design. If one unexpectedly has strong reasons to change the plans, one should be very careful.“ P. Gallo et al. (2006): „Should or should it not be viewed that adaptive designs might remedy inadequate planning?“ Vandemeulebroecke Controversy? What controversy? A4