Using Simulation: Why the Planning is as Important as the Execution

1. Using Simulation: Why the Planning is as Important as the Execution Andy Grieve, Ph.D. SVP Clinical Trials Methodology Innovation Centre Andy.Grieve@aptivsolutions.com

2. Outline • History of Simulation • Introduction • Basic Questions in Planning • Ex. Sample Sizing Simulations • Choice of Output Measures • Random Number Generation • Efficient Programming • Experimental Design

3. Early Monte-Carlo Mechanical Simulation • Pearson Quincunx • Buffoon’s Needle • Prob (Needle Crosses a line)= • Thompson’s Bayesian Randomiser

4. Gossett (Student) t-test and correlation coefficient WW2 Los Alamos – Stanislaw Ulam & John Von Neumann Original work to calculate multi-dimensional integrals Early Monte-CarloBeginnings of Simulation

5. Introduction • Main Purpose of Simulation – • running scenarios, interpret & understand results • Extremely Important to Plan before simulating • Just trying different scenarios, models in a random fashion is an inefficient way to learn • Careful planning of simulations -> Improves efficiency • Both computational and statistical • Simulation is an EXPERIMENT. • Plan simulations as an experiment.

6. IntroductionBasic Questions in Planning • Questions to ask in planning simulations • What scenarios are we interested in? • What factors might be influential? • How can the factors be varied ? • Use the same or different random numbers across scenarios? • How many simulations ? • Analysis of output? • How can we make simulations efficient?

7. Accuracy of SimulationsPosch, Maurer and Bretz (SIM, 2011) • Studied adaptive design with treatment selection at an interim and sample size re-estimation • Control FWER (family-wise error rate) in a strong sense – under all possible configurations of hypotheses • Conclude: That you have to be careful with the assumptions behind the simulations. • Intriguing point: the choice of seed has an impact on the estimated type I error

8. Accuracy of SimulationsPosch, Maurer and Bretz (SIM, 2011) • Studied adaptive design with treatment selection at an interim and sample size re-estimation • Control FWER (family-wise error rate) in a strong sense – under all possible configurations of hypotheses • Conclude: That you have to be careful with the assumptions behind the simulations. • Intriguing point: the choice of seed has an impact on the estimated type I error

9. Accuracy of SimulationsPosch, Maurer and Bretz (SIM, 2011) • Their Argument: • Monte Carlo estimates of the Type I error rate are not exact – subject to random error -> the choice of the seed of the random number generator impacts Type I error rate estimate. • A strategy of searching for a seed that minimizes estimated Type I error rate can lead to an underestimation of Type I error rate. • Ex: Type I error rate is estimated in a simulation experiment by the percentage of significant results among 104 (105) simulated RCTs, on average the evaluation of only 4 (45) different seeds will ensure a simulated Type I error rate below 0.025 when the actual error rate is 0.026.

10. Accuracy of SimulationsPosch, Maurer and Bretz (SIM, 2011) • If it is important to be able to differentiate between 0.025 and 0.026 then we should power our simulation experiment for it • A sample of 104 has only 10% power to detect HA=0.026 vs. H0=0.025, (105: 50%) • 80% power requires n=194,000 – search 380 seeds • 90% power requires n=260,000 – search 1600 seeds

11. Average Run Length to find a “Good Seed” 10000000000 N=106: number of seeds =8*109 1000000000 100000000 10000000 1000000 100000 Average Run Length 10000 1000 100 10 1 100 1000 10000 100000 1000000 Simulation Sample Size

12. Output Measures • Need to decide what aspects of output you are interested in – some may be defined by purpose of simulations – operating characteristics (power, type I error) • Often outputs are observations from probability distributions • BEST - estimate the whole distribution – may be a stretch goal ! BUT can be important • Usually summarise the distributions • Means – definitely too much focus • Should be interested in extremes – how bad could it be • Variability • Quantiles of output distribution • Output desired can affect model, data structure

13. Bayesian AD – Thall & Wathen(EJC,2007) N=200Randomisation Probabilities (105 simulations) 0.30 0.25 0.20 Standard Deviation of Randomisation Probability 0.15 0.10 W=1 W=n/(2N) 0.05 0.00 pA=0.25 , pB=0.45 1.0 0.9 0.8 Randomisation Probability 0.7 0.6 0.5 0.4 0 20 40 60 80 100 120 140 160 180 200 Patient Number

14. Posterior Distribution of ED95Estimator – Posterior Mean 25 20 15 10 5 0 0 10 16 22 27 33 38 45 52 59 67 76 84 96 108 120 Dose (mgs)

15. Random NumbersUtilising Their Strength / Dangers of their Use • Whatever software used for simulation has its own Random Number Generator • Rely on random-Number Generator • Algorithm to produce a sequence of values that appear independent, uniformly distributed on [0, 1] • Generators are recursive formulae generating the same sequence THAT WILL EVENTUALLY CYCLE AND REPEAT • We want a “good” generator with LONG cycle length

16. A sceptical prior can be set up formally for d Prior N(0, s2/(fn)), with a small probability g of achieving the alternative (dA) - p(d>dA) = g From which Now suppose the trial has been designed with size a and power 1-b to detect the alternative hypothesis dA. So that From which: Example: α=0.05, 1-=0.90, =0.05 => f ~ 1/4 Bayesian Monitoring of RCTs (Interim Analysis) Sceptical Prior for a parameter d 0 dA sceptical prior

17. Bayesian Monitoring of RCTs (Interim Analysis)Sceptical Prior • Monitor Using: which is equivalent to increasing the critical region by a factor Grossman et al(1994) call f the “handicap”

18. Bayesian Monitoring of RCTs (Interim Analysis) Handicap) To Control the Two-sided afor Up to 25 Analyses a 1.0 0.30 0.8 0.25 0.20 0.6 0.15 Handicap (f) 0.10 0.05 0.4 0.01 0.2 0.0 0 5 10 15 20 25 Number of Analyses (T)

19. Bayesian Monitoring of RCTs (Interim Analysis) Frequentist Properties • The frequentist properties of this handicapping are not so easy to derive. • For T > 3 Grossman et al (1994) use simulation to determine the handicap f that controls the two-sided type I error at 5% and 1% (20,000,000 trials) • Alternatively use can be made of the algorithm derived by Armitage, MacPherson and Rowe (JRSSA, 1969)

20. Generated random numbers are controllable This control can be used to reduce variance of output, without simulating more (EFFICIENCY) Part of designing simulation experiments is to decide how to allocate random numbers First thought –independent (no reuse) throughout Certainly valid and simple statistically But gives up variance-reduction possibility Usually takes active intervention in simulation software –New run always starts with same random numbers –override Using Random Numbers Appropriately

21. Use Random Numbers Appropriately • Better idea when comparing scenarios • Use the same random numbers across scenarios • Not confounding scenario differences with random number differences • Probabilistic rationale: • Var(A–B) = Var(A) + Var(B) –2 Cov(A, B) • Use the same Random Numbers for the same purposes

22. Use Random Numbers AppropriatelyPredictive Calculations • Planning an RCT (150 patients) • 24 month outcome • Looking to find a time point (based on # pts recruited) to conduct an interim analysis to stop for futility • 1st Approach • Based on 60, 70, 80 and 90 pts predict final outcome and the probability of success • Generate final outcome for each of the 150 patients and choose time and decision criterion based on “diagnostic criteria2 • 2nd Approach • Generate data on all 150 patients (interim + final) and choose time and decision criterion based on “diagnostic criteria”

23. Ebbutt et al (1997). The analysis of trials using a minimisation algorithm. In PSI Annual Conference Report Re-randomisation Analysis Re-randomise order of patients entering study Minimise Calculate statistic Evaluate empirical cuff %DO > 2 days Data steps ~ 10 mins Novikov (The Statistician, 2003) Efficiency in SimulationImpact of Minimisation on Type I Error

24. & Adapt Complex Bayesian Adaptive Design for Dose Selection Increase Number of Doses • Minimises Predictive variance of ED95 • SAS / WinBUGS to fit NDLM, a series of linear regressions • Problem : Data file that WinBUGS needs to process increases after every patient • Parameterize Likelihood using sufficient statistics Response Dose

25. Complex Bayesian Adaptive Design for Dose Selection Setting Up the Simulations to Determine OC • Complex Design • Many parameter choices (101 control paramours for simulation and analysis) • Searching for “optimal” choices • Statisticians met to discuss what needed to be simulated • It was a long meeting – many parameter settings suggested for testing • BUT – no process, no structure – random scenarios • Conclusion – simulations would take too long

26. Complex Bayesian Adaptive Design for Dose Selection Setting Up the Simulations • How do we design a simulation experiment ? • What are the factors ? • Can we use a fractional factorial / who can help us ? • The expert asked : • Not only about the factors • What were the outcomes ? • bias, sample size, operating characteristics, allocation to placebo • relevant importance

27. Termination Rule Decision theoretic Posterior Probabilities Prior Distribution Uninformative (flat) prior Informative prior Variance of Prior Constant Less variable at low end of the dose range Level of Smoothing High smoothing Low Smoothing Allocation Criteria Var(ED95) x Var(f(z*)) Var(f(z*)) Var(ED95) Determinant of Covar ((ED95) and Var(f(z*)) Randomisation Method Probability of allocation proportional to expected utility Allocate to maximum utility Probability uniform over doses st 0.9f(z0) < E[f(z)|Y] < 1.10f(z0) Probability uniform over doses st 0.9f(z0) < E[f(z)|Y] Complex Bayesian Adaptive Design for Dose Selection Factors of the Simulation Experiment • 24 x 42 experiment 1/4 replicate

28. Complex Bayesian Adaptive Design for Dose Selection Aliasing Structure

29. 20 18 16 14 Change from baseline 12 10 8 0.0 0.5 1.0 1.5 Dose Complex Bayesian Adaptive Design for Dose Selection Simulated Dose-Response Curves Height : 2 or 4 pts

30. Complex Bayesian Adaptive Design for Dose Selection Simulation Experiment • Simulated • 20 replicates • 64 experiments • 9 curves • 11520 individual studies • Run-on a network of PCs/Workstations • Having established “optimal” settings • 1000 simulations were performed to establish operating characteristics

31. Complex Bayesian Adaptive Design for Dose Selection Brute Force Solution

32. Conclusions • Grieve’s First Law of Influential Statisticians: The three most important areas where statisticians can contribute, and be influential are – design, design and design