When is Small Beautiful?

1. When is Small Beautiful? Richard Simon, D.Sc. Chief, Biometric Research Branch National Cancer Institute http://brb.nci.nih.gov

2. For Demonstrating a Large Treatment Effect

3. When the Size of the Treatment Effect is Large Relative to Inter-patient Variability

6. =.05, z1-=1.96 • =.10, z1-=1.28 • HR=0.67, =log(.67)=.40, Events=263 • HR=0.5, =log(.5)=.69, Events=88

7. Clinical Trials Show Small Treatment Effects Because(choose one) • Treatments are minimally effective uniformly across patients • Ineffectiveness of treatments for most patients dilutes average effects

8. Develop Predictor of Response to New Drug Using phase II data, develop predictor of response to new drug Patient Predicted Responsive Patient Predicted Non-Responsive Off Study New Drug Control

9. Evaluating the Efficiency of Targeting Clinical Trials to Best Candidates • Simon R and Maitnourim A. Evaluating the efficiency of targeted designs for randomized clinical trials. Clinical Cancer Research 10:6759-63, 2004; Correction and supplement 12:3229, 2006 • Maitnourim A and Simon R. On the efficiency of targeted clinical trials. Statistics in Medicine 24:329-339, 2005. • reprints and interactive sample size calculations at http://linus.nci.nih.gov

10. Relative efficiency of targeted design depends on • proportion of patients test positive • effectiveness of new drug (compared to control) for test negative patients • When less than half of patients are test positive and the drug has little or no benefit for test negative patients, the targeted design requires dramatically fewer randomized patients • The targeted design may require fewer or more screened patients than the standard design

11. Comparison of Targeted to Untargeted DesignSimon R,Development and Validation of Biomarker Classifiers for Treatment Selection, JSPI

12. TrastuzumabHerceptin • Metastatic breast cancer • 234 randomized patients per arm • 90% power for 13.5% improvement in 1-year survival over 67% baseline at 2-sided .05 level • If benefit were limited to the 25% assay + patients, overall improvement in survival would have been 3.375% • 4025 patients/arm would have been required

13. Small is Beautiful • When treatment effect can be measured with precision on individual patients • Little placebo effect • Comparative treatment effect not of interest

14. Small is Beautiful • When there is substantial prior information about the effect of the treatment compared to control

15. Frequentist Meta-Analysis of Two Trials of the Same Treatment

16. Random effects meta-analysis tests whether hypothetical distribution F from which 1 and 1 are drawn has mean zero. • With only two-trials, random effects meta-analysis does not have any information on variance of F and so no meaningful combined inference is possible

17. Principles of Bayesian Analysis • Evidence from data for a hypothesis should be based upon the likelihood of the actual data given the hypothesis, not upon the probability of data “as extreme” • Evidence from data for a hypothesis should be modulated by the prior probability of the hypothesis

18. Bayes Theorem

19. Specifying Prior Distributions • Non-informative • Elicit opinion • Skeptical/optimistic • Past data • Community concensus

20. Frequentist Methods are in many cases equivalent to Bayesian Methods Based on “Non-informative” Prior Distributions

21. “Non-informative” Prior Distributions are Sometimes Extreme and Unrealistic

22. Fallacies about Bayesian Methods • Require smaller sample sizes • Require less planning • Are preferable for most problems in clinical trials • Have been limited in application primarily by computing problems

23. Facts About Bayesian Methods • Require careful selection of prior distributions • Are valuable for some problems in clinical trials

24. Simple Bayesian Model

26. Bayesian Analysis May Be More Conservative Than Frequentist Analysis • Two hypotheses =0 and = 1 • Trial data

28. If trial is designed for power  and results are just significant at level  then (Simon, Statistical Science 15:103-105, 2000)

31. Bayesian Posterior Probability of Null Hypothesis When Trial Results are Just Significant

34. Bayesian Posterior Probability of Null Hypothesis When Trial Results are Just Significant(flat prior under alternative)

35. Small May Be Beautiful For • Randomized phase II study comparing a new regimen to control • Objective to obtain unbiased estimate; better than using historical control • Phase II endpoint may provide more events than phase III endpoint and therefore a smaller trial • Phase II endpoint may permit more sensitive estimate of treatment effect but not be a suitable phase III endpoint • Partial surrogate endpoint • Phase II study can be sized based on inflated 

36. Randomized Phase II Design Comparing Vaccine Regimen to Control •  = 0.10 type 1 error rate • Endpoint PFS • Detect large treatment effect • E.g. Power 0.8 for detecting 40% reduction in 12 month median time to recurrence with =0.10 requires 44 patients per arm with all patients followed to progression • Two vaccine regimens can share one control group in a 3 arm randomized trial

37. Small May Be Beautiful • When the objective is to select the most promising regimen from a set of candidates • May or may not contain control arm • Null hypothesis is never tested • All candidate regimens should be equal with regard to endpoints other than the one used as the basis for selection

38. Randomized Phase II Multiple-Arm Designs Using Immunological Response • Randomized selection design to select most promising regimen for further evaluation. 90% probability of selecting best regimen if it’s mean response is at least  standard deviations above the next best regimen

39. Number of Patients Per Arm for Randomized Selection DesignPCS = 90%

40. Patients per Arm for 2-arm Randomized Selection Design Assures Correct Selection When True Response Probabilites Differ by 10%

41. Randomized Selection Design With Binary Endpoint • K treatment arms • n patients per arm • Select arm with highest observed response rate • pi = true response probability for i’th arm • pi = pgood with probability , otherwise pbad • With N total patients, determine K and n to maximize probability of finding a good rx

42. Probability of Selecting a Good Treatment When pbad=0.1, pgood=0.5 and =0.1

43. Probability of Selecting a Good Treatment When pbad=0.1, pgood=0.3 and =0.1

44. Probability of Selecting a Good Treatment When pbad=0.1, pgood=0.3 and =0.25

45. Probability of Selecting a Good Treatment When pbad=0.02, pgood=0.15 and =0.15

46. Probability of Selecting a Good Treatment When pbad=0.02, pgood=0.10 and =0.15

47. Small May Be Beautiful • When the objective is to effectively treat the largest number of patients when the population of patients is small and several good candidate treatments are available

48. N patients in horizon • 2 treatments • Perform RCT with n pts per rx • Select treatment with best observed response rate and use that treatment for the remaining N-2n patients • Binary endpoint with unknown response probabilities p1 and p2

49. Approximate Total Number of Responses

50. N=1000, p1=0.6, p2 =0.4