Methodologic Challenges: Appraisal of Evidence

1. Methodologic Challenges:Appraisal of Evidence Ralph M. Meyer NCIC Clinical Trials Group and Queen’s University

2. Appraisal of Clinical Trials • 101: Some Basics • 202: Strategic Principles of Trial Design • 303: The Interim Analysis • 404: Biomarkers Beyond: Some issues of drug development

3. 101 Some Basics

4. Some Basics • Were the patients really randomized? • Were clinically relevant outcomes reported? • Is the population recognizable? • Was there clinical + statistical significance? • Is the intervention feasible? • Were all accounted for? Sackett et al, 1985 (1st Ed) Guyatt, JAMA 1993

5. Some Basics Beware of: • The incomplete randomization

6. FFTF Diehl, NEJM 2003

7. Some Basics Beware of: • The incomplete randomization • Surrogate outcomes

8. Hierarchy of Outcomes Major Survival QoL Economic FFP Hospitalization Response Toxicity Surrogate Meyer, Kouroukis, Evid-based Oncol: 2001

9. Outcome Measures for Research Trials Basic Research Investigators Investigators Phase I Trials Phase II Trials Phase III Trials, Systematic Reviews Outcomes Pharmacokinetics Toxicity Response Optimum Dose Outcomes Response FFPD Toxicity Case Reports Outcomes FFPD Survival Quality of Life Toxicity Practitioners Practitioners Reviews Haynes, Ann Int Med 1990 Meyer, Kouroukis; Evid-based Oncol, 2001

10. Some Basics Beware of: • The incomplete randomization • Surrogate outcomes • QoL – small differences / no response • Control arm interventions • Over and underpowering

11. 202 Strategic Principles of Trial Design

12. Strategic Designs of Phase III Trials • Explanatory vs. Pragmatic • Large Simple Trials • Non inferiority Trials

13. Explanatory vs. Pragmatic

14. Explanatory vs. Pragmatic Trials Explanatory • Tests a biologic principle / causal effect • Emphasize efficacy Pragmatic • Tests a treatment policy • Emphasize effectiveness

15. Example 1 observe Control Group RT 30 days Expt’l Group RT treat The explanatory trial: Does 30 days of a radiosensitizer have a biologic benefit? Schwartz + Lellough, J Chron Dis, 1967

16. Example 2 Control Group RT 30 days Expt’l Group RT treat The pragmatic trial: Does 30 days of a radiosensitizer improve health outcomes? Schwartz + Lellough, J Chron Dis, 1967

17. Large Simple Trials

18. Large Simple Trials Principles: “The real differences between two treatments in some important outcome will probably not be large, but even a moderate difference in an important outcome may be worthwhile” Peto, Collins, Gray, J Clin Epi, 1995

19. Large Simple Trials Principles / Implications: • Seeking large effect sizes is impractical • If small / moderate effect sizes are sought, the experimental design must get it right: • Minimize bias • Minimize random error • ergo, large sample size

20. N = 17,187 ISIS-2, Lancet 1988

21. Large Simple Trials Principles / Implications: • May be used to compare existing standards • May be used to confirm a meta-analysis • May not test a paradigm change • Are subject to biomarker qualification

22. Large Simple Trials Beware of: • Very small differences / large NNTs • Heterogeneous populations in an era of: - targeted therapy - biomarkers

23. Superiority vs. Non-inferiority

24. Superiority vs. Non-inferiority A new treatment is: • ‘as good’ at disease control and is: • Less toxic • Associated with a better QoL • More cost effective • More convenient

25. Superiority vs. Non-inferiority Key Principles: • Include superiority for a 2o outcome • Define the non-inferiority boundary The benchmark will be the upper 95% CI • Be better than ‘putative placebo’ • Include an as-treated analysis Kaul, Ann Int Med 2006

26. Treatment Differences in Noninferiority Trials Piaggio, JAMA 2006

27. The Putative Placebo Kaul, Ann Int Med 2006

28. Superiority vs. Non-inferiority Beware of: • A wolf in sheep’s clothing (superiority trial that fails to meet endpoint) • The lack of a superior 2o outcome

29. 303 The Interim Analysis

30. Interim Analysis • Trials test hypothesis; equipoise exists • Cumulative data address the hypothesis • These data can confirm or reject the hypothesis • If conclusively addressed, it would be both unethical and an unwise use of resources to continue to conduct the trial • Interim analysis are therefore appropriate • The issue is methodological soundness

31. Interim Analysis Issues of methodologic soundness: • There should be sufficient events • There should be predefined boundaries • The boundaries should be based on sound application of statistical principles e.g., O'Brien-Fleming boundaries • Data should be independently reviewed (DSMC) • Follow-up should continue

32. Jennison and Turnbull

33. Interim Analysis Beware of: • Repetitive reporting • No statement about boundaries • Lack of an independent DSMC • Lack of data cleaning processes • Results that are ‘too good’ (but be careful)

34. 404 Biomarker Development

35. Some Definitions Biomarker (Biological Marker): • A characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes or pharmacologic responses to a therapeutic intervention NIH Biomarkers Definition Working Group, Clin Pharmacol Ther, 2001

36. Potential Role of Biomarkers • Define causation • Early detection / screening • Assist in making a diagnosis • Define a therapeutic target • Facilitate anti-tumour response assessment • Influence getting therapy, through prognosis • Determine who gets which therapy, through prediction • Define details of intervention (e.g., dose)

37. Some Definitions Prognostic Marker: • Identify patients with differing risks of specific outcomes, such as progression or death Predictive Marker: • Predicts the differential efficacy of a particular therapy based on the marker status Sargent, J Clin Oncol, 2005

38. Predictive Markers Confirmation of a predictive marker follows the same principles as confirming best therapy: An RCT is required Principles are aligned with those of a subset analysis

39. Register Indirect Test Biomarker Biomarker-’ve Biomarker+’ve R R R HR HR Rx A Rx B Rx A Rx B Sargent, J Clin Oncol, 2005

40. Register Indirect Test Biomarker Biomarker-’ve Biomarker+’ve Role of biomarker can be tested through statisitical interaction R R R HR HR Rx A Rx B Rx A Rx B Sargent, J Clin Oncol, 2005

41. Register The statistical test for interaction is crucial: Rx B may just be better therapy Indirect Test Biomarker Biomarker-’ve Biomarker+’ve R R R HR HR Rx A Rx B Rx A Rx B Sargent, J Clin Oncol, 2005

42. NCIC CTG CO.17 HR = .77 P=.005 Med. estimates 6.1 vs. 4.6 mos Jonker, NEJM 2007

43. Laurent-Puig, Clin Cancer Res 2009

44. Karapetis, NEJM 2008

45. K-ras is not a prognostic marker Karapetis, NEJM 2008

46. Overall Survival HR = .98 HR = .55 Test for interaction P < 0.001 K-ras is a predictive marker Karapetis, NEJM 2008

47. The statistical test for interaction is crucial: Rx B may just be better therapy (P < 0.001) Register Indirect Test Biomarker K-ras wild type K-ras mutant HR = .98 HR = .55 R R R Rx A Rx B Rx A Rx B Sargent, J Clin Oncol, 2005

48. Register Direct Test Biomarker Biomarker determines Rx Rx is standard R Biomarker –’ve Biomarker +’ve Rx A Rx A Rx B Sargent, J Clin Oncol, 2005

49. Carde, J Clin Oncol: 1993

50. The biomarker Carde, J Clin Oncol: 1993