Moderators of Treatment Effects in the General Medicine Literature: Looking for Improvement

1. Moderators of Treatment Effects in the General Medicine Literature: Looking for Improvement Nicole Bloser, MHA, MPH University of California, Davis June 5, 2007

2. Background • Parallel group randomized controlled trials (RCTs) are the cornerstone of evidence based medicine • Result: Average of usually immeasurable individual effects • Some benefit, some harmed • Heterogeneity of treatment effects

3. Background • Examining treatment impact in similar individuals or subgroups • N-of-1 clinical trial • Prospective stratification with a multivariable risk index • Subgroup analysis

4. Background • Subgroup analysis – problems • Low power • Multiple testing • Not all subgroup analysis is the same • P-value within subgroup ≠ interaction analysis • Interaction analysis is the correct way to examine moderators of treatment effects (MTEs)

5. Background • MTEs necessary to maximize benefit and minimize harm • MTEs often not examined (<50% of RCTs report interaction analysis) • Studies that examine MTE reporting: • Majority in cardiovascular literature • None since revised CONSORT statement (2001)

6. Research Objective • We sought to identify current practice in evaluating moderators of treatment effects (MTEs) and to elucidate trends • Persistent low rate of analyses would suggest • Missed opportunities • Slower progress towards personalized medicine

7. Study Design • Systematic review • Annals, BMJ, JAMA, Lancet, NEJM • Odd months, 1994, 1999, 2004 • Randomized controlled trials • Unit of randomization as the individual • All independently reviewed and coded by two investigators • Adjudication by a third

8. Study Selection 4863 articles from initial search Exclude: All articles that were not clinical trials. N=4,322 N=541, Random sample of N=379 selected Include Exclude N=303 articles N=76 articles Reasons for exclusion: Not RCT (N=61 trials) Unit of randomization not individual patient (N=25 trials) 319 trials included 9 trials excluded 77 trials excluded All articles from Annals, BMJ, JAMA, Lancet, and NEJM for odd months of 1994, 1999, and 2004. N = 4863

9. Methods • Trials were coded as having • MTE analysis (utilizing a formal test for heterogeneity) • Subgroup analysis only (no formal test) • Neither • Chi-square test used for bivariate comparisons • Multiple logistic regression used to identify predictors of MTE analysis

10. Trial Characteristics

11. Trial Characteristics • 38% of trials had authors from North America (US and Canada) • 95% utilized a parallel group design • Study sample size ranged from 6-41,000 (Median: 262, IQR: 101-708)

12. MTE and Subgroup Reporting • For those trials reporting MTE analysis: • 43 (47%) reported on one covariate • 24 (26%) reported on 2-4 covariates • 17 (18%) reported on 5-10 covariates • 7 (7%) reported on 11-19 covariates

13. Covariates examined for MTE Among those trials that reported MTE, major covariates examined included: Only one trial reported MTE analysis using a composite multivariable risk index.

14. Bivariate analysis • Journals published in North America (Annals, JAMA) and first authors writing from North America were more likely to publish trials with MTE analysis • Sample size (p<0.0001 for trend) • Quintile 1: 14%, Quintile 5: 52% • More MTE analysis reported in each of the three successive time periods (p=0.047)

15. Logistic Regression • Prediction of MTE analysis • Included in model: study year, journal clinical condition, first author’s region, and sample size • Journal and sample size were significant • Reference categories: BMJ, Quintile 1 • JAMA: 4.4 (1.4, 13.5), Annals: 4.2 (1.2, 15.1) • Quintile 5: 7.5 (2.9, 19.3)

16. Limitations • Limited number of trials reviewed • MTE results are published elsewhere • MTE examined, but not reported due to non-significance

17. Conclusions • Missed opportunities for MTE analysis abound • Conservative reaction that stifles hypothesis generation • Impairs recognition of patient strata • Impedes future research • When subgroups are reported, the reporting is not done correctly in many (~50%) cases. This could lead to erroneous conclusions.

18. Conclusions • NIH guidelines regarding subgroup specific results recommends reporting both significant and non-significant results, yet only half of the trials reported any MTE analysis. • In the face of broad NIH mandates for inclusion of subjects by race/ethnicity, the low proportion of trials examining race/ethnicity as a treatment effect modifier is puzzling.

19. Policy Implications • MTE analysis critical to future research • The genomic revolution is only going to increase the desire to individualize treatment effects • Kraemer et al. argue that exploratory moderator analysis is critical for designing future confirmatory studies • Significant exploratory effects are later used as guidance for future stratification Kraemer, HC, E Frank, and DJ Kupfer, Moderators of treatment outcomes: clinical, research, and policy importance. Jama, 2006. 296(10): p. 1286-9.

20. Policy Implications • Rigorous and routine exploratory MTE analysis is necessary and should be encouraged • Standards are essential for developing practice guidelines that are appropriate to the needs of complex patients

21. University of California, Davis Richard Kravitz, MD, MSPH - PI Elizabeth Yakes, MS University of California, Los Angeles Naihua Duan, PhD - PI Diana Liao, MPH Kiavash Nikkhou Research Team

22. Thank you