Variable Selection for Tailoring Treatment

1. Variable Selection for Tailoring Treatment L. Gunter, J. Zhu & S.A. Murphy ASA, Nov 11, 2008

2. Outline • Motivation • Need for Variable Selection • Characteristics of a Tailoring Variable • A New Technique for Finding Tailoring Variables • Comparisons • Discussion

3. Motivating Example

4. Simple Example Nefazodone - CBASP Trial Nefazodone Randomization Nefazodone + Cognitive Behavioral Analysis System of Psychotherapy (CBASP) 50+ baseline covariates, both categorical and continuous

5. Simple Example Nefazodone - CBASP Trial Which variables in X are important for tailoring the treatment?

6. Optimization • We want to select the treatment that “optimizes” R • The optimal choice of treatment may depend on X

7. Optimization • The optimal treatment(s) is given by • The value of d is

8. Need for Variable Selection • In clinical trials many pretreatment variables are collected to improve understanding and inform future treatment • Yet in clinical practice, only the most informative variables for tailoring treatment can be collected. • A combination of theory, clinical experience and statistical variable selection methods can be used to determine which variables are important.

9. Current Statistical Variable Selection Methods • Current statistical variable selection methods focus on finding good predictors of the response • Also need variables to help determine which treatment is best for which types of patients, e.g. tailoring variables • Experts typically have knowledge on which variables are good predictors, but intuition about tailoring variables is often lacking

10. What is a Tailoring Variable? • Tailoring variables help us determine which treatment is best • Tailoring variables qualitatively interact with the treatment; different values of the tailoring variable result in different best treatments. No Interaction Non-qualitative Interaction Qualitative interaction

11. Qualitative Interactions • Qualitative interactions have been discussed by many within stat literature (e.g. Byar & Corle,1977; Peto, 1982; Shuster & Van Eys, 1983; Gail & Simon, 1985; Yusuf et al., 1991; Senn, 2001; Lagakos, 2001) • Many express skepticism concerning validity of qualitative interactions when found in studies • Our approach for finding qualitative interactions should be robust to finding spurious results

12. Qualitative Interactions • We focus on two important factors • The magnitude of the interaction between the variable and the treatment indicator • The proportionof patients for whom the best choice of treatment changes given knowledge of the variable big interaction small interaction big interaction big proportion big proportion small proportion

13. Ranking Score S • Ranking Score: where • S estimates the quantity described by Parmigiani (2002) as the value of information.

14. Ranking Score S • Higher Sscorescorrespond to higher evidence of a qualitative interaction between X and A • We use this ranking in a variable selection algorithm to select important tailoring variables. • Avoid over-fitting in due to large number of X variables • Consider variables jointly

15. Variable Selection Algorithm • Select important predictors of R from (X, X*A) using Lasso -- Select tuning parameter using BIC • Select all X*A variables with nonzero S. -- Use predictors from 1. to form linear regression estimator of to form S. (using linear models)

16. Lasso • Lasso on (X, A, XA) (Tibshirani, 1996) • Lasso minimization criterion: where Zi is the vector of predictors for patient i, λ is a penalty parameter • Coefficient for A not penalized • Value of λ chosen by Bayesian Information Criterion (BIC) (Zou, Hastie & Tibshirani, 2007)

17. Variable Selection Algorithm • Rank order (X, X*A)variables selected in steps 1 & 2 using a weighted Lasso -- Weight is 1 if variable is not an interaction -- Otherwise weight for kth interaction is -- is a small positive number. -- Produces a combined ranking of the selected (X, X*A)variables (say p variables).

18. Variable Selection Algorithm • Choose between variable subsets using a criterion that trades off maximal value of information and complexity. -- The ordering of the p variables creates p subsets of variables. Estimate the value of information for each of the p subsets -- Select the subset, k with largest

19. Simulations • Data simulated under wide variety of realistic decision making scenarios (with and without qualitative interactions) • Used X from the CBASP study, generated new Aand R • Compared: • New method: S with variable selection algorithm • Standard method: BIC Lasso on (X, A, XA) • 1000 simulated data sets: recorded percentage of time each variable’s interaction with treatment was selected for each method

20. Simulation Results * Over the total possible increase; 1000 data sets each of size 440

21. Simulation Results • Pros: when the model contained qualitative interactions, the new method gave significant increases in expected response over BIC-Lasso • Cons: the new method resulted in a slight increase in the number of spurious interactions over BIC-Lasso

22. Nefazodone - CBASP Trial Aim of the Nefazodone CBASP trial – to compare efficacy of three alternate treatments for major depressive disorder (MDD): • Nefazodone, • Cognitive behavioral-analysis system of psychotherapy (CBASP) • Nefazodone + CBASP Which variables might help tailor the depression treatment to each patient?

23. Nefazodone - CBASP Trial • For our analysis we used data from 440 patients with

24. Method Application and Confidence Measures • When applying new method to real data it is desirable to have a measure of reliability and to control family-wise error rate • We used bootstrap sampling to assess reliability • On each of 1000 bootstrap samples: • Run variable selection method • Record the interaction variables selected • Calculate selection percentages over bootstrap samples

25. Error Rate Thresholds • To help control family-wise error rate, compute the following inclusion thresholdsfor selection percentages: • Repeat 100 times • Permute interactions to remove effects from the data • Run method on 1000 bootstrap samples of permuted data • Calculate selection percentages over bootstrap samples • Record largest selection percentage over the p interactions • Threshold: (1-α)th percentile over 100 max selection percentages • Select all interactions with selection percentage greater than threshold

26. Error Rate Thresholds • When tested in simulations using new method, error rate threshold effectively controlled family-wise error rate • This augmentation of bootstrap sampling and thresholding was also tested on BIC Lasso and effectively controlled family-wise error rate in simulations

27. Nefazodone - CBASP Trial ALC OCD ALC OCD

28. Interaction Plot

29. Interaction Plot

30. Discussion • This method provides a list of potential tailoring variables while reducing the number of false leads. • Replication is required to confirm the usefulness of a tailoring variable. • Our long term goal is to generalize this method so that it can be used with data from Sequential, Multiple Assignment, Randomized Trials as illustrated by STAR*D.

31. Email Susan Murphy at samurphy@umich.edu for more information! • This seminar can be found at http://www.stat.lsa.umich.edu/~samurphy/seminars/ ASA11.11.08.ppt • Support: NIDA P50 DA10075, NIMH R01 MH080015 and NSF DMS 0505432 • Thanks for technical and data support go to • A. John Rush, MD, Betty Jo Hay Chair in Mental Health at the University of Texas Southwestern Medical Center, Dallas • Martin Keller and the investigators who conducted the trial `A Comparison of Nefazodone, the Cognitive Behavioral-analysis System of Psychotherapy, and Their Combination for Treatment of Chronic Depression’

32. Interaction Plot

33. Interaction Plot

34. Lasso Weighting Scheme • Lasso minimization criterion equivalent to: so smaller wj means greater importance • Weights where • vj = 1for predictive variables • vj = for prescriptive variables

35. AGV Criterion • For a subset of k variables, X{k} the Average Gain in Value ( AGV) criterion is where • The criterion selects the subset of variables with the maximum proportion of increase in E[R] per variable

36. Simulation Results (S-score) ×Qualitative Interaction Spurious Interaction ×Qualitative Interaction Non-qualitative Interaction Spurious Interaction