HSRP 734: Advanced Statistical Methods June 12, 2008

1. HSRP 734: Advanced Statistical MethodsJune 12, 2008

2. General Considerations for Multivariable Analyses • Variable Selection • Residuals • Influence diagnostics • Multicollinearity

3. (2) Candidate Model Selection (1) Preliminary Analysis (4) Collinearity and Influential Observation Detection (3) Assumption Validation Yes (6) Prediction Testing (5) Model Revision No An Effective Modeling Cycle

4. Overview • Model building: applies outside of Logistic regression • Model diagnostics: specific to Logistic regression

5. Model Building

6. Model selection • “Proper model selection rejects a model that is far from reality and attempts to identify a model in which the error of approximation and the error due to random fluctuations are well balanced.” - Shibata, 1989

7. Model building • Models are just that: approximating models of a truth • How best to quantify approximation? • Depends upon study goals (prediction, explanatory, exploratory)

8. Principle of Parsimony • “Everything should be made as simple as possible, but no simpler.” – Albert Einstein • Choose a model with “the smallest # of paramters for adequate representation of the data.” – Box & Jenkins

9. Principle of Parsimony • Bias vs. Variance trade-off as # of variables/parameters increases • Collect sample to learn about population (make inference) • Models are just that: approximating models of a truth • Balance errors of underfitting and overfitting

10. Why include multiple predictors in a model? • Interaction (effect modification) • Confounding • Increase precision (reduce unexplained variance) • Method of adjustment • Exploratory for unknown correlates

11. Interpreting Coefficients • When you have more than 1 variable in the model the interpretation is different • Continuous: “β1: For a unit change in X, there is a β1 change in Y, adjusting for the other variables in the model.”

12. Relationship between Variables Exposure (X) Disease (Y) Third Variable (Z) • Two main complications: • Confounding • Interaction (Effect Modification) Bias Useful Information

13. Interaction vs. Confounding • Confounding is a BIAS we want to REMOVE • Interaction is a PROPERTY we want to UNDERSTAND • Confounding • Apparent relationship of X (exposure of interest) with Y is distorted due to the relationship of Z (confounder) with X (and Y) • Interaction • Relationship between X and Y differs by the level of Z (when X and Z interact)

14. Model building • Science vs. Art • Different philosophies • Some agreement on what is worse • Not many agree on a best approach

15. Model building: Two approaches • Data-based approach • Non-data based

16. How do you decide what predictor variables to include? Well, what is the goal?

17. Selecting Predictor Variables • Different goals of analyses: • Estimate and test a treatment group effect • Explore which of a set of predictors are associated with an outcome • Maximize the variation explained/Best predict an outcome

18. Rule of Model Parsimony • Include just enough variables and no more. • Use a smaller number of variables if they accomplish the same goal (about the same c statisticor precision in treatment effect)

19. Variable Selection • Mechanics: • Automatic selection based on p-values • Select based on AIC or BIC • Select based on predictive ability • Select based on theoretical or prior literature considerations • Select based on changes in treatment group effects (confounding, interaction, precision)

20. Data-based: Using p-values • Popular (Remember Johnny from Cobra Kai?) • Selection methods: Forward, Backwards, Stepwise • Bivariate screening, then multivariable on those initially significant

21. Automatic Selection • Select predictor variables based on p-value cutoff rules • Cutoff rules aren’t necessarily considered at p<0.05 • Three types: Forward, Backwards, Stepwise

22. Forward Selection • Start off with no predictors in the model • First, add in the most significant variable with p < pcutoff • Next, add in the most significant variable with p < pcutoff , given a model with the 1st variable in the model • Stop when no addition variables have p < pcutoff

23. Backwards Elimination • Start off with all the predictors in the model • First, remove the least significant variable with p > pcutoff • Next, remove the least significant variable with p > pcutoff , given a reduced model with the 1st variable out of the model • Stop when no addition variables have p > pcutoff

24. Stepwise Selection • Start with no predictors in the model • 1st step: Proceed with Forward Selection • 2nd step: Add in the most significant variable with p<pFcutoff or remove the least significant variable in the model with p>pBcutoff • Continue until there are no more predictors with p<pFcutoff or p>pBcutoff

25. Criticisms of P-value based Model Building • Does not incorporate thinking into the problem/automates • Multiple comparisons issue • If multicollinearity is present, selection is made arbitrarily • β’s, SEβ’s are biased (Harrell Jr., 2001) • Test statistics don’t have right distribution (Grambsch, O’Brien, 1991)

26. Selection methods using p-values • If using these methods there is some preference given to Backwards elimination selection • Some evidence of performing better than Forward selection (Mantel, 1970) • At least initial full model is accurate

27. Non P-value based Methods • Theoretical Considerations • Prior Literature Considerations • Information Criteria: AIC, BIC

28. Theoretical Considerations • Adjust for theoretically associated predictors, regardless of p-value • One line of logic is, why would you want to examine the association of P with outcome, without adjusting for T?

29. Prior Literature Considerations • Adjust for predictors in prior literature, regardless of p-value • One line of logic is, why would you want to examine the association of P with outcome, without adjusting for L? • Example: Outcome=Survival, P=New treatment, L=Patient age

30. Information Criteria: AIC, BIC • Use non p-value based criteria that maximize the relative fit of competing models • Complex theoretical motivation for IC • Can be used for complicated modeling: non-nested models, functional form of same predictors, etc.

31. Data-based: Using AIC • AIC is unbiased estimator of the theoretical distance of a model to an unknown true mechanism (that actually generated the data)

32. Data-based: Using AIC • AIC is unbiased estimator of the theoretical distance of a model to an unknown true mechanism (that actually generated the data) • How is this so??? • If you are really curious…

33. A Gross Simplification of AIC

34. Data-based: Using AIC • Useful for selecting best model out of candidate model set (not great if all are poor) • The size of 1 AIC value is not important but rather relative size to other AIC’s • Models need not be nested but have same sample size (Burnham & Anderson, 2002)

35. Treatment Effect Approach • Adjust for any and all confounders and effect-modifiers (interactions), regardless of p-values • From theoretical & prior literature • Goal is to get most accurate and precise estimate of treatment effect

36. Model Building for Treatment Effect Goal • If we don’t include confounders or interactions that were important then that could obscure picture of outcome-exposure relationship

37. Still will consider Parsimony • If we include many covariates (not confounders or interactions) perhaps some will only add “noise” to model • Noise added could obscure picture of outcome-exposure relationship

38. Data-based: Prediction goal • When Parsimony matters: find most accurate model that is most parsimonious (smallest # of predictors) • When doesn’t matter: pure accuracy = goal at any cost • Example: Quality control • Plausible but not typical

39. Best Predictive Model Approach • Adjust for any predictors that non-trivially increase c statistic (trivial is subject specific) • P-values are not considered • Goal is to maximize predictive ability of model • Future prediction is utmost; “Manage what you measure”

40. Book on Model building • Chapters 6, 7 • Basically takes the approach of trying to accurately establish the outcome-exposure relationship

41. Book recommendations • Multistage strategy: • Determine variables under study from research literature and/or that are clinically or biologically meaningful • Assess interaction prior to confounding • Assess for confounding • Additional considerations for precision

42. Book recommendations • Use backwards elimination of modeling terms • Retain lower-order terms if higher-orders are significant: • Keep 2 variables if 2-way interaction if significant • Keep lower power terms if highest power is significant

43. Model building • We will focus on treatment effect goal • Will consider book guidelines

44. Note about Model Building • Differences between “Best” model and nearest competitors may be small • Ordering among “Very Good” models may not be robust to independent challenges with new data

45. Note about Model Building • Be careful not to overstate importance of variables included in “Best” model • Remember that “Best” model odds ratios & p-values tend to be biased away from the null • Cross-validation approaches allow estimation of prediction errors associated with variable selection and also provide comparisons between sets of best models

46. SAS Lab: ICW

47. Model Diagnostics

48. After selecting a model • Want to check modeling fit and diagnostics to ensure adequacy • Could be worried about: • Influential data points • Correlated predictor variables • Leaving out variables or using wrong form • Overall model fit and prediction value

49. Problems to check for • Convergence problems • Model goodness-of-fit • Functional form (confounding, interaction, higher order for continuous) • Multicollinearity • Outlier effects

50. Convergence problems • SAS usually converges but sometimes will get a message: “There is possibly a quasicomplete separation in the sample points. The ML estimate may not exist. Validity of the model fit is questionable.”