Problems with Statistical Predictions and Decision-Making Approaches

5. Why Most Statistical Predictions Cannot Reliably Support Decision-Making: Problems Caused By Common RegressionModeling Approaches Ewout Steyerberg Professor of Clinical Biostatistics and Medical Decision Making Leiden, October 2017

6. Why most predictionmodels are false • Methods at development • Rigorous validation

7. Validation

8. Three competitive models • MMRPredict(NEJM, 2006) - Common regressionmodeling approach; small data set • MMRPro(JAMA 2006)- Bayesian modeling approach; moderate size data set • PREMM (JAMA 2006; Gastroenterology 2011; JCO 2016)- Sensible regression modeling approach; large data set Which model wins? Which may do harm?

9. 6 clinic-based, 5 pop-based cohorts

10. Discrimination Clinic-based Population-based

11. Calibration plots: obs vs predicted • Calibration slope as a measure of overfitting

12. Calibration

13. Calibration

14. Clinical usefulness • Statistical performance: Discrimination and calibration • Consider full range of predictions • Decision-analytic performance: • Define a decision threshold: act if risk > threshold • TP and FP classifications • Net Benefit as a summary measure:NB = (TP – w FP) / n, with w = harm/benefit(Vickers & Elkin, MDM 2006)

15. Decision curve analysis Clinic-based Population-based

16. Overview • Clinical context: testing for Lynch syndrome • Statistical and decision-analytic performance • Could poor performance have been foreseen? Prevented?

17. Example of “barbarian modeling strategy”

18. Selection based on statistical significance

19. Many predictors, >37 df; dichotomized

20. Exaggeratedeffects

21. Sample size issues Robust: strong, vigorous, sturdy, tough, powerful, powerfully built, solidly built, as strong as a horse/ox, muscular, sinewy, rugged, hardy, strapping, brawny, burly, husky

22. Poor performance foreseeable? • Simulate modeling strategy • Small sample size • 38 events at development • 35 events vs >2000 at validation • Stepwise selection • Univariate and multivariable statistical testing • Dichotomization • New cohort: n=19,866; 2,051 mutations

23. Poor calibration Poor discrimination

24. Poor decision-makingIllustration with 10 random samples

25. Could poor performance be prevented? • PREMM modeling strategy • Coding of family history • Continuous age

26. SiM 2007

27. Could poor performance be prevented? • PREMM modeling strategy • Coding of family history • Continuous age • Larger sample size

28. Better discrimination and calibration if a) more sensible modeling and b) larger sample size

29. Substantially better decision-making if a) more sensible modeling and b) larger sample size

30. Discussion • Avoid stepwise selection • Prespecification with summary variables • Advanced estimation • Avoid dichotomization • Keep continuous • Increase sample size • Combining development and validation sets • Collaborative efforts • Rigorous validation • Statistical and decision-analytic perspective

32. Evaluation of decision-making • Net Benefit: “utility of the method” • Peirce, Science 1884 • Youden index: sens + spec – 1 • Net Benefit • Vickers, MDM 2006 • Weight FP:TP = H:B = odds(threshold) (Vergouwe 2003) • Decision Curve Analysis

34. Youden index and Net Benefit

35. Avoid miscalibration by overfitting • ShrinkageReduce coefficients by multiplying by s, s<1E.g.: multiply by 0.8 • PenalizationRidge regression: shrink during fittingLASSO: shrink to zero; implicit selectionElastic Net: combination of Ridge and LASSO • Machine learning ?