SC-PM6: Prediction Models in Medicine: Development, Evaluation and Implementation

1. SC-PM6: Prediction Models in Medicine: Development, Evaluation and Implementation Michael W. Kattan, Ph.D. Ewout Steyerberg, Ph.D. Brian Wells, M.S., M.D.

2. When The Patient Wants A Prediction, What Options Does The Clinician Have? • Quote an overall average to all patients • Predict based on knowledge and experience • Assign the patient to a risk group, i.e. high, intermediate, or low • Deny ability to predict at the individual patient level • Apply a model

3. How do we typically compute risk? • Based on features, we make a crude tree. • Most cancer staging systems do this. BT=high N Y H=Agg And DE=E Y HIGH RISK LOW RISK N

4. The problem with crude trees • They are very easy to use. • But they do not predict outcome optimally. • High risk groups are very heterogeneous. • A single risk factor may qualify a patient as high risk. • Other approaches, like a Cox regression statistical model, predict more accurately. • Readily applicable presentation essential.

5. Preoperative Nomogram for Prostate Cancer Recurrence 0 10 20 30 40 50 60 70 80 90 100 Points PSA 4 20 0.1 1 2 3 6 7 8 9 10 12 16 30 45 70 110 T2a T2c T3a ClinicalStage T1c T1ab T2b  2+3  4+ ? 3+  2 Biopsy Gleason Grade  2+  2 3+3  3+ 4 Total Points 0 20 40 60 80 100 120 140 160 180 200 60MonthRec. Free Prob. .96 .93 .9 .85 .8 .7 .6 .5 .4 .3 .2 .1 .05 Instructions for Physician: Locate the patient’s PSA on the PSA axis. Draw a line straight upwards to the Points axis to determine how many points towards recurrence the patient receives for his PSA. Repeat this process for the Clinical Stage and Biopsy Gleason Sum axes, each time drawing straight upward to the Points axis. Sum the points achieved for each predictor and locate this sum on the Total Points axis. Draw a line straight down to find the patient’s probability of remaining recurrence free for 60 months assuming he does not die of another cause first. Instruction to Patient: “Mr. X, if we had 100 men exactly like you, we would expect between <predicted percentage from nomogram - 10%> and <predicted percentage + 10%> to remain free of their disease at 5 years following radical prostatectomy, and recurrence after 5 years is very rare.” • 1997 Michael W. Kattan and Peter T. Scardino Kattan MW et al: JNCI 1998; 90:766-771.

6. Some simple steps that will make a difference • Build the most accurate model possible. • Take model to bedside • As a nomogram, • In stand-alone software (desktop, handheld, web) • Built into the electronic medical record • Doing this will predict patient outcome more accurately, resulting in • better patient counseling • better treatment decision making

7. What is a Nomogram? • A prediction device • Usually a regression model presented graphically • Continuous-based prediction

8. Making a nomogram • Usually a regression model (Cox or logistic) • Could consider machine learning techniques (neural nets, optimized trees like CART) • Keep continuous variables continuous but relax linearity assumptions • P-values for predictors don’t matter • No variable selection or univariable screening • Bottom line is its predictive accuracy

9. CaPSURE Heterogeneity within Risk Groups Nomogram Values by Prostate Cancer Risk Group 1.0 0.9 0.8 0.7 Preoperative NomogramPredicted Probability 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Low Intermediate High Risk Group J Urol. 2005 Apr;173(4):1126-31

10. Kattan MW, et al., J Clin. Oncol., 2000.

11. 3D Conformal Radiation Therapy Nomogram for PSA Recurrence 50 10 T3c T2c 3 5 2 4 6 Kattan MW, et al., J Clin. Oncol., 2000.

12. Discrimination on Cleveland Clinic data (N=912) 0.5 0.55 0.6 0.65 0.7 0.75 0.8 Kattan MW et al., J Clin. Oncol., 2000.

13. Why Nomograms Matter: A Particular Example • Mr. X, from the Cleveland Clinic: • PSA=6, clinical stage = T2c, biopsy Gleason sum=9, planned dose of 66.6 Gy without neoadjuvant hormones • Shipley risk stratification: 81% @ 5 yr. • Surgery nomogram: 68% @ 5yr. • Radiation therapy nomogram: 24% @ 5yr. Kattan MW, et al., J Clin. Oncol., 2000.

14. When The Patient Wants A Prediction, What Options Does The Clinician Have? • Quote an overall average to all patients • Predict based on knowledge and experience • Assign the patient to a risk group, i.e. high, intermediate, or low • Deny ability to predict at the individual patient level • Apply a model

15. Urologists vs. Preoperative Nomogram • 10 case descriptions from 1994 MSKCC patients presented to 17 urologists • In addition to PSA, biopsy Gleason grades, and clinical stage, urologists were provided with patient age, systematic biopsy details, previous biopsy results, and PSA history. • Preoperative nomogram was provided. • Urologists were asked to make their own predictions of 5 year progression-free probabilities with or without use of the preoperative nomogram. • Concordance indices: • Nomogram = 0.67 • Urologists = 0.55, p<0.05 Ross P et al., Semin Urol Oncol, 2002.

16. Nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy Vanzee K, et al., Ann Surg Oncol., 2003.

17. Breast Cancer Prediction: 17 Clinicians vs. Nomogram on 33 Patients Nomogram AUC 0.72 Sensitivity: Proportion of women withpositive nodespredicted tohave positivenodes Specificity: Proportion ofwomen withnegative nodespredicted to havenegative nodes Clinician AUC 0.54

18. ROC CurvesIndividual Clinicians and Nomogram Areas 0.75 0.72 Nomogram 0.68 0.65 0.65 0.63 0.59 0.58 0.55 0.55 0.53 0.52 0.50 0.49 0.47 0.43 0.42 0.40

20. Nomogram to Predict Bone Scan Positivity (Cont.) Nonogram Used to Predict Patient-Specific Probabilities of Metastasis-free Survival at 1 and 2 Years, and the Median Progression-free Survival Time Points bPSA, ng/mL 0 10 20 30 40 50 60 70 80 90 100 3+ PSADT, mo 1-2 T Stage AUC=0.69 7 Gleason 2 12 10 8 6 4 0 0.37 1.0 2.7 7.4 20 55 150 245 0 20 40 60 80 100 120 140 160 180 Total Points 1-Year PFS 0.9 0.7 0.5 0.3 0.1 48 36 24 12 6 0.8 0.7 0.5 0.3 0.1 6 8-9 2-Year PFS Median PFS Slovin SF, et al. Clin Can Res. 2005;11:8669-8673.

21. Predictions by Docs and Nomogram, by Patient

22. Survey Results

23. Survey Results

24. Discrimination accuracy for nomogram and docs

25. Biases in Human Prediction Feedback recall, overconfident, hindsight bias, chance adapted from Hogarth, 1988

26. Conclusions for Part 1 • Continuous statistical prediction models offer accuracy advantages over: • Crude risk groups • Clinical judgment • Remaining challenges: • How to evaluate accuracy • If accuracy is acceptable, how to deploy/disseminate