Physician Prognostic Accuracy for In-Hospital Mortality in Percutaneous Coronary Intervention

1. Physician Prognostic Accuracy for In-Hospital Mortality in Percutaneous Coronary Intervention Michael E. Matheny, MD Medical Informatics Fellow Decision Systems Group Brigham & Women’s Hospital Boston, MA

2. Specific Aims • Primary Hypothesis • Accuracy of subjective physician estimations of in-hospital mortality will be similar or improved when compared with accepted objective risk assessment methods for percutaneous coronary intervention (PCI)

3. Specific Aims • Secondary Hypotheses • Accuracy of subjective physician estimations of in-hospital major adverse cardiac events (MACE) will be similar or improved when compared with accepted objective risk assessment methods for PCI • Qualitative collection of risk factors could identify additional important risk factors currently not included in the objective risk models • Incorporating subjective physician estimates into an objective risk model will outperform either separately

4. Background • Discrimination • Ability to predict an outcome on a population level • Area under the Receiver Operating Characteristic Curve (AUC)

5. Background • Calibration • Ability to predict an outcome on a case/small group level • Hosmer-Lemeshow Goodness-of-Fit Test (HL-GF) • Brier Score

6. Background • Subjective vs APACHE II Medical ICU Mortality 1 • Discrimination: Objective Better • Calibration: Subjective Better • Forecasting Improves with Training • Subjective vs APACHE II Medical ICU Mortality 2 • Discrimination: Subjective Better • Calibration: No Difference

7. Background • Subjective vs LR Acute CHF 90 day and 1 year Mortality 3 4 • Discrimination: No Difference • Calibration: No Difference • All estimations poor • Subjective vs SNAP Neonatal ICU Mortality 5 • Discrimination: No Difference • Calibration: No Difference

8. Background • Subjective + PRISM III Pediatric ICU Mortality 6 • Discrimination: No Difference • Calibration: No Difference • Combined model • Discrimination: Improved from either • Calibration: Improved from either

9. Background • Subjective vs LR Model of Post-Op mortality for Open Heart Surgeries 7 • Discrimination: No Difference • Calibration: No Difference • Combined model • Discrimination: No Difference • Calibration: No Difference • Subjective assessments were more calibrated at the extremes of probability

10. Background • Subjective Physician Assessments • Multiple Forms of Bias 8 • Ego Bias • Regret • Ignoring Negative Evidence • Framing

11. Background • No work has been done evaluating subjective physician estimates for in-hospital mortality in percutaneous coronary interventions.

12. BackgroundPCI Objective Risk Model Gold Standards • Logistic Regression Models • National • American College of Cardiology 9 • 50123 pts 1998 - 2000 • Regional • Northern New England 10 • 15331 pts 1994 - 1996 • Local • Brigham & Women’s Hospital 11 • 2804 pts 1997 - 1999

13. BackgroundPilot Data • Recent Evaluation of Models on Local Institution Data 12 • Discrimination (AUC) • ACC 0.90 • NNE 0.89 • BWH 0.89 • Calibration (HL-GF) • ACC <0.001 • NNE <0.001 • BWH <0.001

14. Background • Objective Assessment Models • Multiple Forms of Bias • Population/Demographic Bias • Regional Variances • Selection Bias • Population referral bias • Temporal Bias • Medical Care Standards • Data Documentation • Data Noise • Heterogeneous Data Standards • Variation in Data Element Collection • Data Entry Quality Variations

15. Background • Incomplete model information? • Best Described Risk Factors

16. StudyDesign • Prospective Cohort Study

17. StudyPopulation • Location • Brigham & Women’s Interventional Cardiology Suites • Inclusion Criteria • All Patients presenting for pre-operative evaluation for PCI • Exclusion Criteria • Procedural Team declines to participate in survey

18. StudyData Collection • Paper Survey • Administration refused to allow survey to be part of medical record • Subjective mortality assessment (0-100%) before and after procedure • Attendings • Fellows • Scrub Nurse • Qualitative additional risk factors from Attendings

19. Exposures • Percutaneous Coronary Transluminal Angiography with or without Coronary Stenting

20. Covariates/Confounders

21. Outcomes • In-Hospital Death • In-Hospital MACE

22. Analysis Plan • Measure Discrimination & Calibration on local data for: • Objective MACE & Mortality Models • National • Regional • Local • Subjective MACE & Mortality “Models” • Pair-wise Comparison of Objective and Subjective models for statistical differences • Develop LR model with subjective data as a covariate, and perform pair-wise comparisons with objective and subjective models to determine if new model shows improvement

23. Analysis Plan • Local Institution Data • ~1% Death Rate • ~5% MACE Rate • ~200 Cases / month

24. Analysis PlanRough Guess • Sample Size Calc • Binomial Fisher’s Exact • Α = 0.05 • Power = 0.80 • Effect Size & Estimated Sample • Mortality • 1% to 1.5% = 8150 • 1% to 2% = 2514 • MACE • 5% to 7.5% = 1550 • 5% to 10% = 473

25. Analysis PlanRecruitment • Multi-Center • Exploring recruitment possibilities from Beth-Israel and Massachusetts General Cath Labs • No Termination Date • Implemented as Quality Control method in BWH Cath Lab

26. Limitations • Sample Size • Paper Survey • Study Population Compliance • Selection Bias

27. Time Table • IRB Approval • Completed • Physician Survey Template • August 2005 • IC Lab Tech Data Collection Training • September 2005 • Data Collection • September 2005 - Open

28. Acknowledgements • Co-Authors • Nipun Arora, MD • Lucila Ohno-Machado, MD, PhD • Frederic S. Resnic, MD, MS • Funding • NLM 1-T15-LM-07092

29. The End mmatheny@dsg.harvard.eduMichael Matheny, MD Brigham & Women’s HospitalThorn 30975 Francis StreetBoston, MA 02115