Prognostic models in the ICU

1. Prognostic models in the ICU From development to clinical practice L. Minne, MSc. Dr. S. Eslami, PharmD Dr. D.A. Dongelmans, MD Prof. Dr. S.E.J.A. de Rooij, MD Prof. Dr. A. Abu-Hanna Dept. of Medical Informatics Dept. of Intensive Care Academic Medical Center Amsterdam, the Netherlands Prof. Dr. E. de Jonge, MD Dept. of Intensive Care Leiden University Medical Center Leiden, the Netherlands

2. Hospital 2 Hospital 1 Use of prognostic models 1) Benchmarking 2)Decision-making Observed mortality: 25% 15% Expected mortality: 30% 12% Estimates from prognostic model SMR: 0.83 1.25

3. Use of prognostic models 1) Benchmarking 2) Decision-making Your probability to survive is: -7.7631 + (SAPS II score * 0.0737) + (0.9971 * (ln (SAPS II score + 1)))

4. Barriers for use in clinical practice • Lack of evidence for: • External validity • Clinical credibility • Impact on decisions and patient outcomes • Selffulfilling prophecy • Population level vs individual level

5. Overview of our research project • Identifyprognostic models, their validity and use in clinical practice • Assess prognostic model behaviour over time + effects on benchmarking • Assess clinicians’ predictions, (need for) prognostic models, their validity and impact in decision-making

6. Red (critical) zone > mean + 4 sigma Yellow (warning) zone mean + 2 sigma : mean + 4 sigma Green (safe) zone mean : mean + 2 sigma Green (safe) zone mean : mean - 2 sigma mean - 2 sigma : mean - 4 sigma Yellow (warning) zone Red (critical) zone < mean - 4 sigma Benchmarking – Temporal validation Upper control limit (usually at 3 sigma) Mean value Standardized Mortality Ratio Lower control limit (usually at 3 sigma) Time

7. Benchmarking – Temporal validation SMR > 1 in 15% of the hospitals

8. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Benchmarking – Temporal validation Effect of continuous updating (first level recalibration) Data used for recalibration Data used for recalibration p=16 p=19 Time

9. Benchmarking – Temporal validation SMR > 1 in 35% of the hospitals  effect on quality of care assessment!

10. SAPS score Decision-making – Model development Demography Admission Age Gender ... Physiology Laboratory ... Outcomes Mortality (Length of Stay) (...) During Stay organ scores day1 organ scores day2 organ scores day3 … SOFA

11. 4 4 1 0 3 3 3 3 0 0 1 2 M H H H H Decision-making – Model development 25 SAPS Day 1 Day 2 Day 3 Day 4 Day 5 1 1 3 Renal 0 0 0 Hepatic 3 4 3 Circulatory 4 4 3 Respiratory 0 0 0 Neurological 0 0 0 Coagulation 8 9 9 12 12 SOFA score d = 3

12. Decision-making – Model development LP = a0 + a1SAPS + a2admission_type + a3day + A4number_of_readmissions + + b1 Pattern1 + b2 Pattern2 + … Example at day 3 LP = -9.3 +0.005*SAPS -0.034*3 + 1.23*2 + 1.85 SOFA{H,H} + 1.1 SOFA{M,H,H}

13. Decision-making – Model performance

14. Decision-making – Model performance

15. Decision-making – The end-of-life decision-making process • Observation of multidisciplinary meetings  poorly structured no clear guidelines • Factors (implicitly) considered in decision: • Degree of organ failure • Patient preferences • Severity of illness • Chance of cognitive limitations • Wish to receive objective information

16. Conclusions and future work • Decision-making process unstructured • Possible role for mathematical models • But… insufficient evidence on their impact and external validation • Before-after study to measure impact on decision-making

17. Any questions?

18. Decision-making – Human predictions Kappa = 47.3-55.1%