Machine Learning for Infant Sepsis Prediction: Clinical Insights and Model Performance

1. Machine Learning for Infant Sepsis Prediction Patient Engagement, Communication, and Education S96 Aaron J. Masino, PhD Perelman School of Medicine at the University of Pennsylvania Children’s Hospital of Philadelphia Twitter: @aaronmasino #AMIA2018

2. Disclosure • I and my spouse have no relevant relationships with commercial interests to disclose. AMIA 2018 | amia.org

3. Learning Objectives • After participating in this session the learner should be better able to: • Identify features that enable a machine learning model to identify infant sepsis • Describe the use of learning curves to assess model bias and variance in machine learning models AMIA 2018 | amia.org

4. Motivation & Objective • Premature infant sepsis: • 7-28% mortality • 30-35% of survivors incur serious impairments • Early recognition & treatment is associated with improved outcomes • Clinical heterogeneity delays recognition • Identify sepsis in NICU patients at least 4 hours prior to sepsis workup order Clinical sepsis workup (blood culture) t0 Prediction data t-4 Goal: high performance prediction t-48 AMIA 2018 | amia.org

5. Methods / Study Setting • Retrospective study of infants hospitalized in Children’s Hospital of Philadelphia NICU (September 2014 – November 2017) • Data derived from CHOP NICU sepsis registry – automatic extraction from CHOP EHR (Epic Systems, Inc.) for any individual receiving a sepsis workup • 618 unique individuals with 1,188 sepsis evaluations • 110 Culture Positive: positive blood culture for known bacterial pathogen • 265 Clinically Positive: negative blood culture, antibiotic treatment ≥ 120 hours • Remaining evaluations were negative or indeterminate AMIA 2018 | amia.org

6. Methods / Case & Control Episode Selection • Episode – any 44 hour period during NICU hospitalization • Case episodes (365) • Culture positive (110) & clinically positive (265) • Data from episode ending 4 hours prior to culture blood draw • Control episodes (1,100) • All infants in study allowed as potential control episodes • Episodes randomly selected from periods least ten days removed from any evaluation AMIA 2018 | amia.org

7. Methods / Machine Learning / Model Design • Goal: develop a classifier that can differentiate case & control episodes • 8 ML methods considered / 2 models per method (CPOnly & CP+Clinical) • Mean imputation of missing values • Mutual information feature selection AMIA 2017 | amia.org

8. Methods / Machine Learning / Features • Feature Categories (36 features) • Clinical assessments (3) • Comorbidities (5) • Indwelling lines (2) • Respiratory support (2) • Vital signs • Most recent value (10) • Difference from 24 hour mean (4) • Threshold (2) • Laboratory tests (8) AMIA 2017 | amia.org

9. Results / Feature Analysis AMIA 2017 | amia.org

10. Results / SVM Performance (CPOnly) *Value at 0.80 sensitivity **Over 10 validation folds AMIA 2017 | amia.org

11. Results / SVM Performance (CP+Clinical) *Value at 0.80 sensitivity **Over 10 validation folds AMIA 2017 | amia.org

12. Results / Learning Curves AMIA PowerPoint Template

13. Conclusion • Machine learning may identify sepsis in infants prior to clinical recognition, but … • Limitations • Small dataset • Missing (i.e. unobserved) data model bias – particularly labs • Difficult to assess prediction certainty • Difficult to assess identify relevant features for a given prediction • Future work • Collect more data (isn’t this always the answer?) • Bayesian generative modeling • Pilot clinical deployment and evaluation AMIA 2017 | amia.org

14. Thank you! Email me at: masinoa@email.chop.edu