Estimating the Expected Warning Time of Outbreak-Detection Algorithms

1. Estimating the Expected Warning Time of Outbreak-Detection Algorithms Yanna Shen, Weng-Keen Wong, Gregory F. Cooper RODS Laboratory, Center of Biomedical Informatics, University of Pittsburgh 2005 Syndromic Surveillance

2. Overview • Objective • Background • Methods • Experimental Results • Conclusions • Future Work 2005 Syndromic Surveillance

3. Objective • A new measure for evaluating alerting algorithms, which is called Expected Warning Time (EWT). • It is a generalization of the standard AMOC curve. 2005 Syndromic Surveillance

4. Why Useful? • Can compare expected clinician detection time to expected computer-based algorithm detection time • Can provide a promising new approach for optimizing and comparing outbreak detection algorithms 2005 Syndromic Surveillance

5. Incubation Time Computer Detection Time Warning Time Computers raise alert Clinicians detect outbreak Last outbreak case appears False Alerts (in red) Maximum meaningful WT Maximum meaningful detection time Background hit hit hit hit … … … Time Release occurs 2005 Syndromic Surveillance

6. Model • A simple model of clinician outbreak detection • Assumes that • People with disease D are diagnosed independently of each other. • The probability of a person with disease D being diagnosed is constant (p). 2005 Syndromic Surveillance

7. Equation Definitions: • p – probability that a person with D is diagnosed as having D upon presentation with that disease • time(i) – maps patient case i to the time at which that patient presented with D to clinicians • M – total # of patient cases with D • t – time at which the alerting score first exceeds a given threshold 2005 Syndromic Surveillance

8. Equation x x + Probability that clinicians will detect the outbreak on the ith case WT if clinicians never detect the outbreak WT if clinicians first detect the outbreak on the ith case Probability that clinicians will never detect the outbreak 2005 Syndromic Surveillance

9. Experiment Setup • Apply PANDA to simulated cases of inhalational anthrax • For various value of p, derive EWT for PANDA 2005 Syndromic Surveillance

10. PANDA • An outbreak detection system • Uses causal Bayesian networks to model spatio-temporal patterns of a non-contagious disease in a population (Cooper, 2004) • Contains a model to detect inhalational anthrax 2005 Syndromic Surveillance

11. BARD • BARD simulator produces the simulated cases of anthrax. • It models the effects of an outdoor airborne anthrax release using the Gaussian plume model of atmospheric dispersion and a model of inhalational anthrax (Hogan, 2004). 2005 Syndromic Surveillance

12. Performance of Clinicians • p – Clinician detection proficiency • P(CD) – Probability that clinicians will detect the outbreak at all • ECDT – Expected clinician detection time given that clinicians detect the outbreak 2005 Syndromic Surveillance

13. ? Experimental Results • p increases, EWT decreases • p = 1, EWT = 0 2005 Syndromic Surveillance

14. Experimental Results False alert rate = 1 per month • Ifand false alert rate = 1 per month, then EWT minutes. 2005 Syndromic Surveillance

15. Conclusions • The Expected Warning Time (EWT) is a useful concept for evaluating outbreak-detection algorithms. • We illustrated the general idea of EWT using a simple model of clinician detection and simulated cases of inhalational anthrax. • Our example analysis suggests that PANDA is most helpful when clinicians’ detection proficiency < 5%. 2005 Syndromic Surveillance

16. Future Work • Extend the model: • Instead of a constant (p), use the function p(t), where t is time • Develop and apply more disease-specific models of clinician detection (please see the poster by Christina Adamou) 2005 Syndromic Surveillance

17. Acknowledgements • This research was supported by grants from the National Science Foundation (IIS-0325581), the Department of Homeland Security (F30602-01-2-0550), and the Pennsylvania Department of Health (ME-01-737). • We thank members of the Bayesian Biosurveillance Project for helpful comments. 2005 Syndromic Surveillance