Overview

1. Using Artificial Intelligence Techniques in Epidemiology ResearchMasoumeh Izadi mtabae@cs.mcgill.caClinical & Health Informatics Research Group McGill University

2. Overview • Motivation • Bio-surveillance systems • Sequential decision making problem • Proposed solution • Evaluation and results • Concluding remarks

3. Motivation • Threat of bioterrorist attacks • October 2002 Anthrax attacks • Threat of infectious disease outbreaks • Influenza • Sudden Acute Respiratory Syndrome • Timely detection and proactive action can save lives.

4. Problem • Current detection methods are inaccurate in practice • Public health professionals take an ad hoc approach in response to these methods. • Critical need for decision support systems!

5. Addressing Uncertainty •  uncertainty in the experts knowledge; •  uncertainty inherent in the domain; •  uncertainty in the knowledge engineering; •  uncertainty as to the accuracy and actual availability.

6. Outbreak Detection in Bio-surveillance Systems

7. The Surveillance Cycle 1. Identifying individual cases 2. Detecting population patterns 3. Conveying information for action Individual Event Definitions Population Pattern Definitions Intervention Guidelines Public Health Action Event Reports Pattern Report Event Detection Algorithm Pattern Detection Algorithm Intervention Decision Data Describing Population Population Under Surveillance (Buckeridge DL & Cadieux G, 2007)

8. Surveillance Research • Achieving the National Electronic Disease Surveillance System (NEDSS) architecture; • New data sources; • Case definitions (automation/validation); • Forecasting; • Evaluation and quality control.

9. Potential Outbreak Scenarios • Large scale bioaerosol (e.g., Anthrax) • Communicable (e.g., SARS) • Waterborne • Building contamination • Food-borne • Sexual/blood borne

10. Data Sources

11. Features Shared by Most Detection Methods • Methods are non-specific – they look for anything unusual in the data • Design a baseline. • Define an aberration when some statistics are more than expected values by the baseline.

12. Example • Define a threshold for the # of ED visits per day . • Signal an alarm when the # of ED visits per day exceeds the threshold.

13. Anthrax Attacks Flu Flu Flu Data courtesy of Medstar & Georgetown University Anthrax Cases in DC

14. Existing Detection Methods • Temporal methods • Spatio-temporal methods

15. Objectives • Precise detection • Early detection Research to address these: • Novel or ‘improved’ data streams • Better forecasts or detection methods • Improve decision making w.r.t. alarms

16. Manual Decision Making in Public Health Surveillance

17. Our Approach Instead of trying to improve the detection method, we ‘post-process’ the signals: • Use a standard detection method to provide signals; • Feed this signal to a decision support model to find the optimal action w.r.t. specific disease.

18. Important Factors • Economical impacts; • Progression of symptoms; • Incubation period; • Spatial dispersion pattern;

19. Anthrax?! • Limited data from actual anthrax attacks available • We can coherently incorporate different types of simulation data • The literature contains studies on the characteristics of inhalational anthrax: • reviews of human cases; • experimental exposures with animals.

20. Reinforcement Learning in Partially Observable Settings

21. Reinforcement Learning Key features: • Learner is not told which action to take. • Learner is goal-oriented and interacts with an uncertain environment. • Trial-and-error search. • Needs to explore and exploit. • Learn a policy by evaluating the quality of an action for a state.

22. Partially Observable Markov Decision Processes • POMDPs are characterized by: • States: sS • Actions: aA • Observations: oO • Transition probabilities: T(s,a,s’)=Pr(s’|s,a) • Observation probabilities: T(o,a,s’)=Pr(o|s,a) • Rewards: R(s,a) • Belief state: b(s)

23. Graphical View st-1 st What goes on: ot-1 ot What we see: at-1 at bt-1 bt What we infer: Joelle Pineau

24. 1 1 P(s1) P(s1) 0 0 P(s2) 1 Understanding the belief space • A belief summarizes the history of actions and observations • Dim(B) = |S|-1 Joelle Pineau

25. Reachable states under p … … … p(b) b … … Behavior Evaluation Use a value estimate at these states

26. Solving POMDPs • To solve a POMDP is to find, for any belief state, the action that maximizes the expected discounted reward. V(b)= max a [Σs R(s,a)b(s)+ γΣs’ [T(s,a,s’)O(s’,a,z)α(s’)]] OUTCOME: an optimal policy over belief space

27. Challenges in Solving POMDPs • Real world problems have large dimensionality • Belief space is a continuous.

28. Approximation Method • Point-based: - Compute policy for a set B of (reachable) points in the belief space; - Expand B; - repeat • Point selection by heuristics.

29. e Heuristics Distance-Based Heuristic • Choose all candidate beliefs “ bc ” such that: (Izadi & Precup, 2006)

30. Feasibility of Using POMDPs in Biosurveillance • The ‘true’ state of the outbreak cannot be observed; • Statistical detection algorithms provide imperfect measurements of the true state; • That probability of success of (i.e., effectiveness) of actions can be determined; • The costs and benefits of actions can be determined

31. POMDP Model Components • S - True epidemic state: {No Outbreak, Day1, ….Day4} • O - Output from detection algorithm: {Alarm, NoAlarm} • A - Possible public health actions: {Do nothing, Review records, Investigate cases, Declare outbreak}

32. The POMDP Model Action Transition Do nothing Review records Investigate cases Declare outbreak

33. Clear Day 1 s Day 2 Day 3 Day 4 Detected Clear D1 D2 D3 D4 Det Transition Functions • The transition functions reflect the probability of moving to another state if an action is performed in each state of the model. T: Review records 0.99 0.01 0.0 0.0 0.0 0.0 0.0 0.0 0.9 0.0 0.0 0.1 0.0 0.0 0.0 0.7 0.0 0.3 0.0 0.0 0.0 0.0 0.5 0.5 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 T: Investigate 0.99 0.01 0.0 0.0 0.0 0.0 0.0 0.0 0.7 0.0 0.0 0.3 0.0 0.0 0.0 0.4 0.0 0.6 0.0 0.0 0.0 0.0 0.1 0.9 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 s’

34. Observation Functions Observations are noisy output of the detection algorithm: {0,1} • Alarm - sensitivity at outbreak states and 1 - specificity in the no outbreak state. • No Alarm - specificity at normal states and 1 - sensitivity in each outbreak state.

35. Costs and Reward • Costs • Investigation (false and true positive) • Intervention (false and true positive) (Kaufman, 1997) • Outbreak by day (false negative) calculated as (# deaths* future earnings) + (# hospitalized * cost of hospitalization) + (# outpatient visits * cost of visit) (Kaufman, 1997) • Rewards • Preventable loss at each day - investigation or intervention costs

36. Costs and Reward • Population exposed: 100,000

37. Automated Approach

38. Performance Measurements • Timeliness • Sensitivity • Economic

39. Timeliness • If two algorithms have identical sensitivity and false alarm rates, then the algorithm that detects outbreaks earlier is better • Number of days between the start of a true outbreak and the time of confirming the outbreak.

40. Sensitivity • Fraction (or percentage) of outbreaks detected • Determined in the literature for: • each day of outbreak • different detection methods • different outbreak sizes Reis et al. (2003)

41. Economic • Health outcomes of the applied policy • Total benefit-cost over some period of time

42. Experimental Design • A range of detection methods used. • The outbreak results in a mean increase in ED visits: 10% (small), 20%(medium), 30%(large). • POMDP policy and a manual policy are evaluated and compared for ten years simulation. • Robustness of the model w.r.t. parameter perturbation is examined.

43. Sensitivity in Days of Outbreak

44. Sensitivity Results • POMDP operating on detection method, compared to detection method alone Small size outbreak Day of Outbreak

45. Sensitivity Results Larger size outbreak Day of Outbreak

46. Policy Return w.r.t Specificity

47. Using Different Detection Methods

48. Policy Return w.r.t. Outbreak Magnitude

49. Conclusion • AI techniques may prove useful in guiding public health management decisions. • POMDP improves the timeliness and the sensitivity of outbreak detection processes. • The machinery is simple but potentially powerful • Handle uncertainty well

50. Limitations • Learn how to tailor appropriate resolutions for different variables. • Important to transfer across domains. • Our current planning strategy starts by assuming the world is deterministic, making a plan, and then elaborating it to deal with uncertainty.