What’s Strange About Recent Events (WSARE)

1. What’s Strange About Recent Events (WSARE) Weng-Keen Wong (Carnegie Mellon University) Andrew Moore (Carnegie Mellon University) Gregory Cooper (University of Pittsburgh) Michael Wagner (University of Pittsburgh) DIMACS Tutorial on Statistical and Other Analytic Health Surveillance Methods

2. Motivation Suppose we have access to Emergency Department data from hospitals around a city (with patient confidentiality preserved)

3. The Problem From this data, can we detect if a disease outbreak is happening?

4. The Problem From this data, can we detect if a disease outbreak is happening? We’re talking about a non-specific disease detection

5. The Problem From this data, can we detect if a disease outbreak is happening? How early can we detect it?

6. The Problem From this data, can we detect if a disease outbreak is happening? How early can we detect it? The question we’re really asking: In the last n hours, has anything strange happened?

7. Traditional Approaches What about using traditional anomaly detection? • Typically assume data is generated by a model • Finds individual data points that have low probability with respect to this model • These outliers have rare attributes or combinations of attributes • Need to identify anomalous patterns not isolated data points

8. Traditional Approaches What about monitoring aggregate daily counts of certain attributes? • We’ve now turned multivariate data into univariate data • Lots of algorithms have been developed for monitoring univariate data: • Time series algorithms • Regression techniques • Statistical Quality Control methods • Need to know apriori which attributes to form daily aggregates for!

9. Traditional Approaches What if we don’t know what attributes to monitor?

10. Traditional Approaches What if we don’t know what attributes to monitor? What if we want to exploit the spatial, temporal and/or demographic characteristics of the epidemic to detect the outbreak as early as possible?

11. Traditional Approaches Diarrhea cases among children Number of cases involving people working in southern part of the city We need to build a univariate detector to monitor each interesting combination of attributes: Respiratory syndrome cases among females Number of cases involving teenage girls living in the western part of the city Viral syndrome cases involving senior citizens from eastern part of city Botulinic syndrome cases Number of children from downtown hospital And so on…

12. Traditional Approaches Diarrhea cases among children Number of cases involving people working in southern part of the city We need to build a univariate detector to monitor each interesting combination of attributes: Respiratory syndrome cases among females Number of cases involving teenage girls living in the western part of the city You’ll need hundreds of univariate detectors! We would like to identify the groups with the strangest behavior in recent events. Viral syndrome cases involving senior citizens from eastern part of city Botulinic syndrome cases Number of children from downtown hospital And so on…

13. Our Approach • We use Rule-Based Anomaly Pattern Detection • Association rules used to characterize anomalous patterns. For example, a two-component rule would be: Gender = Male AND 40  Age < 50 • Related work: • Market basket analysis [Agrawal et. al, Brin et. al.] • Contrast sets [Bay and Pazzani] • Spatial Scan Statistic [Kulldorff] • Association Rules and Data Mining in Hospital Infection Control and Public Health Surveillance [Brossette et. al.]

14. WSARE v2.0 • Inputs: 1. Multivariate date/time-indexed biosurveillance-relevant data stream 2. Time Window Length 3. Which attributes to use? “Emergency Department Data” “Ignore key” “Last 24 hours”

15. WSARE v2.0 • Inputs: 1. Multivariate date/time-indexed biosurveillance-relevant data stream 2. Time Window Length 3. Which attributes to use? 3. And here’s how seriously you should take it 2. Here’s why • Outputs: 1. Here are the records that most surprise me

16. WSARE v2.0 Overview • Obtain Recent and Baseline datasets 2. Search for rule with best score All Data Recent Data 3. Determine p-value of best scoring rule through randomization test Baseline 4. If p-value is less than threshold, signal alert

17. Step 1: Obtain Recent and Baseline Data Recent Data Data from last 24 hours Baseline Baseline data is assumed to capture non-outbreak behavior. We use data from 35, 42, 49 and 56 days prior to the current day

18. Step 2. Search for Best Scoring Rule For each rule, form a 2x2 contingency table eg. • Perform Fisher’s Exact Test to get a p-value for each rule => call this p-value the “score” • Take the rule with the lowest score. Call this rule RBEST. • This score is not the true p-value of RBEST because we are performing multiple hypothesis tests on each day to find the rule with the best score

19. The Multiple Hypothesis Testing Problem • Suppose we reject null hypothesis when score < , where  = 0.05 • For a single hypothesis test, the probability of making a false discovery =  • Suppose we do 1000 tests, one for each possible rule • Probability(false discovery) could be as bad as: 1 – ( 1 – 0.05)1000 >> 0.05

20. Step 3: Randomization Test • Take the recent cases and the baseline cases. Shuffle the date field to produce a randomized dataset called DBRand • Find the rule with the best score on DBRand.

21. Step 3: Randomization Test Repeat the procedure on the previous slide for 1000 iterations. Determine how many scores from the 1000 iterations are better than the original score. If the original score were here, it would place in the top 1% of the 1000 scores from the randomization test. We would be impressed and an alert should be raised. Estimated p-value of the rule is: # better scores / # iterations

22. Day by Day If we want to run WSARE just for the current day… …then we end here. Historical Analysis If we want to review all previous days and their p-values for several years and control for some percentage of false positives… …then we’ll once again run into overfitting problems …we need to compensate for multiple hypothesis testing because we perform a hypothesis test on each day in the history Two Kinds of Analysis

23. We only need to do this for historical analysis! False Discovery Rate [Benjamini and Hochberg] • Can determine which of these p-values are significant • Specifically, given an αFDR, FDR guarantees that • Given an αFDR, FDR produces a threshold below which any p-values in the history are considered significant

24. WSARE v3.0

25. WSARE v2.0 Review • Obtain Recent and Baseline datasets 2. Search for rule with best score All Data Recent Data 3. Determine p-value of best scoring rule through randomization test Baseline 4. If p-value is less than threshold, signal alert

26. Obtaining the Baseline Baseline Recall that the baseline was assumed to be captured by data that was from 35, 42, 49, and 56 days prior to the current day.

27. Obtaining the Baseline Baseline Recall that the baseline was assumed to be captured by data that was from 35, 42, 49, and 56 days prior to the current day. What if this assumption isn’t true? What if data from 7, 14, 21 and 28 days prior is better? We would like to determine the baseline automatically!

28. Temporal Trends • But health care data has many different trends due to • Seasonal effects in temperature and weather • Day of Week effects • Holidays • Etc. • Allowing the baseline to be affected by these trends may dramatically alter the detection time and false positives of the detection algorithm

29. Temporal Trends From: Goldenberg, A., Shmueli, G., Caruana, R. A., and Fienberg, S. E. (2002). Early statistical detection of anthrax outbreaks by tracking over-the-counter medication sales. Proceedings of the National Academy of Sciences (pp. 5237-5249)

30. WSARE v3.0 Generate the baseline… • “Taking into account recent flu levels…” • “Taking into account that today is a public holiday…” • “Taking into account that this is Spring…” • “Taking into account recent heatwave…” • “Taking into account that there’s a known natural Food-borne outbreak in progress…” Bonus: More efficient use of historical data

31. Time Conditioning on observed environment: Well understood for Univariate Time Series Signal • Example Signals: • Number of ED visits today • Number of ED visits this hour • Number of Respiratory Cases Today • School absenteeism today • Nyquil Sales today

32. Time An easy case Upper Safe Range Signal Mean • Dealt with by Statistical Quality Control • Record the mean and standard deviation up the the current time. • Signal an alarm if we go outside 3 sigmas

33. Time Conditioning on Seasonal Effects Signal

34. Time Conditioning on Seasonal Effects Signal Fit a periodic function (e.g. sine wave) to previous data. Predict today’s signal and 3-sigma confidence intervals. Signal an alarm if we’re off. Reduces False alarms from Natural outbreaks. Different times of year deserve different thresholds.

35. Example [Tsui et. Al] Weekly counts of P&I from week 1/98 to 48/00 From: “Value of ICD‑9–Coded Chief Complaints for Detection of Epidemics”, Fu-Chiang Tsui, Michael M. Wagner, Virginia Dato, Chung-Chou Ho Chang, AMIA 2000

36. Seasonal Effects with Long-Term Trend Weekly counts of IS from week 1/98 to 48/00. From: “Value of ICD‑9–Coded Chief Complaints for Detection of Epidemics”, Fu-Chiang Tsui, Michael M. Wagner, Virginia Dato, Chung-Chou Ho Chang, AMIA 2000

37. Seasonal Effects with Long-Term Trend Called the Serfling Method [Serfling, 1963] Weekly counts of IS from week 1/98 to 48/00. Fit a periodic function (e.g. sine wave) plus a linear trend: E[Signal] = a + bt + c sin(d + t/365) Good if there’s a long term trend in the disease or the population. From: “Value of ICD‑9–Coded Chief Complaints for Detection of Epidemics”, Fu-Chiang Tsui, Michael M. Wagner, Virginia Dato, Chung-Chou Ho Chang, AMIA 2000

38. Day-of-week effects From: Goldenberg, A., Shmueli, G., Caruana, R. A., and Fienberg, S. E. (2002). Early statistical detection of anthrax outbreaks by tracking over-the-counter medication sales. Proceedings of the National Academy of Sciences (pp. 5237-5249)

39. Day-of-week effects Another simple form of ANOVA Fit a day-of-week component E[Signal] = a + deltaday E.G: deltamon= +5.42, deltatue= +2.20, deltawed= +3.33, deltathu= +3.10, deltafri= +4.02, deltasat= -12.2, deltasun= -23.42 From: Goldenberg, A., Shmueli, G., Caruana, R. A., and Fienberg, S. E. (2002). Early statistical detection of anthrax outbreaks by tracking over-the-counter medication sales. Proceedings of the National Academy of Sciences (pp. 5237-5249)

40. Analysis of variance (ANOVA) • Good news: If you’re tracking a daily aggregate (univariate data)…then ANOVA can take care of many of these effects. • But… What if you’re tracking a whole joint distribution of events?

41. Idea: Bayesian Networks Bayesian Network: A graphical model representing the joint probability distribution of a set of random variables “Patients from West Park Hospital are less likely to be young” “On Cold Tuesday Mornings the folks coming in from the North part of the city are more likely to have respiratory problems” “On the day after a major holiday, expect a boost in the morning followed by a lull in the afternoon” “The Viral prodrome is more likely to co-occur with a Rash prodrome than Botulinic”

42. WSARE Overview • Obtain Recent and Baseline datasets 2. Search for rule with best score All Data Recent Data 3. Determine p-value of best scoring rule through randomization test Baseline 4. If p-value is less than threshold, signal alert

43. Obtaining Baseline Data All Historical Data Today’s Environment • Learn Bayesian Network 2. Generate baseline given today’s environment Baseline

44. Obtaining Baseline Data All Historical Data Today’s Environment What should be happening today given today’s environment • Learn Bayesian Network 2. Generate baseline given today’s environment Baseline

45. Step 1: Learning the Bayes Net Structure Involves searching over DAGs for the structure that maximizes a scoring function. Most common algorithm is hillclimbing. Initial Structure 3 possible operations: Add an arc Delete an arc Reverse an arc

46. Step 1: Learning the Bayes Net Structure Involves searching over DAGs for the structure that maximizes a scoring function. Most common algorithm is hillclimbing. Initial Structure But hillclimbing is too slow and single link modifications may not find the correct structure (Xiang, Wong and Cercone 1997). We use Optimal Reinsertion (Moore and Wong 2002). 3 possible operations: Add an arc Delete an arc Reverse an arc

47. T Optimal Reinsertion 1. Select target node in current graph 2. Remove all arcs connected to T T

48. Optimal Reinsertion ? 3. Efficiently find new in/out arcs ? ? T ? ? ? ? ? 4. Choose best new way to connect T T

49. The Outer Loop • Until no change in current DAG: • Generate random ordering of nodes • For each node in the ordering, do Optimal Reinsertion

50. The Outer Loop • For NumJolts: • Begin with randomly corrupted version of best DAG so far • Until no change in current DAG: • Generate random ordering of nodes • For each node in the ordering, do Optimal Reinsertion