“To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals"

1. “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Reflections from San Diego County Biosurveillance Information Exchange Working Group 2/23/06 Jeffrey Johnson, MPH San Diego County Health & Human Services Agency

2. SAN DIEGO COUNTY • Nearly 3 million population • International border • Large military presence • Biotechnology Hub • 21 Emergency Departments

3. Early Event Detection in San Diego • Evolving effort since pre - 9/11 • Data sources: ER Visits, Paramedic transports, 911 calls, school surveillance, OTC sales • Systems: Local SAS/Minitab system, ESSENCE, and BioSense • Statistical • Methods: Descriptive, time series, CUSUM, EWMA, process control methods (P&U Charts) • Multiple syndromes • Visualization and alerting • Incident Characterization • Follow-up to signals County of San Diego Health & Human Services Agency

4. If We Ignore A Signal…… • We take no action or follow-up • Save staff resources • Avoid bothering hospital staff yet again • Another data source may signal • “The Feds may pick it up” • Might lose an earlier start to a response • We might be dead wrong to ignore

5. If We Do Not Ignore a Signal…… • Will it be another “false alarm” • May detect an event earlier • Earlier response • Continued interaction with the medical community • Gain experience with follow-up • Increased situational awareness

6. Characterization of Detections • Detection Method • Syndrome group • % Admitted • Deaths? • Geographic cluster? • Prior day’s level? • Recent level? • Age groups? • Severe syndrome? • Detections in other data sources? • Other epidemiological intelligence? • Other diagnostic information Follow-up? Action or No Action or Watch

7. Detection Follow-up with Medical Community What is the final diagnosis of Patients A, B, C? Is there a common pattern among admitted patients? Did any have lab test results that might suggest a larger event? Among patients with a common zip code, was there a shared living setting or common exposure? Can we send someone out to review medical charts? What is your facility’s assessment of the situation? County of San Diego Health & Human Services Agency

8. ABERRATION DETECTION RESPONSE GUIDELINES Routine Surveillance Activities Ignore? IDENTIFY Aberration detected NO YES Rule out system error Potential false positive VERIFY Preliminary evaluation Ignore? Describe initial results Ignore? “False Positive” YES NO True Positive NOTIFY Inform key departmental staff Inform key divisional staff Intensive monitoring & surveillance Evaluate other data sources Ignore? County of San Diego Health & Human Services Agency Cluster check

9. GI Syndrome Over Time (10/31/04 – 8/24/05) ED 911 Paramedic Runs

10. The Significant Aspects of Syndromic Surveillance • StatisticalSignificance • Public Health Significance • Significant Event • Significant Public Awareness • SignificantBiological Agent Detection

11. Statistical significance vs. public health significance HAZMAT FLAG – 12/04/2004 County of San Diego Health & Human Services Agency

12. Statistical significance vs. public health significance County of San Diego Health & Human Services Agency

13. County of San Diego Health & Human Services Agency

14. Significant event with statistically significance outcomes Syndromic Surveillance for Natural Disasters San Diego Wild Fires, 2003 San Diego County

15. Significant event with statistically significance outcomes Syndromic surveillance for natural disasters

16. Significant Public Awareness “The Clinton Effect” September 4, 2004 While spikes in both datasets are apparent, normalized counts show a relatively larger increase in ED visits on Sept. 6, 2004.

17. Significant Public Awareness 7/7/05 London Bombings San Diego County Paramedic Transports for “Chest Pain”

18. Significant BT Agent Detection Biowatch • BioWatch Detection • Tells us agent, sensor site and date • Plume plot may help us narrow surveillance on a geographic area Application of Syndromic Surveillance Agent: Syndrome categories Specific word search in CC or DX fields Sensor site: Zip codes, population (schools) Date: Temporal based surveillance New pre-detection baselines

19. Anatomy of a Detection (a case example)

20. Daily Email Report Feb 5, 2006 911 Call Data Attached Table

21. 911 Call Center - GI Syndrome Signal

23. Line listing for review Non-specific call complaints

24. 911 Call Center - GI Syndrome Signal The count for the signals include a consistent range Various statistical signals 21 Signals since 07/01/03

25. What did we do? • Magnitude of cases • Which method(s) signaled? • Check the other call centers • Check the other data sources • (ED data, EMS transports) • Review the line listing • Our conclusion….. …... 14 vs mean of 7.8 …... CUSUM (2), P-Chart, U-Chart …… No signals …… No Signals ……. No apparent pattern >>>>> • Super Bowl Sunday • Fewer trauma calls • Smaller denominator (P-Chart) • Traditional increase in GI on this day • Watch next day’s results

26. Case Example #2 Hospital 9 ED Data Respiratory Syndrome

27. Hospital 9 - Daily Results Table

30. Hospital 9 Respiratory Syndrome 01/01/04 - 02/03/06 Many signals…. So what’s the context? Do we ever ignore the signals?

31. Hospital 9 Respiratory Syndrome • 24 signals over a 37 day period • Count range: 11 – 34 • Over time an increasing mean

32. Greater Syndrome Specificity…… Hospital 9 Influenza-like-illness (ILI) Syndrome • “ILI syndrome” has greater syndrome specificity than “Respiratory” syndrome” • 16 signals over a 37 day period

33. What We Have Learned • S$gn&ls Happen! • Make sure you see flames before yelling “Fire” • CUSUM 2 & 3 STD may be too sensitive • We lose precision with non-specific syndromes • Everyone wants to know what’s going on all the time • Increasing focus on situational awareness • Further evaluation and testing required

34. Hype Cycle of Emerging “Syndromic Surveillance” Technologies Adapted from the Gartner Hype Cycle Dual use, situational awareness, appropriate signals Too many signals?, IT Costs, poor syndrome specificity, evaluation results The “magic bullet” 9/11, Anthrax attacks Prioritized data sets, protocols in place, Event or Technology Trigger Peak of Inflated Expectations Trough of Disillusionment Slope of Enlightenment Plateau of Productivity

35. Considerations • More work in all areas of syndromic surveillance is needed • Knowledge requires responsibility • The enemy is studying our efforts • Current/future funding levels require reliability, efficiency and sustainability of systems and approaches • The Future: • Neural networks and Artificial Intelligence (AI)? • Are we ready?

36. Contact Information Jeffrey Johnson 619-531-4945 jeffrey.johnson@sdcounty.ca.gov Thank You