Automated surveillance methods for notifiable diseases: theory, practice, and evaluation

1. Automated surveillance methods for notifiablediseases: theory, practice, and evaluation Matthew Samore, MD Utah Center of Excellence in Public Health Informatics Epicenter for Prevention of Healthcare Associated Infection Salt Lake VA Health Care System University of Utah

2. Objectives • Define situation awareness and situation comprehension in the context of public health surveillance • Application of an inference engine to electronic health records for case identification • Evaluation of impact on public health surveillance work processes

3. Outline • Surveillance theory • Surveillance practice • Surveillance evaluation

4. Surveillance Theory

5. Paradigms and Models • The Panopticon • Archetype: “Total information awareness” • Signal detection theory • Trade-off between hit rate and false alarm rate • Situation awareness • Detection, diagnosis, prediction

6. What is situation awareness? Comprehension of a dynamic environment

7. Understanding comprehension • A “phenomenon that emerges from an orchestra of cognitive processes” • Perception • Surface level • “Eventbase” • Semantic information extracted from perceptual input • Situation model • Integration of semantic information with prior knowledge Durso, et. al: Handbook of Applied Cognition

8. A tragic error • “On a beautiful night in October 1978, in the Chesapeake Bay, two vessels sighted one another visually and on radar. On one of them, the Cost Guard cutter training vessel Cuyahoga, the captain saw the other ship up ahead as a small object on the radar, and visually he saw two lights, indicating that it was proceeding in the same direction as his own ship. He thought it possibly was a fishing vessel. • Since the two ships drew together so rapidly, the captain decided that it must be a very slow fishing boat that he was about to overtake.

9. At the last moment the captain of the Cuyahoga realized that in overtaking the supposed fishing boat, which he assumed was on a near-parallel course, he would cut off that boat’s ability to turn as both of them approached the Potomac River. • So he ordered a turn to the port. This brought him directly in the path of the oncoming freighter, which hit the cutter. Eleven coastguardsmen perished.

10. The captain’s mental model

11. Cell phone use during driving • How it may affect comprehension of the dynamic environment of the road • Fail to check rearview mirror (attention) • At 4 way stop, may fail integrate spatial and temporal information to determine who has right of way and should enter intersection first • May recognize that driver in oncoming vehicle is distracted but fail to infer the appropriate situation model, that the car will not stop

12. Situation awareness and public health • Relevance • The health of a population is indeed dynamic • Time horizons cover a wide range • The concept captures the data hungry nature of epidemiology and public health • Current health status • Current health care capacity • Early detection

13. Limitations: • Vaguely expressed as goal of surveillance • Decisions or actions not specified • More importantly: • Description of the situation model is lacking

14. More effective use of the concept of situation awareness • Probe deeper • Interpretation of current state • Projected states • Measurement • Links theory to practice

15. Describing the current situation • As given to computer scientist Yarden Livnat by Mary Hill, epidemiologist at Salt Lake Valley HD • “There is significant flu in the valley” • “Fever and coughs are rising” • “12 people hospitalized throughout the valley” • “20% are children” • “Most cases are rapid A” • “School absenteeism is high in middle/high schools” • “Btw, the hunting season started yesterday”

16. Mental maps of epidemiologists • GI outbreak scenario • Cluster of individuals with acute diarrhea who had eaten at a fast food restaurant • Preliminary analysis: • Approach to problem space varies substantially across public health personnel

17. Surveillance Practice

18. Why syndromic surveillance (e.g., RODS) was perceived to be useful during Winter Olympic games (and other special events) At not very useful at other times

19. Thesis • Utility of syndromic surveillance depends on situation model • Under normal conditions, its value is highly limited • Current level of threat • Suspicion of departure from normality

20. Public Health Practice Based Research

21. Core Principles • Partnerships and synergies • Active participation of health departments at local and state levels • Public health leadership or co-leadership of projects • Vision of public health research laboratory • Interdisciplinary collaboration at local and national levels

22. Utah Department of HealthOffice of Public Health Informatics • Directed by Wu Xu, PhD • Goals • Coordinate statewide eHealth initiatives • Coordinate integration of projects • Create a laboratory for applied public health research • Develop theory and methods that advance information and population science in the context of public health practice

23. How CoE Projects Support Public Health Case Identification RT-CEND Case Investigation & Management INTERACT Analysis Linkage projects Rx Drug Deaths Readmission/Mortality Actions based on Results DSIDE Data Dissemination Outbreak response, advisories, other actions

24. Leveraging the electronic health record • Real-Time Clinical Electronic Notifiable Disease Surveillance • Electronic health record-based syndromic surveillance • Text processing • Leads • Utah Department of Health • Lisa Wyman, Melissa Stevens Dimond, David Jackson, Corona Nigatuvai, Robert Rolfs • University of Utah • Catherine Staes, Deepthi Rajeev • Intermountain Healthcare • Scott Evans, Per Gesteland • VA • Brett South, Matthew Samore, Adi Gundlapalli, Sylvain DeLisle

25. Data visualization and decision support • Pathogen-specific surveillance (GermWatch) • Heterogenic data visualization • Public health decision support • Interactive simulation • Key investigators include: • Per Gesteland, Carrie Byington, Andy Pavia, Adi Gundlapalli, Yarden Livnat, Frank Drews, Laverne Snow, Chris Barrett, Stephen Eubank, Madhav Marathe, Jim Koopman, Yong Yang, Robert Rolfs

26. RT-CEND Project • Health care system: • Rule-based detection of notifiable diseases • Message sending • Electronic case transmission • Health department • Message receiving • Integration into workflow at local and state level

28. Electronic Medical Record Data Driver Laboratory Data 1 4 Medical Logic Modules Decision Support Engine Code Tables 2 3 5 Alert File

29. Time Driver Alert File Reportable Disease Monitor ICP Daily Printout Electronic Medical Record

30. Counts of notifiable diseases alerted

31. CURRENT data flow Intermountain HC Local Health Dept EMR NETSS Manual entry faxed Daily report Alert file Email or fax State Health Dept NETSS Manual entry

32. Simplified view of NEW data flow Intermountain HC Local Health Dept EMR NETSS Manual entry faxed Daily report Alert file Email or fax HL7message State Health Dept NETSS Study/ NEDSS Manual entry Data views, reports, extracts

33. Electronic case transmission • Work led by Deepthi Rajeev, Catherine Staes, Scott Evans • Compiled required data fields • Modeled the HL7 message structure for reporting from healthcare systems to local & state health departments • Evaluated existing messaging models, for instance PHIN implementation • Allows transmission of multiple lab tests based on one or multiple specimens in a single message • Message structure implemented

34. Planned work on health care system side • Automated HIPAA documentation • Chart review validation

35. Message receipt &workflow integration • Appropriate data flow between state and local health departments • Utah Department of Health • Lisa Wyman, Corona Nigatuvai, David Jackson, Melissa Stevens Dimond, Robert Rolfs • Salt Lake Valley Health Department • Heath Harris, Mary Hill, Ilene Risk • Davis County Health Department • Brian Hatch, Nicole Stone

36. NEDSS implementation in Utah • Current status: • Vendor: Collaborative Software Initiative • Funded by grant from Novell • Open source software model • Agile, rapid development method underway

37. Enhancing situation comprehension • Leveraging the electronic health record to support public health investigation • Start with possible event of public health interest • Use pre-defined and ad hoc queries on a health care system data warehouse to support evaluation • Collect additional epidemiologic data: how are cases linked? • Who, what, where, when

38. How “good” are electronic health record-based case criteria? • May be better or worse than conventional criteria • Conventional case criteria constitute a reference standard but are never a true gold standard • Wit h respect to disease occurrence in a target population • Sensitivity is virtually always less than 100% because of incomplete clinical evaluation and testing • Lack of true gold standard with respect to

39. Surveillance Evaluation

40. Conceptual framework • Six core activities • Detection • Registration • Confirmation • Reporting • Analyses • Feedback. • Public health action • Acute (epidemic-type) responses • Planned responses Mcnabb, et. al. , BMC Public Health, 2002

41. Evaluation of public health information system implementation • Formative evaluation • Semi-structured interviews • Survey administration • Information technology acceptance model • Fit to workflow • Observation of efficiency of work processes • Measurement of timeliness, completeness, accuracy • Assessment of the application development process

42. Time Line

43. Synthesis

44. Tier 1: Event Detection Our project activities • Trade-offs • Reliability versus validity of case criteria • “False alarm rate” versus “hit rate” • Text processing • Rash syndromes • Rule development • Central nervous system • infections

45. Tier 2: Cognitive processes Our related research activities • Center of Excellence • in Public Health • Informatics • Public health • decision-making • Simulation-based • decision support

46. Tier 3: Broader Public Health Goals Our related research activities • Informatics • Information • exchange • Data linkage • Epidemiology • Epidemic models • Valid ecologic • inference

47. Tier 4: System Performance Syndromes Local/state health dept- centered activities Sensors • Evaluation of • existing systems • (ELR, RODS, others) • New system implementation • BioSense • NEDSS • RT-CEND Notifiable diseases Pathogen- specific