210 likes | 551 Views
University of Pittsburgh. Dept. of Biomedical Informatics. A Review of Chief Complaint Classifiers for Syndromic Surveillance. Wendy W. Chapman, PhD. Syndromic Surveillance from Chief Complaints. Triage clerk. Chief complaint Cough/fever. CC Classifier. Syndromic Category
E N D
University of Pittsburgh Dept. of Biomedical Informatics A Review of Chief Complaint Classifiers for Syndromic Surveillance Wendy W. Chapman, PhD
Syndromic Surveillance from Chief Complaints Triage clerk Chief complaint Cough/fever CCClassifier • Syndromic • Category • Respiratory • GI • Neurological • Rash • Hemorrhagic • Botulinic • Constitutional Alarms Epidemiology Team
Overview • Chief complaint classifiers • How do they work? • How are they different? • Evaluation of chief complaint classification • How do we evaluate their performance? • How well do they work?
Chief Complaint Classifiers Quantify the similarities and differences of automated chief complaint classifiers • Distributed survey to developers of classifiers • Characterized chief complaint classifiers • Syndromes they output • Methods for mapping free-text to syndromes • Type of pre-processing techniques Chief Complaint Classifiers Evaluation Future Work
ESSENCE CC Classifier Biosense EARS Mayo Clinic Vocabulary Server Ontology-enhanced BioPortal CC Classifier QUESST ME Classifier NC DetectSyndrome Case Report CC-MCSVM NGram CC Classifier CoCo SyCo MPLUS NYC Syndromic Macros CC-EDS Chief Complaint Classifiers14 Survey Respondents
Majority of classifiers map to Respiratory Gastrointestinal Neurological Most systems map to Hemorrhagic Rash Botulinic Some systems map to Constitutional (6) Fever (4) Shock/Coma (1) Influenza-like Illness (4) Infectious Derm (1) Trauma (1) Local Lesion (3) Lymphadenitis (2) Sudden illness/death (2) Injury (1) Specific infection (1) Syndromes The Classifiers Output
Difference Between Similar Syndromes Many syndromes represent a more specific presentation of a general syndrome • Single symptom • diarrhea • Severity • severe GI • Anatomic location • upper respiratory, lower GI • Likelihood of being caused by a bioterrorist agent or being infectious • Meningoencephalitis • Febrile syndrome (febrile rash) • Not infectious • Chronic respiratory (asthma and COPD)
Symptoms or Syndromes • Classify directly to syndromes “SOB” = Respiratory • Classify to symptoms then to syndromes “SOB” = Dyspnea = Respiratory • Can generate new syndromes on the fly
Multiple Problems • Assigns a singlesyndrome to a string “short of breath/vomiting” = Respiratory • Assigns multiplesyndromes to a string “short of breath/vomiting” = Respiratory, GI
Synonyms Short of breath dyspnea Coughing cough Coughs cough Abbreviations ha headache abd abdominal gx ground transportation Acronyms n/v nausea/vomiting sob shortness of breath Truncations diar diarrhea poss possible Concatenations blurredvision burred vision flus sxs flu symptoms Misspellings & typographic errors nausa nausea diahrea diarrhea Preprocessing Substantial word variation in chief complaints
Preprocessing • General cleanup • replace symbols with spaces, lowercase, etc. • Synonym replacement • Truncation replacement • Abbreviation replacement • Spell-checking • Removal of certain words • Map words in string to concepts in a standardized vocabulary (e.g., UMLS) • Split multiple complaints
BioPortal EMT-P (Travers) NC Detect
Feature Identification Case Identification Outbreak Detection Does the NLP application accurately identify features? Do the features represent a patient’s actual state? Can we detect outbreaks from the identified features? Yes Yes No No Stop Stop Three Stages of Evaluationfor Biosurveillance Chief Complaint Classifiers Evaluation Future Work
What Have we Learned from Evaluations of Classifiers? • Classifiers are good at feature detection • “n/v/d” Gastrointestinal • Classified chief complaints show high specificity and moderate sensitivity in case detection • 10% - 70% sensitivity • Classified chief complaints can be used to detect *some* outbreaks more accurately and earlier than traditional methods
Preprocessing does not improve case identification performance* • Exception: splitting chief complaints into multiple problems for classifiers that assign single syndromes • We have no idea which classifiers perform best • Partly depends on preference of mapping methods • No standardized syndromic definitions for comparing classifiers • classifiers with different output are difficult to compare * One study of two preprocessors and two classifiers
Future Directions • Chief complaints are • Ubiquitous • Timely • Simple to process • Chief complaints are not • Always accurate • Complete • We need to monitor multiple data sources as they come available • Pre-diagnostic (over-the-counter sales) • Diagnostic • ED Reports • Chest radiograph reports Chief Complaint Classifiers Evaluation Future Work