1 / 20

Creating and Evaluating a Consensus for Negated and Speculative Words in a Swedish Clinical Corpus

Creating and Evaluating a Consensus for Negated and Speculative Words in a Swedish Clinical Corpus. Hercules Dalianis Maria Skeppstedt Stockholm University Department of Computer and Systems Sciences. Intro and Contents. An experiment with annotated clinical text Background

nolcha
Download Presentation

Creating and Evaluating a Consensus for Negated and Speculative Words in a Swedish Clinical Corpus

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Creating and Evaluating a Consensus for Negated and Speculative Words in a Swedish Clinical Corpus Hercules Dalianis Maria Skeppstedt Stockholm UniversityDepartment of Computer and Systems Sciences

  2. Intro and Contents • An experiment with annotated clinical text • Background • Creation of a consensus • Automatic detection of cues and the class • Comparison with the BioScope Corpus • Conclusion and next step Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  3. What is special about clinical text? Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  4. Example of clinical text (Swedish) Kvinna med hjärtsvikt, förmaksflimmer, angina pectoris. Ensamstående änka. Tidigare CVL med sequelae högersidig hemipares och afasi. Tidigare vårdad för krampanfall misstänkt apoplektisk. Inkommer nu efter att ha blivit hittad på en stol och sannolikt suttit så över natten. Inkommer nu för utredning. Sonen Johan är med. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  5. Example of clinical text Woman with heart failures, atrial fibrillation, and angina pectoris. Single, widow. Former CVL with sequele, right hemiparesis and aphasia. Prior hosp. care for seizures, apoplectic suspected. Arrive to hospital after being found in a chair and probably been sitting there over night. Arrive for further investigation and care. Accompanied by her son Johan. Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  6. Related research: Negation and speculation detection in clinical text • Both rule-based systems and machine learning systems • Precision and recall from just above 80% to just below 100% • Most on English text Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  7. The Stockholm EPR Corpus • Clinics in Stockholm • 2006-2008 • >800 clinics, >1 million patients • In Swedish Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  8. The annotation • Three annotators • The assessment part of health records • 6 740 sentences Annotated: • Cues for negation and speculation • Classify the sentence as either certain or uncertain, or break it up the into sub-clauses Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  9. The annotation <Sentence> <Uncertain> <Speculative_words> <Negation>Not</Negation> really </Speculative_words> much worse than before </Uncertain> <Sentence> Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  10. Construction of a consensus General idea: • Choose the majority annotation Discarded: • The first annotation rounds discarded (16%) • 2% too different to be resolved, also discarded In the resulting consensus: • 92% identically annotated by at least two persons • 6% identically annotated by at least two persons for class. (For cues, only identical when disregarding the scope. Ex. could perhaps) • 2% only identical for class, only when scope of class disregarded.

  11. Differences between the individual annotations and the consensus • Fewer uncertain expressions • Fewer cues for speculation • Fewer sentences that were divided into sub-clauses Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  12. The BioScope Corpus • Cues for speculation and negation • The scope of speculation and negation <sentence id="S1345.2">Correlation with the patient's height and weight <xcope id="X1345.2.1"><cue type="speculation" ref="X1345.2.1">may</cue> be some value</xcope>.</sentence> Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  13. Comparison between the BioScope Corpus and our corpus Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  14. Our corpus/the BioScope Corpus • Not so detailed guidelines/More detailed guidelines • Consensus with majority decision/Resolving differences with chief annotator (also higher inter-annotator agreement) • Assessment part from many clinics/Radiology reports Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  15. Experiment with the Stanford Named Entity Recognizer Based on Conditional Random Fields • Detections of cues and certain/uncertain • Comparison between our corpus and the BioScope Corpus Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  16. Result of automatic detection of cues for negation Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  17. Result of automatic detection of cues for speculation Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  18. Result of automatic detection of class and scope Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  19. Conclusion and next step • Low results for detecting cues for speculation and class in our constructed corpus • Simplifying the task can hopefully result in: • Higher inter-annotator agreement • Easier to automatically learn to detect speculation Dalianis & Skeppstedt, NeSp-NLP July 10, 2010

  20. Thank you!Questions? Hercules Dalianishercules@dsv.su.se Maria Skeppstedtmariask@dsv.su.se

More Related