1 / 34

Speaker: Yu-Ching Fang Advisors: Hsueh-Fen Juan and Hsin-Hsi Chen

Automated recognition of malignancy mentions in biomedical literature BMC Bioinformatics 2006, 7:492. Speaker: Yu-Ching Fang Advisors: Hsueh-Fen Juan and Hsin-Hsi Chen. Outline. Background Methods Results Discussion Conclusion. Background - Motivation.

tanginika-morris
Download Presentation

Speaker: Yu-Ching Fang Advisors: Hsueh-Fen Juan and Hsin-Hsi Chen

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automated recognition of malignancy mentions in biomedical literature BMC Bioinformatics 2006, 7:492 Speaker: Yu-Ching Fang Advisors: Hsueh-Fen Juan and Hsin-Hsi Chen

  2. Outline • Background • Methods • Results • Discussion • Conclusion

  3. Background - Motivation • The rapid proliferation of biomedical literature makes it increasingly difficult for researchers peruse, query, and synthesize it for biomedical knowledge gain. • Less biomedical text mining work has been performed to identify disease-related objects and concepts. • Related works about automated disease entity recognition often do not perform well. • More extensive work on medical entity class recognition is necessary.

  4. Related works • Automated extractors for the identification of gene and protein names. • Automated entity recognition to the identification of phenotypic and disease objects. • A machine-learning algorithm to extract gene-disorder relations. • Extract phenotypic attributes from Online Mendelian Inheritance in Man (OMIM).

  5. Goal • Develop a named entity recognizer (MTag), an entity tagger for recognizing clinical descriptions of malignancy presented in text. MTag is based upon the probability model Conditional Random Fields (CRFs). • Minimize manual efforts and still perform with high accuracy.

  6. Conditional Random Fields (CRFs) • CRFs are probabilistic tagging models that give the conditional probability of a possible tag sequence t = t1, ... tngiven the input token sequence o = o1,..., on(Ryan McDonald and Fernando Pereira, 2005). • For example, the identification of gene mentions in text can be implemented as a tagging task. o: t: Begins (B), continues (I), or is outside (O) of a gene mention

  7. Conditional Random Fields (CRFs) input token sequence tag sequence John Lafferty et al., 2001

  8. Methods – Task definition • To develop an automated method that would accurately identify and extract strings of text corresponding to a clinician’s or researcher’s reference to cancer (malignancy). • Label “Malignancy”: the full noun phrase encompassing a mention of a cancer subtype. • For example, “neuroblastoma”, “localized neuroblastoma” and “primary extracranial neuroblastoma” were considered to be distinct mentions of malignancy. • Directly adjacent prepositional phrases were not allowed, such as “cancer <of the lung>”.

  9. Two corpus combination • The first corpus concentrated upon a specific malignancy (neuroblastoma) and consisted of 1,000 randomly selected abstracts identified by querying PubMed with the query terms "neuroblastoma" and "gene". • The second corpus consisted of 600 abstracts previously selected as likely containing gene mutation instances for genes commonly mutated in a wide variety of malignancies.

  10. Two corpus combination (cont.) • 1000+600-158=1442 abstracts (eliminating 158 abstracts that appeared to be non-topical, had no abstract body, or were not written in English.) • Manually annotated for tokenization, part-of-speech assignments, and malignancy named entity recognition.

  11. Two corpus combination (cont.) • Annotations were performed on all documents by experienced annotators with biomedical knowledge. • Discrepancies were resolved through forum discussions. • A total of 7,303 malignancy mentions were identified in the document set.

  12. MTag algorithm • MTag was developed using the probability model Conditional Random Fields (CRFs). • CRFs model the conditional probability of a tag sequence given an observation sequence. • O is an observation sequence, or a sequence of tokens in the text. • t is a corresponding tag sequence in which each tag labels the corresponding token with either Malignancy(meaning that the token is part of a malignancy mention) or Other. O: Lung cancer may be related to gene mutation. t: <Malignancy><Malignancy><Other><Other><Other><Other><Other><Other>

  13. MTag algorithm (cont.) • CRFs are based on a set of feature functions, fi(tj, tj-1, O). • This feature represents the probability of whether the token "cancer" is tagged with label Malignancygiven the presence of "lung" as the previous token. O: Lung cancer may be related to gene mutation. t: <Malignancy><Malignancy><Other><Other><Other><Other><Other><Other>

  14. MTag algorithm (cont.) • Consider many textual features when it makes decisions on classifying whether a word comprises all or part of a malignancy mention. • Word-based features: The frequency of each string of 2, 3, or 4 adjacent characters (character n-grams) within each word of the training text was calculated. For example, lung (lu, lun, lung, un, ung, ng) • The differential frequency of each n-gram within words manually tagged as being malignancy mentions was considered as a series of features. For example: lung (bigram: 3/6, trigram:2/6, fourgram:1/6)

  15. MTag algorithm (cont.) • Orthographic features included the usage and distribution of punctuation, alternative spellings, and case usage. • Domain-specific features comprised a lexicon of 5,555 malignancies and a regular expression for tokens containing the suffix -oma.

  16. Evaluation • The evaluation set: 432 abstracts - 2,031 sentences containing mentions of malignancy - 3,752 sentences without mentions • Correctly identified if the predicted and manually labeled tags were exactly the same in content and both boundary determinations. • The performance of MTag was calculated according to precision, recall and F-measure.

  17. Results - MTag performance • Two separate training experiments were performed, either with or without the inclusion of malignancy-specific features, which were the addition of a lexicon of malignancy mentions and a list of indicative suffixes.

  18. MTag performance (cont.)

  19. MTag performance (cont.) • As expected, the extractor performed with higher accuracy with the more narrowly defined corpus (neuroblastoma). • At least for this class of entities, the extractor performs the task of identifying malignancy mentions efficiently without the use of a specialized lexicon.

  20. Extraction versus string matching • String matching: the NCI (National Cancer Institute) neoplasm ontology, a term list of 5,555 malignancies, was used as a lexicon to identify malignancy mentions. • Lexicon terms were individually queried against text by case-insensitive exact string matching.

  21. Extraction versus string matching (cont.) random selection MTag: automated extractor Testing set (432 abstracts) 39 abstracts (202 malignancy mentions) String matching

  22. Extraction versus string matching (cont.) • MTag identified 190 of the 202 mentions correctly (94.1%), while the NCI list identified only 85 mentions (42.1%), all of which were also identified by the extractor.

  23. Extraction versus string matching (cont.) • Change lexicon for string matching • 79 of 202 mentions (39.1%) • Combining the manually-derived lexicon with the NCI lexicon yielded 124 of 202 matches (61.4%). Malignancy mentions identified in the manually curated training set annotations (1,010 documents) NCI list 85 mentions (42.1%)

  24. Extraction versus string matching (cont.) • 202-124=78 (68) malignancy mentions • Missed by the string matching with combined lists but positively identified by MTag. Minor variations in spelling and form (e.g., "leukaemia" versus "leukemia") acronyms (e.g., "AML" in place of "acute myeloid leukemia") 68 malignancy mentions New mentions of malignancies that were in neither in the NCI list or training set. • This suggests that MTag contributes a significant learning component.

  25. Application to MEDLINE • MTag was used to extract mentions of malignancy from all MEDLINE abstracts through 2005. • 15,433,668 documents • A total of 9,153,340 redundant mentions and 580,002 unique mentions (ignoring case) were identified.

  26. Application to MEDLINE (cont.) • The 25 mentions found in the greatest number of abstracts by MTag

  27. Application to MEDLINE (cont.) • Six false postives: pulmonary, fibroblasts, neoplastic, neoplasm metastasis, extramural, and abdominal • Only "extramural“ is not frequently associated with malignancy descriptions. • The remaining five phrases are likely the result of the extractor: - failing to properly define mention boundaries in certain cases. For example, "neoplasm“ v.s “neoplasm metastasis”. - shared use of an otherwise indicative character string (e.g., "opl" in "brain neoplasm" and "neoplastic") between a true positive and a false positive.

  28. Application to MEDLINE (cont.) • To assess document-level precision, 100 abstracts identified by MTag were randomly selected each for the malignancies "breast cancer" and "adenocarcinoma". • Manual evaluation of these abstracts showed that all of the articles were true positives.

  29. MTag input and output • Directly accept files downloaded from PubMed and formatted in MEDLINE format as input. • Text or HTML file versions of the extractor output results.

  30. MTag HTML output

  31. Discussion • It is evident that anF-measure of 0.83 is not sufficient as a stand-aloneapproach for curation tasks. • However, such an approach provideshighly enriched material for manual curators to utilizefurther. • Substantial improvementand efficiency • MTag appeared to beaccurately predicting malignancy mentions.

  32. Discussion (cont.) • Analysis of mis-annotations would likely suggest additional features and/or heuristics that could boost performance considerably. • It may be no need for extensive domain-specific lexicons because the addition of biological features provided very little boost to the recall rate.

  33. Conclusion • MTag is one of the first directed efforts to automatically extract entity mentions in a disease-oriented domain with high accuracy. • MTag substantially outperformed information retrieval methods using specialized lexicons. • When combined with expert evaluation of output, MTag can assist with vocabulary building for cancer entity class.

  34. Thank you for your attention

More Related