1 / 38

Trends in Clinical QA and MiPACQ

Rodney D. Nielsen UNT. Trends in Clinical QA and MiPACQ. About MiPACQ. Multi-source Integrated Platform for Answering Clinical Questions Two year ARRA grant: Harvard Medical School, Mayo Clinic, University of Colorado at Boulder Built on ClearTK NLP framework (CU Boulder)

austin
Download Presentation

Trends in Clinical QA and MiPACQ

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Rodney D. NielsenUNT Trends in Clinical QA and MiPACQ

  2. About MiPACQ • Multi-source Integrated Platform for Answering Clinical Questions • Two year ARRA grant: Harvard Medical School, Mayo Clinic, University of Colorado at Boulder • Built on ClearTK NLP framework (CU Boulder) • Facilitates use of best-in-class NLP & ML systems • Based on Apache UIMA • cTAKES annotation system (Mayo)

  3. Information Retrieval QA Approach • Document IR approach to Question Answering Scores Question Extract Keywords IR Query Index

  4. Information Retrieval QA Approach • Document IR approach to Question Answering Scores Question Extract Keywords IR Query • Clinques (Ely et al.) • 4500 clinical questions • Physician interviews • Final QA dataset: • 161 questions Examples removed: Let's give these immunizations. That's right isn't it? I wonder if this patient could have a rotator cuff thing? Index

  5. Information Retrieval QA Approach • Document IR approach to Question Answering Scores Question Extract Keywords IR Query • Clinques (Ely et al.) • 4500 clinical questions • Physician interviews • Final QA dataset: • 161 questions How do you diagnose and treat post-streptococcal glomerulonephritis? What are the possible causes of a sciatic nerve injury (including mechanical and inflammatory) and what is the best way to distinguish among the causes? Index

  6. Information Retrieval QA Approach • Document IR approach to Question Answering Scores Question Extract Keywords IR Query PSGN is most common in children between the ages of 3 and 7, although it can strike at any age, and it most often affects boys. It lasts only a brief time and usually allows the kidneys to recover. In a few cases, however, kidney damage may be permanent, requiring dialysis or transplantation to replace renal function. • Medpedia • Wiki • 8351 articles • 660,935 paragraphs • Vetted contributors • Somewhat variablewriting style Index

  7. Information Retrieval QA Approach • Document IR approach to Question Answering Scores Question Extract Keywords IR Query • Lucene IR Engine • Apache open source Index

  8. Document-Level Results • Even simple IR systems can do well at document level

  9. Information Retrieval QA Approach • Paragraph IR approach to Question Answering Scores Question Extract Keywords IR Query IR Query Scores ¶ ¶ ¶ … Index Index ¶ ¶ ¶ …

  10. Information Retrieval QA Approach • Paragraph IR approach to Question Answering Scores × ¶ Question Extract Keywords IR Query IR Query Scores ¶ ¶ ¶ … Scores ¶ ¶ ¶ … Index Index ¶ ¶ ¶ …

  11. Paragraph-Level Results • Lucene baseline • 7.4% of questions have answers in the first position • Mean reciprocal rank = 0.140 ≈ 1/7 • ≈ R = 7th for all questions, or • ≈ R = 1st for 1/7 of the questions

  12. Information Retrieval QA Approach • Semantic expansion of keywords Scores × ¶ Question Extract & Expand Keywords IR Query IR Query Scores ¶ ¶ ¶ … ¶ ¶ ¶ … Index Thesaurus Index ¶ ¶ ¶ … nerve  neuron, neural … Not courage, guts, audacity, face LexGrid

  13. Information Retrieval QA Approach Scores × ¶ Question Extract & Expand Keywords IR Query IR Query Scores ¶ ¶ ¶ … ¶ ¶ ¶ … Index Thesaurus Index ¶ ¶ ¶ …

  14. Information Retrieval QA Approach Question IR Queries ¶ ¶ ¶ …

  15. Answer Filtering • Expected Answer Type/Question Classification Question Expected Answer Type Filter Answers ¶ ¶ ¶ … IR Queries Entity/Text Classification If the retrieved text does not include an entity of the expected answer type or has an inconsistent classification Then filter out that potential answer What antihypertensives do not cause impotence? Expected Answer Type = Drug

  16. Answer Re-ranking • Re-ranking versus Filtering Question Expected Answer Type ¶ ¶ ¶ … Re-rank Answers ¶ ¶ ¶ … IR Queries Entity/Text Classification Rescore top subset of potential answers to increase the score of those with consistent classifications and or decrease the score of those with inconsistent classifications. What antihypertensives do not cause impotence?If Drug is present, then , else  (but not )

  17. Question & Answer Annotations • Syntactic and Named Entity annotation Question Question Annotation Re-rank Answers ¶ ¶ ¶ … IR Queries Answer Annotation Q: What antihypertensives do not cause [Disorderimpotence]? H: … this antihypertensive is impotent in the face of … Q: How do you diagnose [Disorderdepression] in children? H: … slight depression in the skull …

  18. Paragraph-Level Results • Rule-based re-ranking gives a large improvement • 83% relative improvement in Precision at One • 51% relative improvement in Mean Reciprocal Rank • Information used • cTAKES Named Entities and basic syntactic annotations • No Expected Answer Type

  19. Answer Re-ranking • Machine Learning based Re-ranking Question Question Annotation ¶ ¶ ¶ … ML Reranking of Answers ¶ ¶ ¶ … IR Queries Answer Annotation Extract Features: [ExpAnsType, match, NE-types…, %match, …] Annotate Training Data Train a Classifier

  20. Paragraph-Level Results • ML-based re-ranking resulted in further gains • 134% & 27% improvement in P@1 relative to IR & Rules • 90% & 25% improvement in Mean Reciprocal Rank relative to IR baseline and Rule-based Re-ranking, respectively • Information used • cTAKES Named Entities and basic syntactic annotations • No Expected Answer Type

  21. Pattern Matching • Expected Answer Patterns Expected Answer Type Expected Patterns Phrasal IR Queries Question Question Annotation Rerank Answers ¶ ¶ ¶ … Term IR Queries Answer Annotation How do you treat nephrotic syndrome? nephrotic syndrome is treated by $ANSWER treat nephrotic syndrome with $ANSWER

  22. Deeper Semantic Annotation • Relations, Predicate-Argument Structure, … Question Question Annotation Phrasal Queries Rerank Answers ¶ ¶ ¶ … Term Queries Answer Annotation • Can you use Prozac along with lithium?Interacts-with(Prozac, lithium, true) • Top documents: independent mentions • lists, classes (antidepressant), affects on (e.g., breastfeeding), discussions of a third drug, etc. • Page 3: … Prozac is prescribed with lithium … Interacts-with(Prozac, lithium, false)

  23. Deeper Semantic Annotation • Temporal Relations Question Question Annotation Phrasal Queries Rerank Answers ¶ ¶ ¶ … Term Queries Answer Annotation Is peripheral edema usually experienced after other symptoms in Diabetes?

  24. Answer Summarization • Summarize disparate information Summarize Question Question Annotation Phrasal Queries Rerank Answers ¶ ¶ ¶ … ¶ ¶ ¶ … Term Queries Answer Annotation Within a document How do you diagnose and treat post-streptococcal glomerulonephritis? … PSGN can bring on sudden symptoms of swelling (edema), … Tests will show large amounts of protein in the urine and elevated levels of … ... usually allows the kidneys to recover. In a few cases, however, kidney damage may be permanent, requiring dialysis or transplantation to replace renal function.

  25. Answer Summarization • Summarize disparate information Summarize Question Question Annotation Phrasal Queries Rerank Answers ¶ ¶ ¶ … ¶ ¶ ¶ … Term Queries Answer Annotation Multi-document summarization How do you diagnose and treat post-streptococcal glomerulonephritis? ... usually allows the kidneys to recover. In a few cases, however, kidney damage may be permanent, requiring dialysis or transplantation to replace renal function. … This disease cannot be prevented with treatment with antibiotics. Fortunately, this disease is usually self-limiting and resolves in about two weeks.

  26. MiPACQ: Multi-source Integrated Platform • Multiple encyclopedic sources Summarize Question Question Annotation Phrasal Queries Rerank Answers ¶ ¶ ¶ … ¶ ¶ ¶ … Term Queries Answer Annotation Index Index Index ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … Multiple sources Integration and Synthesis

  27. MiPACQ: Multi-source Integrated Platform • Integrating Medical Records Summarize Question Question Annotation Phrasal Queries Rerank Answers ¶ ¶ ¶ … ¶ ¶ ¶ … Term Queries Answer Annotation EMR Index EMR EMR Index Index Notes ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … What medications have been tried for this patient and how effective have they been? When were these symptoms first seen?

  28. Interactive Refinement • Iteratively improve the question with the user Interactive Refinement Question Question Annotation Phrasal Queries Rerank Answers Summary Term Queries Answer Annotation Index Index Index EMR EMR EMR Notes ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … Co-reference resolution Question validation Answer verification

  29. Clinical Research Questions • Cohort Queries Interactive Refinement Cohort Query Question Annotation Phrasal Queries Rerank Answers Summary Term Queries Answer Annotation Index Index Index EMR EMR EMR Notes ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … What pre-menopausal women age 40 to 60 were seen in Department W from 1950 to present with condition X, were given medication Y, and subsequently developed Z.

  30. Clinical Research Questions • Research Query Dialogues Interactive Refinement Research Query Question Annotation Phrasal Queries Rerank Answers Summary Term Queries Answer Annotation Index Index Index EMR EMR EMR Notes ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … What patients were diagnosed with Ω primary or systemic or acquired not secondary not localized or Ω peripheral γ and Ωtrophicβ or β or α Exclusion & expansion dialogue

  31. Research Queries • Data Mining Queries Interactive Refinement Research Query Question Annotation Phrasal Queries Rerank Answers Summary Term Queries Answer Annotation Index Index Index EMR EMR EMR Notes ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … What actions were taken due to a diagnosis of Long QT Syndrome in patients seen at/by …? What factors affect the efficacy of …?

  32. MiPACQ & Trends in Clinical QA Systems How do you diagnose and treat post-streptococcal glomerulonephritis? Question Question Annotation Phrasal Queries Rerank Answers ¶ ¶ ¶ … Is peripheral edema usually experienced after other symptoms in Diabetes? Question Extract Keywords IR Query Interactive Refinement Term Queries Answer Annotation Query Question Annotation Phrasal Queries Rerank Answers Summary Term Queries Answer Annotation Index Index EMR Index EMR Index EMR ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … ¶ ¶ ¶ … Notes What factors effect the efficacy of …?

  33. Online Continuous Mining, Learning & Adaptation

  34. Acknowledgements • Guergana K. Savova, Dann Albright, Brian L. Cairns, AnwenFredricksen, Donna Ihrke, ArrickLanfranchi, James H. Martin, James J. Masanz, Martha Palmer, Wayne H. Ward • The project described was supported by award number NLM RC1LM010608. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NLM/NIH. • Ely et al. for making the clinical questions available via the NLM website, and • All the contributors to Medpedia

  35. Questions

  36. MiPACQ Architecture PoC Clinician or Lab investigator Web UI QA System Manager Question Annotation Keyword Queries Answer Pattern Classificatn Pattern Queries Answer Analysis & Re-ranking Answer Type Classificatn Query Validation Answer Summariztn Result Set Annotation Follow-up Question Preprocsng Query Term Semantic Expansion i2b2 Query Interface Lucene Interface Exclusion/Expansion Analysis ClearTK Unstructured Information Management Architecture (UIMA) LexEVS APIs EDT Query Engine Lucene EDT-specific Exclusion/Expansion EDT-specific Query Validation Data, Features, Annotations LexGrid Data EDT RDB MedPedia

  37. Annotation Pipeline • Multi-stage pipeline built on Apache UIMA • cTAKES(Clinical Text Analysis and Knowledge Extraction System) • Sentence/token segmentation • Normalization • Lemmatization • POS tagging • Chunking • UMLS named entity identification • ClearParser syntactic dependency parsing • Annotates both questions and answers

  38. MiPACQ Text Annotation Identifindergeneral NE Tagger OpenNLP Constituent Parser cTAKES Constituent Parser Interface Predicate Identificatn Verb-basedSemantic Role Labels Entity Coreference Entity Relation Classificatn Paragraph Segmentatn IdentifinderInterface Dependncy Parser Event Detection & Classificatn Noun-basedSemantic Role Labels Event Coreference Resolution Event Relation Classificatn Clinical Text Analysis and Knowledge Extraction System (cTAKES) Named Enitity tagging Context-dependent Tokenizer Context Annotator Sentence Boundary Detector Tokenizer Normalizer Part of Speech Tagger Phrasal Chunker Dictionary lookup Annotator Negation Detector ClearTK Unstructured Information Management Architecture (UIMA) Dictionary Data, Features, Annotations

More Related