1 / 18

Just-in-Time Interactive Question Answering

Just-in-Time Interactive Question Answering. Language Computer Corporation Sanda Harabagiu, PI John Lehmann John Williams Paul Aarseth. Overview. Project Introduction Preparation for “Wizard of Oz” pilot Performance in WOZ pilot Challenges encountered in WOZ pilot

nevin
Download Presentation

Just-in-Time Interactive Question Answering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Just-in-Time Interactive Question Answering Language Computer Corporation Sanda Harabagiu, PI John Lehmann John Williams Paul Aarseth

  2. Overview • Project Introduction • Preparation for “Wizard of Oz” pilot • Performance in WOZ pilot • Challenges encountered in WOZ pilot • Current work and future plans

  3. Research Project Objective • Address the interactive aspect of QA systems by designing and implementing a dialog shell that can be used with any QA system

  4. Tasks in JITIQA

  5. Predicted Challenges in WOZ • We imagined the following about the assessor • Asks complex questions, compared to TREC • Sample showed < 1/3 with “known” answer types • Wants fast responses • Assumes dialogue context (pronouns, ellipses) • Has no knowledge of question formulation • Assumes the QA system’s collection contains answer

  6. Preparation for WOZ • Extend work in factual QA with two approaches • Information/knowledge-centric • Create a Question/Answer Database (QADB) • Develop a question similarity metric • Build higher quality domain-specific document collections • User-centric • Reformulate questions to resolve references and incorporate context • Decompose complex questions into simpler ones

  7. Question/Answer Database • Because domain is closed, we may be able to predict questions and collect answers • How well can we cover the range of possible questions? • Process: • 1. Split up topics between developers • 2. Generate question and answer records • 3. Rotate topics among developers

  8. Domain #recs Afghanistan 24 Africa 35 Black Sea 31 Colombia 40 Indonesia 57 Ivory Coast 17 Japan 33 Microsoft 27 Sanchez 37 Surgery 33 QADB Population • For 10 domains, collected 334 question records, each with answers from multiple sources • Perform retrieval of answers by computing question similarity based on concepts

  9. Question Concepts • Q: “Why does so much opium production take place in Afghanistan?” • Concept 1: cause • Concept 2: popularity • Concept 3: produced • Other questions satisfying 100% of the concepts • Why is so much opium produced in Afghanistan? • Why is poppy farming popular in Afghanistan? • For what reasons is growing poppy common in Afghanistan? • What causes poppy farming to be so popular in Afghanistan? • What makes opium farms so commonplace in Afghanistan?

  10. Document Collection • Reasons for document collection • Alternative to slow Internet searches • Pre-filtering documents for domain relevance • Internet information is of low quality • Keeps experiment repeatable

  11. Documents Collected • News source collections • Documents from major newspapers with dates • Collected with one general query per domain to catch all possibly relevant documents • Used in pilot • Web source collections • Generally poor quality documents • Multiple specific queries used per domain, saving top 500 documents each time • Not used in pilot

  12. Performance in Pilot • Assessors in pilots 1 and 2 graded our dialog based on several performance measures for each domain • Scale: 1-7 with 7 representing “completely satisfied”

  13. Domain Full Part None Total Opium/Afghanistan 3 2 1 6 AIDS/Africa 3 2 3 8 Black Sea Pollution 5 2 7 FARC/Colombia 2 3 5 Indonesian Economy 8 3 2 13 Cell Phones/Ivory Coast 3 6 2 11 Joint Ventures/Japan 2 4 6 Microsoft/Viruses 3 1 4 Elizardo Sánchez 5 1 6 Robotic Surgery 5 4 9 Total 39 16 20 75 QADB Performance • Number of questions QADB answered fully, partially, or not at all, for both pilot experiments combined

  14. Complex Questions • Complex questions require mapping into simpler questions • “Biographical information needed on Elizardo Sanchez, Cuban dissident” • When and where was Elizardo Sanchez born? • Where did Elizardo Sanchez go to school? • Who is in Elizardo Sanchez’s family? • “Give some information on uses of robotic surgery in US and foreign countries?” • What kinds of surgeries do robots perform? • What laws govern robotic surgery in the US? • What are the benefits of robotic surgery?

  15. Complex Answer Types • System only recognizes simple answer types Money - How much money can be made from opium smuggling? Locations - What countries are involved in fighting Afghan opium production? Date - When did opium production begin in Afghanistan? • Most questions sought complex answers Cause – Why does so much opium production take place in Afghanistan? Action – What is being done to fight opium production? Effects – How have recent events affected opium production? Problems – What problems do counter narcotics face in Afghanistan? Policy – What is the United States’ financial commitment to drug control efforts?

  16. Other Challenging Questions • Ambiguous questions • “During the years 1998-2001 what was Indonesia’s currency exchange rate?” • What measure is desired? An average? Each year? • Follow-up questions • A: “Kim currently works three days a week at WIN-TECH, a three-company joint venture...” • Q: “who is kim?” • Misleading questions • Misspellings, slang, capitalization, statements

  17. Current Work • Automatic complex question break-down • Specification of general terms • “What elements cause Black Sea pollution?” • Expand elements into companies, countries, chemicals • Decomposition into members • “What about Sherron Watkins’ family?” • Decompose family into parents, children, spouse • Dialog context understanding • Coreference of anaphors

  18. Future Plans • Enhanced transformations of complex questions into simple ones • Enhanced incorporation of context • Ellipsis resolution • Recognition of intensions • Automatic and interactive generation of topic knowledge and QADB population

More Related