Just-in-Time Interactive Question Answering

1. Just-in-Time Interactive Question Answering Language Computer Corporation Sanda Harabagiu, PI John Lehmann John Williams Paul Aarseth

2. Overview • Project Introduction • Preparation for “Wizard of Oz” pilot • Performance in WOZ pilot • Challenges encountered in WOZ pilot • Current work and future plans

3. Research Project Objective • Address the interactive aspect of QA systems by designing and implementing a dialog shell that can be used with any QA system

4. Tasks in JITIQA

5. Predicted Challenges in WOZ • We imagined the following about the assessor • Asks complex questions, compared to TREC • Sample showed < 1/3 with “known” answer types • Wants fast responses • Assumes dialogue context (pronouns, ellipses) • Has no knowledge of question formulation • Assumes the QA system’s collection contains answer

6. Preparation for WOZ • Extend work in factual QA with two approaches • Information/knowledge-centric • Create a Question/Answer Database (QADB) • Develop a question similarity metric • Build higher quality domain-specific document collections • User-centric • Reformulate questions to resolve references and incorporate context • Decompose complex questions into simpler ones

7. Question/Answer Database • Because domain is closed, we may be able to predict questions and collect answers • How well can we cover the range of possible questions? • Process: • 1. Split up topics between developers • 2. Generate question and answer records • 3. Rotate topics among developers

8. Domain #recs Afghanistan 24 Africa 35 Black Sea 31 Colombia 40 Indonesia 57 Ivory Coast 17 Japan 33 Microsoft 27 Sanchez 37 Surgery 33 QADB Population • For 10 domains, collected 334 question records, each with answers from multiple sources • Perform retrieval of answers by computing question similarity based on concepts

9. Question Concepts • Q: “Why does so much opium production take place in Afghanistan?” • Concept 1: cause • Concept 2: popularity • Concept 3: produced • Other questions satisfying 100% of the concepts • Why is so much opium produced in Afghanistan? • Why is poppy farming popular in Afghanistan? • For what reasons is growing poppy common in Afghanistan? • What causes poppy farming to be so popular in Afghanistan? • What makes opium farms so commonplace in Afghanistan?

10. Document Collection • Reasons for document collection • Alternative to slow Internet searches • Pre-filtering documents for domain relevance • Internet information is of low quality • Keeps experiment repeatable

11. Documents Collected • News source collections • Documents from major newspapers with dates • Collected with one general query per domain to catch all possibly relevant documents • Used in pilot • Web source collections • Generally poor quality documents • Multiple specific queries used per domain, saving top 500 documents each time • Not used in pilot

12. Performance in Pilot • Assessors in pilots 1 and 2 graded our dialog based on several performance measures for each domain • Scale: 1-7 with 7 representing “completely satisfied”

13. Domain Full Part None Total Opium/Afghanistan 3 2 1 6 AIDS/Africa 3 2 3 8 Black Sea Pollution 5 2 7 FARC/Colombia 2 3 5 Indonesian Economy 8 3 2 13 Cell Phones/Ivory Coast 3 6 2 11 Joint Ventures/Japan 2 4 6 Microsoft/Viruses 3 1 4 Elizardo Sánchez 5 1 6 Robotic Surgery 5 4 9 Total 39 16 20 75 QADB Performance • Number of questions QADB answered fully, partially, or not at all, for both pilot experiments combined

14. Complex Questions • Complex questions require mapping into simpler questions • “Biographical information needed on Elizardo Sanchez, Cuban dissident” • When and where was Elizardo Sanchez born? • Where did Elizardo Sanchez go to school? • Who is in Elizardo Sanchez’s family? • “Give some information on uses of robotic surgery in US and foreign countries?” • What kinds of surgeries do robots perform? • What laws govern robotic surgery in the US? • What are the benefits of robotic surgery?

15. Complex Answer Types • System only recognizes simple answer types Money - How much money can be made from opium smuggling? Locations - What countries are involved in fighting Afghan opium production? Date - When did opium production begin in Afghanistan? • Most questions sought complex answers Cause – Why does so much opium production take place in Afghanistan? Action – What is being done to fight opium production? Effects – How have recent events affected opium production? Problems – What problems do counter narcotics face in Afghanistan? Policy – What is the United States’ financial commitment to drug control efforts?

16. Other Challenging Questions • Ambiguous questions • “During the years 1998-2001 what was Indonesia’s currency exchange rate?” • What measure is desired? An average? Each year? • Follow-up questions • A: “Kim currently works three days a week at WIN-TECH, a three-company joint venture...” • Q: “who is kim?” • Misleading questions • Misspellings, slang, capitalization, statements

17. Current Work • Automatic complex question break-down • Specification of general terms • “What elements cause Black Sea pollution?” • Expand elements into companies, countries, chemicals • Decomposition into members • “What about Sherron Watkins’ family?” • Decompose family into parents, children, spouse • Dialog context understanding • Coreference of anaphors

18. Future Plans • Enhanced transformations of complex questions into simple ones • Enhanced incorporation of context • Ellipsis resolution • Recognition of intensions • Automatic and interactive generation of topic knowledge and QADB population