1 / 28

Provably Reliable QA Interfaces

Provably Reliable QA Interfaces. http://cs.washington.edu/research/nli. Outline. Motivation Reliable NLIs Semantic tractability theory Implementation and experiments Joint work with Ana-Maria Popescu, Alex Yates, Henry Kautz, and Dan Weld. I. The Dominant UI Paradigm.

lalasa
Download Presentation

Provably Reliable QA Interfaces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Provably Reliable QA Interfaces http://cs.washington.edu/research/nli

  2. Outline • Motivation • Reliable NLIs • Semantic tractability theory • Implementation and experiments Joint work with Ana-Maria Popescu, Alex Yates, Henry Kautz, and Dan Weld. Etzioni, 473

  3. I. The Dominant UI Paradigm • Menu buttons • Hyperlinks • Remote controls Just Click it! Etzioni, 473

  4. What to click? Etzioni, 473

  5. Clicking breaks when.. • Click-challenged • Hands busy (driving). • Disabled. • Limited Screen/keyboard Real estate • Cell phones, Microwave. • Ubiquitous computing.. • Complexity: SQL, shell scripts, menu hell. Etzioni, 473

  6. Alternative Interface Paradigm Speech + NL Understanding + Agents. • “Where is Lord of the Rings showing?” • “Defrost my corn.” • “Delete all my old messages except the ones from Mom.” Substantial Research Challenges! Etzioni, 473

  7. State of the Art • Commercial Speech Systems allow single word input. • NLIDBs are unreliable (try Microsoft’s English Query). • Nontrivial Autonomous Agents that respond to complex requests?! Etzioni, 473

  8. II. Our Focus • Not speech. • 1991 – 1997: built softbots. • Today: Reliable Natural Language Interfaces Etzioni, 473

  9. Why Reliable? Imagine “intelligent” interfaces that.. • Sometimes delete the wrong file. • Can report incorrect flight times. AI cannot be an excuse for incompetence (Norman, Schneiderman). Etzioni, 473

  10. Some Common Objections • Can’t the interface just confirm? • Yes, but will users attend? • Speech understanding isn’t reliable. • That will gradually change. • Still need reliable language module. • NL understanding is AI-complete! Etzioni, 473

  11. III. Semantics is hard • Syntactic, scopal ambiguity • Word-sense ambiguity • Time, events, liquids, holes,… • Shakespeare, Faulkner,… • Discourse, Pragmatics (speech acts) We have to think about tractable classes! Etzioni, 473

  12. Our Basic Hypothesis There are common situations where Semantic Interpretation is tractable. Sentence  Target expression Etzioni, 473

  13. Our Research Strategy • Identify easy-to-understand NL sentences. • Develop Taxonomic Theory of Semantic Tractability. • Build Reliable NLIs. • Test NLIs experimentally. Etzioni, 473

  14. Semantic Tractability Easy to understand questions: What are the Chinese restaurants in Seattle? What Microsoft jobs require 2 years of experience? What rivers run through Texas? Semantically tractable questions are quite common: 77.5% - 97% Etzioni, 473

  15. Semantic Intractability Q: What is the second highest mountain in the US? A: The word ‘second’ is unknown; please rephrase your query. Q: What are the states bordering the states bordering the states bordering Montana? A: huh? Etzioni, 473

  16. Semantically Tractable Qs Given a lexicon and a parse, IF • Q contains at least one wh-word. • There exists a valid mapping from Q to a set of database elements. • Q maps to a nonrecursive datalog clause. THENQ is Semantically Tractable. Etzioni, 473

  17. Valid Mapping Words (phrases) DB elements. • Lexical constraints: Lord of the Rings. • Attachment constraints: • What is the population of Atlanta? • Semantic constraints: • Attributes need values: Cuisine  Chinese. • Values can have implicit attributes. Etzioni, 473

  18. Guarantees on ST Qs. Precise = NLIDB implementation. • Precise can detect ST Qs. • Precise is sound for ST Qs. • Precise is complete for ST Qs. Will Precise capture “user intent?” Etzioni, 473

  19. IV. Precise Implementation • Lexicon extracted from DB + Wordnet. • Parser plug in (Charniak 2000). • Semantic constraints via graph match. • Word1 DB_EL1 • Phrase2 DB_EL2 DB_EL3 Match is computed via maxflow. Etzioni, 473

  20. PRECISE graph representation DB Values DB Attributes Attribute Tokens Value Tokens Movies = what Movies film What Actor = Woody Allen Actor E S Woody Allen Director = Woody Allen Director I K 2 Paul Mazursky Director = Paul Mazursky “What are the Paul Mazursky films with Woody Allen?” Etzioni, 473

  21. PRECISE Architecture Database Lexicon Parser Tokenizer Matcher Query Generator Equivalence Checker English Question SQL Query + Answer Set Etzioni, 473

  22. Ambiguity meets Reliability • Precise computes all valid mappings. • Many possibilities  clarifying Q. • What is the population of New York? • The population of New York City is… • The population of New York State is.. Etzioni, 473

  23. Experiments Systems Mooney, PRECISE, EnglishQuery Datasets Sets of NL questions labeled with SQL queries Geography (846) Jobs (577) Restaurants (224) Measure PRECISE’s performance Prevalence of semantically tractable questions Etzioni, 473

  24. Fraction Answered Recall = Qanswered/Q Recall > 75 % Etzioni, 473

  25. Error Rate Precision = Qcorrect/Qanswered PRECISE made no mistakes on semantically tractable questions Etzioni, 473

  26. Exact – Case Study • Exact is an NLI to the Panasonic KX-TC1040W telephone/answering machine • It uses Precise to formulate goals for the Blackbox planner (Kautz & Selman) • The database model for the phone has 5 relations (between 2 and 11 attributes each) and the action set includes 37 actions. Etzioni, 473

  27. Exact – Diagram Etzioni, 473

  28. Conclusion • Click-it UI paradigm is fraying. • Need reliable NLIs. • Semantically Tractable sentences. • Taxonomic Theory. • Precise and Exact implementations. • Promising, preliminary experiments. Etzioni, 473

More Related