1 / 43

Language Games and Other Meaningful Pursuits

Language Games and Other Meaningful Pursuits. Michael L. Littman mlittman@research.att.com AT&T Labs  Research (sort of). Motivation. Software that can understanding language. Question answering Constructing databases from text Natural-language interaction

ducharme
Download Presentation

Language Games and Other Meaningful Pursuits

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Language Games andOther Meaningful Pursuits Michael L. Littman mlittman@research.att.com AT&T LabsResearch (sort of) Language Games

  2. Motivation Software that can understanding language. • Question answering • Constructing databases from text • Natural-language interaction • Automatic summarization/briefing Semantics tricky; word meanings informal. Language Games

  3. What is Meaning? I don’t really like bananas, but have long respected their humorous potential. How capture this sentence? • Predicate-argument structure? • Bag of words plus associated terms? Evaluate by use. Might need multiple reps. or none at all. Can we make incremental progress? Language Games

  4. Language Games Like other games: • Evaluation process clean. • Challenging (and fun!) for people. Unlike logical games: • Meaning matters! • No closed world assumption; messy. • Learning necessary... moving target. Machine performance far from humans’. Language Games

  5. Word Games Super-human performance common: • Scrabble™: Maven, near-perfect (Sheppard 02) • Boggle™: millisecond solutions (Boyan 98) • Hangman (Littman 00) • 99.97% 9-letter words under 5 guesses • 1.35 misses on average Language Games

  6. Language Game Examples • Crossword puzzles Acrostics • Cryptic crosswords Wheel of Fortune™ • Trivial Pursuit™ Tribond™ • College Bowl Taboo™ • Who Wants to Be a Scattergories™ Millionaire™ • Weakest Link™ Dictionary • Synonyms/Analogies Language Games

  7. Word meaning matters: • Pages semantically relevant to query Multiple methods of determining relevance: • query words near each other on page • query words appear anywhere on page Successful theme: combine • precision high, coverage low • precision low, coverage high Language Games

  8. Combination Rule Module set: coverage ci and precision pi Let Rjbe the modules selecting response j. • Pick argmaxjPiin RjpiPiin URj’(1-pi) If modules have independent precision: • Expected accuracy beats maxicipi. Language Games

  9. Crossword Puzzles (Shortz 98) ThesaurusCut off _ _ _ _ _ _ _ _ Puns & WordplayMonk’s head? _ _ _ _ _ Arts & Literature“Foundation Trilogy” author _ _ _ _ _ _ Popular CulturePal of Pooh _ _ _ _ _ _ EncyclopedicMountain known locally as Chomolungma _ _ _ _ _ _ _ CrosswordeseKind of coal or coat _ _ _ ISOLATED ABBOT ASIMOV TIGGER EVE REST PEA Language Games

  10. PROVERB: System Design Candidate generation(Keim et al. 99) • Like information retrieval: clue implies target • Variety of approaches used simultaneously Merging • Like meta search engine: create master list Grid filling(Shazeer et al. 99) • Like constraint satisfaction: fit answer to grid • Solved via probabilistic optimization Language Games

  11. Modules: ClueDB Collection of published clues. Nymph pursuer: SATYR Bugs pursuer: ELMER Nymph chaser: SATYR Place for an ace: SLEEVE Highball ingredient: RYEHighball ingredient: ICE exact: Highball ingredient: RYE partial: Ace place?: SLEEVE TransModule: Bugs chaser: ELMER X chaser Xpursuer Language Games

  12. Modules: Special Purpose Database modules: Transform clue to DB query. imdb: Warner of Hollywood: OLAND wordnet: Fruitless: ARID Syntactic: Variations of fill-in-the-blanks. blanks_movies: “Heavens ____!”: ABOVE Web search: Not used in experimental system. google: “The Way To Natural Beauty” author, 1980: TIEGS Language Games

  13. Modules: Backstops Word lists: Ignore clue, return all words. wordList: 10,000 words, perhaps: NOVELETTE Implicit modules: Probability distributions over all strings of words (e.g., bigram). segmenter: 1934 Hall and Nordhoff adventure novel: PITCAIRNISLAND Language Games

  14. Grid Filling Language Games

  15. Grid Filling Language Games

  16. Grid Filling Language Games

  17. Grid Filling Language Games

  18. Grid Filling Language Games

  19. Grid Filling Language Games

  20. Grid Filling Language Games

  21. Grid Filling Language Games

  22. Grid Filling Language Games

  23. Grid Filling Language Games

  24. Grid Filling Language Games

  25. Final: 88% words, 97% letters Language Games

  26. PROVERB Results Test collection • 95% words, 98% letters, 46% puzzles • NYT: 89.5% (95.5% MTW, 85.0% TFSS) American Crossword Puzzle Tournament • 1998: 190/251, 80% words (vs. 100%) • tricks: letter pairs, words in single square • 1999: 147/261, 75% words • tricks: cloonerized spues! spoonerized clues! Language Games

  27. Cryptic Crosswords Popular in UK A clue like this about Japanese religion (6) SHINTO HINT SO = about( HINT,SO) Language Games

  28. Crossword Maestro (Tunstall-Pedoe) Solves this clue: `japanese religion' is the definition. I am not sure about the `japanese' bit but `Shinto' can be an answer for `religion'. `a clue like this about' is the subsidiary indication. `a clue‘ becomes `hint', `like this' becomes `so', `about' means one lot of letters goes inside another. `hint' placed inside `so' is `shinto'. http://www.genius2000.com/cm.html Language Games

  29. CM Analysis CM “knows”: • Shinto is a religion (although not Japanese) • Back-off: “Japanese religion” kind of religion. • other classes: manmade objects, bodily activities, natural acts, verbs in base form, adjectives… • “a clue” is hint, “like this” is so • “about” indicates containment (game knowl.) Wordnet? http://www.cogsci.princeton.edu/~wn/ Language Games

  30. CM Results Ads claim 75% on individual clues (perhaps full puzzles). New York Times puzzles, under 5%. Very sophisticated: • NL explanations • broad coverage • real time Language Games

  31. Question Answering Blend of IR and NLP. • query, tag type • find relevant passages • extract entities, sort by score (combination) Language Games

  32. Who invented the first computer? • The stunning shortsightedness and obtusity of XEROX's top-level suits has been well anatomized in “Fumbling The Future: How XEROX Invented, Then Ignored, the First Personal Computer” by Douglas K. Smith and Robert C. Alexander (William Morrow & Co., 1988, ISBN 0-688-09511-9). • 1876 Alexander Graham Bell invented the telephone. 1958 First electronic computers were built. • Steve Jobs and Steve Wozniak were two young men who invented the first-ready made computer, the Apple. • A Brief History of Computers and Data Communications. … 1820 Charles Babbage invented the difference engine … 1879 First private telephone switch board was installed. Language Games

  33. Trivial Pursuit™ Race around board, answer questions. Categories: Geography, Entertainment, History, Literature, Science, Sports Language Games

  34. Wigwam QA via AQUA (Abney et al. 00) • back off: word match in order helps score. • “When was Amelia Earhart's last flight?” • 1937, 1897 (birth), 1997 (reenactment) Move selection via MDP (Littman 00) • Estimate category accuracy. • Minimize expected turns to finish. Language Games

  35. Simulation Results Language Games

  36. TOEFL Synonyms Used in college applications. fish • scale • angle • swim • dredge Language Games

  37. Synonym Approaches Latent Semantic Indexing (Landauer & Dumais 97) • Analyze 30k paragraphs, 300d embedding • 64% ~Boulder http://lsa.colorado.edu/ Pointwise Mutual Information-IR (Turney 01) • Counts in Altavista (350M); 74% (us: 77%) Thesaurus • http://Wordsmyth.net: 98% prec.; 74% cov. • Combine with PMI-IR: 92% Language Games

  38. Verbal Analogies Used in college boards (SATs, GREs), and as an intelligence test. cat : meow :: • mouse : scamper • bird : peck • dog : bark • horse : groom • lion : scratch Language Games

  39. Preliminary Results Undergraduate project at Princeton • 17 groups • 229 examples (ETS, web, class) • 76 test problems from SATs Language Games

  40. Analogy Results: SATs Three groups sig. better than chance. Language Games

  41. Some Successful Techniques Individual groups used combinations: • Wordnet for synonyms, antonyms, is-a • web search with “linking words” • x (characterized by OR possesses OR possessed OR possess OR characterizes) y • automatically identify linking words • word association in definitions Language Games

  42. Conclusion Language games interesting, challenging. Progress on solving “meaning” problems. • Word order • Words, as a backoff • Meaning-preserving transformations • Probabilistic models • Combinations! Exciting learning and reasoning to be done Language Games

  43. Challenges What year was Dartmouth College founded? Dr. Koop graduated from DartmouthCollege in 1937 and received his M.D. degree from Cornell Medical College in 1941. Although DartmouthCollege was established in 1769 to educate American Indians and… Language Games

More Related