1 / 34

Natural Logic and Natural Language Inference

Natural Logic and Natural Language Inference. Bill MacCartney Stanford University / Google, Inc. 8 April 2011. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion. Two disclaimers.

robinrobert
Download Presentation

Natural Logic and Natural Language Inference

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Natural Logic and Natural Language Inference Bill MacCartney Stanford University / Google, Inc. 8 April 2011

  2. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Two disclaimers • The work I present today isn’t exactly fresh • Essentially, it’s my dissertation work from 2009 • I hope it can usefully provide context for more recent work • I’m a computer scientist, not a semanticist or a logician • Consequently, I emphasize pragmatism over rigor

  3. Some Some no Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Natural language inference (NLI) • Aka recognizing textual ‘entailment’ (RTE) • Does premise P justify an inference to hypothesis H? • An informal, intuitive notion of inference: not strict logic • Emphasis on variability of linguistic expression P Every firm polled saw costs grow more than expected,even after adjusting for inflation. H Every big company in the poll reported cost increases. yes • Necessary to goal of natural language understanding (NLU) • Can also enable semantic search, question answering, …

  4. robust,but shallow deep,but brittle lexical/semanticoverlap Jijkoun & de Rijke 2005 FOL &theoremproving Bos & Markert 2006 patternedrelationextraction Romano et al. 2006 semantic graph matching MacCartney et al. 2006 Hickl et al. 2006 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion naturallogic (this work) NLI: a spectrum of approaches Solution? Problem:hard to translate NL to FOL idioms, anaphora, ellipsis, intensionality, tense, aspect, vagueness, modals, indexicals, reciprocals, propositional attitudes, scope ambiguities, anaphoric adjectives, non-intersective adjectives, temporal & causal relations, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, comparatives, phrasal verbs, … Problem:imprecise  easily confounded by negation, quantifiers, conditionals, factive & implicative verbs, etc.

  5. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion What is natural logic? ( natural deduction) • Characterizes valid patterns of inference via surface forms • precise, yet sidesteps difficulties of translating to FOL • A long history • traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, … • the term “natural logic” was introduced by Lakoff (1970) • van Benthem & Sánchez Valencia (1986-91): monotonicity calculus • Nairn et al. (2006): an account of implicatives & factives • We introduce a new theory of natural logic • extends monotonicity calculus to account for negation & exclusion • incorporates elements of Nairn et al.’s model of implicatives • …and implement & evaluate a computational model of it

  6. Yesentailment Nonon-entailment 2-wayRTE1,2,3 Yesentailment Unknowncompatibility Nocontradiction 3-wayFraCaS, PARC, RTE4 Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion P = Qequivalence P < Qforwardentailment P > Qreverseentailment P # Qnon-entailment containmentSánchez-Valencia ‘Entailment’ relations in past work

  7. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion 16 elementary set relations Assign sets x, y to one of 16 relations, depending on emptiness or non-emptiness of each of four partitions y y x x empty non-empty

  8. x ^ y x‿y x⊐y xy Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion x⊏y x | y x # y 16 elementary set relations But 9 of 16 are degenerate: either x or y is either empty or universal. I.e., they correspond to semantically vacuous expressions, which are rare outside logic textbooks. We therefore focus on the remaining seven relations.

  9. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion The set of 7 basic entailment relations Relations are defined for all semantic types: tiny⊏small, hover⊏fly, kick⊏strike,this morning⊏today, in Beijing⊏in China, everyone⊏someone, all⊏most⊏some

  10. ? R S ? fish human nonhuman Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Joining entailment relations y x y z ⊏ | ^

  11. What is | | ? ⋈ Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion | |  {, ⊏, ⊐, |, #} ⋈ Some joins yield unions of relations!

  12. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion The complete join table Of 49 join pairs, 32 yield relations in ; 17 yield unions Larger unions convey less information — limits power of inference In practice, any union which contains # can be approximated by #

  13. atomic edit: DEL, INS, SUB compound expression entailment relation Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Lexical entailment relations x e(x) •  will depend on: • the lexical entailment relation generated by e: (e) • other properties of the context x in which e is applied (, ) • Example: suppose x is red car • If e is SUB(car, convertible), then (e) is ⊐ • If e is DEL(red), then (e) is ⊏ • Crucially, (e) depends solely on lexical items in e, independent of context x • But how are lexical entailment relations determined?

  14. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Lexical entailment relations: SUBs (SUB(x, y)) = (x, y) For open-class terms, use lexical resource (e.g. WordNet)  for synonyms: sofa couch, forbid  prohibit ⊏ for hypo-/hypernyms: crow⊏bird, frigid⊏cold, soar⊏rise | for antonyms and coordinate terms: hot| cold, cat | dog  or | for proper nouns:USA United States, JFK | FDR # for most other pairs:hungry# hippo Closed-class terms may require special handling Quantifiers: all⊏some, some^no, no | all, at least 4‿at most 6 See paper for discussion of pronouns, prepositions, …

  15. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Lexical entailment relations: DEL & INS Generic (default) case: (DEL(•)) = ⊏, (INS(•)) = ⊐ • Examples: red car⊏car, sing⊐sing off-key • Even quite long phrases: car parked outside since last week⊏car • Applies to intersective modifiers, conjuncts, independent clauses, … • This heuristic underlies most approaches to RTE! • Does P subsume H? Deletions OK; insertions penalized. Special cases • Negation: didn’t sleep ^ did sleep • Implicatives & factives (e.g. refuse to, admit that): discussed later • Non-intersective adjectives: former spy | spy, alleged spy # spy • Auxiliaries etc.: is sleepingsleeps, did sleepslept

  16. How is (x, y) projected by f? @ @ x y f f ?  Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion The impact of semantic composition How are entailment relations affected by semantic composition? ] [ @ means fn application The monotonicity calculus provides a partial answer If f has monotonicity… But how are other relations (|, ^, ‿) projected?

  17. Each projectivity signature is a map ↦ Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A typology of projectivity Projectivity signatures: a generalization of monotonicity classes In principle, 77 possible signatures, but few actually realized

  18. ↦ Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A typology of projectivity Projectivity signatures: a generalization of monotonicity classes Each projectivity signature is a map In principle, 77 possible signatures, but few actually realized See my disseration for projectivity of various quantifiers, verbs

  19. @ @ ⊐ ⊐ ⊐ @ @ ⊏ ⊏ @ @ Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion @ @ nobody nobody can can without without clothes a shirt enter enter Projecting through multiple levels Propagate entailment relation between atoms upward, according to projectivity class of each node on path to root nobody can enter without a shirt⊏nobody can enter without clothes

  20. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Implicatives & factives [Nairn et al. 06] 9 signatures, per implications (+, –, or o) in positive and negative contexts

  21. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Implicatives & factives We can specify relation generated by DEL or INS of each signature Room for variation w.r.t. infinitives, complementizers, passivation, etc. Factives not fully explained: he didn’t admit that he knew | he didn’t know Some more intuitive when negated: he didn’t hesitate to ask | he didn’t ask

  22. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Putting it all together • Find a sequence of edits e1, …, en which transforms p into h. Define x0 = p, xn = h, and xi = ei(xi–1) for i [1, n]. • For each atomic edit ei: • Determine the lexical entailment relation (ei). • Project (ei) upward through the semantic composition tree of expression xi–1 to find the atomic entailment relation (xi–1, xi) • Join atomic entailment relations across the sequence of edits:(p, h) = (x0, xn) = (x0, x1) ⋈ … ⋈ (xi–1, xi) ⋈ … ⋈ (xn–1, xn) Limitations: need to find appropriate edit sequence connecting p and h;tendency of ⋈ operation toward less-informative entailment relations; lack of general mechanism for combining multiple premises Less deductive power than FOL. Can’t handle e.g. de Morgan’s Laws.

  23. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion An example P The doctor didn’t hesitate to recommend Prozac. H The doctor recommended medication. yes ‿ | | ^ ^ ⊏ ⊏ ⊏ ⊏ yes

  24. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Different edit orders? Intermediate steps may vary; final result is typically (though not necessarily) the same

  25. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Implementation & evaluation The NatLog system: an implementation of this model in code For implementation details, see [MacCartney & Manning 2008] Evaluation on FraCaS test suite 183 NLI problems, nine sections, three-way classification Accuracy 70% overall; 87% on “relevant” sections (60% coverage) Precision 89% overall: rarely predicts entailment wrongly Evaluation on RTE3 test suite Longer, more natural premises; greater diversity of inference types NatLog alone has mediocre accuracy (59%) but good precision Hybridization with broad-coverage RTE system yields gains of 4%

  26. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion :-) Thanks! Questions? Conclusions Natural logic is not a universal solution for NLI Many types of inference not amenable to natural logic approach Our inference method faces many limitations on deductive power More work to be done in fleshing out our account Establishing projectivity signatures for more quantifiers, verbs, etc. Better incorporating presuppositions But, our model of natural logic fills an important niche Precise reasoning on negation, antonymy, quantifiers, implicatives, … Sidesteps the myriad difficulties of full semantic interpretation Practical value demonstrated on FraCaS and RTE3 test suites

  27. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion Backup slides follow

  28. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion An example involving exclusion P Stimpy is a cat. H Stimpy is not a poodle. yes | | | ^ ^ ⊏ ⊐ ⊏ ⊏ yes

  29. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion An example involving an implicative P We were not permitted to smoke. H We smoked Cuban cigars. no ⊐ ⊏ ⊏ | ^ ^ | ⊐ ⊐ no

  30. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion de Morgan’s Laws for quantifiers P Not all birds fly. H Some birds do not fly. yes ^ ^ ^ ‿ ⊏ ⊏ ‿ ^ ⊏⊐‿# wtf??

  31. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion de Morgan’s Laws for quantifiers (2) P Not all birds fly. H Some birds do not fly. yes ^ ^ ^ | ^ ⊐ ⊏ ⊏ ⊏⊐‿# wtf??

  32. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A more complex example P Jimmy Dean refused to move without blue jeans. H James Dean didn’t dance without pants. yes

  33. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A more complex example (2) P Jimmy Dean refused to move without blue jeans. H James Dean didn’t dance without pants. yes

  34. Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion A more complex example (3) P Jimmy Dean refused to move without blue jeans. H James Dean didn’t dance without pants. yes

More Related