1 / 36

PLIS : a Probabilistic Lexical Inference System

PLIS : a Probabilistic Lexical Inference System Eyal Shnarch , Ido Dagan, Jacob Goldberger. The entire talk in a single sentence. lexical textual inference. principled probabilistic model . improves state-of-the art. Outline. lexical textual inference.

yon
Download Presentation

PLIS : a Probabilistic Lexical Inference System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PLIS : a Probabilistic Lexical Inference System Eyal Shnarch,Ido Dagan, Jacob Goldberger PLIS - Probabilistic Lexical Inference System

  2. The entire talk in a single sentence • lexical textual inference • principled probabilistic model • improves state-of-the art PLIS - Probabilistic Lexical Inference System

  3. Outline • lexical textual inference • principled probabilistic model improves state-of-the art 1 2 3 PLIS - Probabilistic Lexical Inference System

  4. lexical textual inference • principled probabilistic model improves state-of-the art 1 2 3 PLIS - Probabilistic Lexical Inference System

  5. Textual entailment • T entails Hif typically a human reading T • would infer that H is most likely true Text Hypothesis At Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo in Belgium Napoleon was defeated • lexical textual inference • principled probabilistic model • improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  6. Textual inference – useful in many NLP apps At Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo Napoleon engaged in a series of wars, and won many Napoleon was Emperor of the French from 1804 to 1815. Napoleon was not tall enough to win the Battle of Waterloo In the Battle of Waterloo, 18 Jun 1815, the French army, led by Napoleon, was crushed. in Belgium Napoleon was defeated • lexical textual inference • principled probabilistic model • improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  7. BIU NLP lab ChayaLiebeskind PLIS - Probabilistic Lexical Inference System

  8. Lexical textual inference • Complex systems use parser • Lexical inference rules link terms from T to H • Lexical rules come from lexical resources • H is inferred from Tiff all its terms are inferred 1st or 2nd order co-occurrence In the Battle of Waterloo, 18 Jun 1815, the French army, led by Napoleon, was crushed. in Belgium Napoleon was defeated Text Hypothesis • lexical textual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  9. Textual inference for ranking 1 2 At Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo Napoleon engaged in a series of wars, and won many Napoleon was Emperor of the French from 1804 to 1815. Napoleon was not tall enough to win the Battle of Waterloo In the Battle of Waterloo, 18 Jun 1815, the French army, led by Napoleon, was crushed. In which battle was Napoleon defeated? 3 5 4 • lexical textual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  10. Ranking textual inference – prior work • Syntactic-based methods • Transform T’s parsed tree into H’s parsed tree • Based on principled ML model • (Wang et al. 07,Heilman and Smith 10, Wang and Manning 10) • Heuristic lexical methods • Fast, easy to implement, highly competitive • Practical across genres and languages • (MacKinlay and Baldwin 09, Clark and Harrison 10, • Majumdar and Bhattacharyya 10) • lexical textual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  11. Lexical entailment scores – current practice • principled probabilistic model • Count covered/uncovered • (Majumdar and Bhattacharyya, 2010; Clark and Harrison, 2010) • Similarity estimation • (Corley and Mihalcea, 2005; Zanzotto and Moschitti, 2006) • Vector space • (MacKinlay and Baldwin, 2009)  Mostly heuristic PLIS - Probabilistic Lexical Inference System

  12. lexical textual inference • principled probabilistic model • improves state-of-the art 1 2 3 PLIS - Probabilistic Lexical Inference System

  13. Probabilistic model – overview t1 t2 t3 t4 t5 t6 T Battle of Waterloo French army led by Napoleon was crushed knowledge integration h1 h2 h3 H which battle was Napoleon defeated term-level x1 x2 x3 sentence-level annotations are available at sentence-level only • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  14. Knowledge integration • Distinguish resources reliability levels • WordNet >> similarity-based thesauri (Lin, 1998; Pantel and Lin, 2002) • Consider transitive chains length • The longer a chain is the lower its probability • Consider multiple pieces of evidence • More evidence means higher probability Battle of Waterloo French army led by Napoleon was crushed rule1 r is a rule multiple evidence transitive chain t rule2 which battle was Napoleon defeated PLIS - Probabilistic Lexical Inference System

  15. lexical textual inference • 1st approach • Forward • improves state-of-the art 1 2 3 PLIS - Probabilistic Lexical Inference System

  16. Probabilistic model – term level t1 t2 t3 t4 t5 t6 T Battle of Waterloo French army led by Napoleon was crushed r is a rule multiple evidence OR t' is the reliability level of the resource which suggested r h1 h2 h3 H which battle was Napoleon defeated this level parameters: one per input lexical resource ACL 11 short paper • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  17. Probabilistic model – overview T Battle of Waterloo French army led by Napoleon was crushed knowledge integration H which battle was Napoleon defeated term-level sentence-level • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  18. Probabilistic model – sentence level h1 h2 h3 H which battle was Napoleon defeated x1 x2 x3 we define hidden binary random variables: xt = 1 iff ht is inferred from T (zero otherwise) AND • Modeling with AND gate: • Most intuitively • However • Too strict • Does not model terms dependency HN-PLM y final sentence-level decision • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  19. Probabilistic model – sentence level h1 h2 h3 H which battle was Napoleon defeated xt = 1 iff ht is inferred by T (zero otherwise) x1 x2 x3 we define another binary random variable: yt – inference decision for the prefix h1… ht P(yt= 1) is dependent on yt-1 and xt y1 y2 y3 final sentence-level decision M-PLM Markovian - Probabilistic this level parameters Lexical Model • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  20. M-PLM – inference h1 h2 h3 H which battle was Napoleon defeated x1 x2 x3 y1 y2 y3 can be computed efficiently with a forward algorithm qij(k) final sentence-level decision • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  21. Parametersresource M-PLM – summary ObservedLexical rules which link terms M-PLM limits • Processing is term order dependent • does not normalize well across hypotheses length Annotation final sentence-level decision • Learning • we developed EM scheme to jointly learn all parameters • Hidden • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  22. lexical textual inference • 2nd approach • Backward • improves state-of-the art 1 2 3 PLIS - Probabilistic Lexical Inference System

  23. The main idea: each resource is a witness t1 t2 t3 t4 t5 t6 Battle of Waterloo French army led by Napoleon was crushed t' h1 h2 h3 which battle was Napoleon defeated • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  24. Backward witnesses model t1 t2 t3 t4 t5 t6 Battle of Waterloo French army led by Napoleon was crushed h1 h2 h3 which battle was Napoleon defeated x1 x2 x3 AND Likelihood y • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  25. Advantages of the second approach Inference: • Hypothesis length is not an issue • Learn from non-entailing resources • and provide a recall and precision estimation for a resource • lexicaltextual inference • principled probabilistic model • Improves state-of-the-art PLIS - Probabilistic Lexical Inference System

  26. lexical textual inference • principled probabilistic model • improves state-of-the art 1 2 3 PLIS - Probabilistic Lexical Inference System

  27. Evaluations – data sets Ranking in passage retrieval for QA (Wang et al. 07) 5700/1500 question-candidate answer pairs from TREC 8-13 Manually annotated Notable line of work from recent years: Punyakanoket al. 04, Cui et al. 05, Wang et al. 07, Heilmanand Smith 10, Wang and Manning 10 Recognizing textual entailment within a corpus 20,000 text-hypothesis pairs in each RTE-5, RTE-6 Originally constructed for classification • lexicaltextual inference • principled probabilistic model • improves sate-of-the-art PLIS - Probabilistic Lexical Inference System

  28. Evaluations – baselines Syntactic generative models • Require parsing • Apply sophisticated machine learning methods (Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilmanand Smith 10, Wang and Manning 10) Lexical model – Heuristically Normalized-PLM • AND-gate for the sentence-level • Add heuristic normalizations to addresses its disadvantages (TextInfer workshop 11) • Performance in line with best RTE systems HN-PLM • lexicaltextual inference • principled probabilistic model • improvessate-of-the-art PLIS - Probabilistic Lexical Inference System

  29. QA results – syntactic baselines • lexicaltextual inference • principled probabilistic model • improvessate-of-the-art PLIS - Probabilistic Lexical Inference System

  30. QA results – syntactic baselines + HN-PLM +0.7% +1% • lexicaltextual inference • principled probabilistic model • improvessate-of-the-art PLIS - Probabilistic Lexical Inference System

  31. QA results – baselines + M-PLM +3.2% +3.5% M-PLM • lexicaltextual inference • principled probabilistic model • improvessate-of-the-art PLIS - Probabilistic Lexical Inference System

  32. RTE results – M-PLM vs. HN-PLM +1.9% +7.3% +3.6% +6.0% • lexicaltextual inference • principled probabilistic model • improvessate-of-the-art PLIS - Probabilistic Lexical Inference System

  33. Future work:Cross Lingual Textual Entailment Battle of Waterloo French army led by Napoleon was crushed English monolingual English-Italian phrase table Italian monolingual quale battaglia fu sconfitto Napoleone • lexicaltextual inference • principled probabilistic model improvesstate-of-the-art PLIS - Probabilistic Lexical Inference System

  34. summary Clean probabilistic lexical model • As a lexical component or as a stand alone inference system • Superiority of principled methods over heuristic ones • Attractive passage retrieval ranking method • Code available– Kateryna has already took it – be our next user! Thank You • lexicaltextual inference • principled probabilistic model improvesstate-of-the-art PLIS - Probabilistic Lexical Inference System

  35. PLIS - Probabilistic Lexical Inference System

  36. Demo examples: [Bap,WN] no transitivity Jack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid [WN,Wiki] <show graph> Barak Obama's Buick got stuck in Dublin in a large Irish crowd United_States_President's car got stuck in Ireland, surrounded by many people Barak Obama - WN is out of date, need a new version of Wikipedia Bill_Clinton's Buick got stuck in Dublin in a large Irish crowd United_States_President's car got stuck in Ireland, surrounded by many people ------------------------------------------------------------------------------ [Bap,WN] this time with <transitivity & multiple evidence> Jack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid [VO,WN,Wiki] in the Battle_of_Waterloo the French army led by Napoleon was crushed in which battle Napoleon was defeated? ------------------------------------------------------------------------------ [all] 1. in the Battle_of_Waterloo the French army led by Napoleon was crushed 72% 2. Napoleon was not tall enough to win the Battle_of_Waterloo 47% 3. at Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo 34% 4. Napoleon engaged in a series of wars, and won many 47% 5. Napoleon was Emperor of the French from 1804 to 1815 9% [a bit long run] PLIS - Probabilistic Lexical Inference System

More Related