1 / 50

Hitting The Right Paraphrases In Good Time

Hitting The Right Paraphrases In Good Time. Stanley Kok Dept. of Comp. Sci. & Eng. Univ. of Washington Seattle, USA. Chris Brockett NLP Group Microsoft Research Redmond, USA. Motivation Background Hitting Time Paraphraser Experiments Future Work. Overview. 2. Motivation

kevyn
Download Presentation

Hitting The Right Paraphrases In Good Time

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hitting The Right Paraphrases In Good Time Stanley Kok Dept. of Comp. Sci. & Eng. Univ. of Washington Seattle, USA Chris Brockett NLP Group Microsoft Research Redmond, USA

  2. Motivation Background Hitting Time Paraphraser Experiments Future Work Overview 2

  3. Motivation Background Hitting Time Paraphraser Experiments Future Work Overview 3

  4. What’s a paraphrase of… Paraphrase System • “is friendly with” • “is a friend of” • … “is on good terms with” Applications • Query expansion • Document summarization • Natural language generation • Question answering • etc. 4

  5. What’s a paraphrase of… Paraphrase System Bilingual Parallel Corpora • “is friendly with” • “is a friend of” • … “is on good terms with” 5

  6. Bilingual Parallel Corpus …the cost dynamic is under control… …die kostenentwicklungunterkontrolle… …keep the cost in check… …die kostenunterkontrolle… … … Phrase Table 6

  7. BCB system [Bannard & Callison-Burch, ACL’05] P(E2|E1) ¼CGP(E2|G) P(G|E1) SBP system [Callison-Burch, EMNLP’08] P(E2|E1) ¼CG P(E2|G,syn(E1)) p(G|E1, syn(E1)) State of the Art 7

  8. Graphical View (unterkontrolle) P(E2|F2) F2 F1 G3 G2 G1 P(F2|E1) P(E2|G1) P(G1|E1) E4 E3 E2 E1 8 (in check) (under control)

  9. Random Walks Hitting Times Graphical View Path lengths > 2 General graph Add nodes to represent domain knowledge F2 F1 G3 G2 G1 E4 E3 E2 E1 9

  10. Motivation Background Hitting Time Paraphraser Experiments Future Work Overview 10

  11. Random Walk • Begin at node A • Randomly pick neighbor n E B D A A F C 11

  12. Random Walk • Begin at node A • Randomly pick neighbor n • Move to node n E 2 B D A F C 12

  13. Random Walk • Begin at node A • Randomly pick neighbor n • Move to node n • Repeat E B D A F 2 C 13

  14. Expected number of steps starting from node ibefore node jis visited for first time Smaller hitting time → closer to start node i Truncated Hitting Time [Sarkar & Moore, UAI’07] Random walks are limited to Tsteps Computed efficiently & with high probability by sampling random walks [Sarkar, Moore & Prakash ICML’08] Hitting Time from node i to j 14

  15. Finding Truncated Hitting Time By Sampling E B D A 1 F C T=5 A 15

  16. Finding Truncated Hitting Time By Sampling E B 4 D A F C T=5 A D 16

  17. Finding Truncated Hitting Time By Sampling E 5 B D A F C T=5 A D E 17

  18. Finding Truncated Hitting Time By Sampling E B 4 D A F C T=5 A D E D 18

  19. Finding Truncated Hitting Time By Sampling E B D A 6 F C T=5 A D E D F 19

  20. Finding Truncated Hitting Time By Sampling E 5 B D A F C T=5 A D E D F E 20

  21. Finding Truncated Hitting Time By Sampling E B hAE=2 hAB=5 hAD=1 hAA=0 D A F C hAC=5 hAF=4 T=5 A D E D F E 21

  22. Motivation Background Hitting Time Paraphraser Experiments Future Work Overview 22

  23. Hitting Time Paraphraser (HTP) English-German English-French German-French etc. HTP Paraphrase System Phrase Tables • “is friendly with” • “is a friend of” • … “is on good terms with” Phrase Paraphrases 23

  24. Graph Construction 24

  25. Graph Construction 25

  26. Graph Construction • BFS from query phrase up to depth d or up to max. number n of nodes • d = 6, n = 50,000 … … … … … … … … … 26

  27. Graph Construction 0.25 0.35 … … … … … … … … … 27

  28. Graph Construction 0.6 … … … … … … … … … 28

  29. Graph Construction 0.5 0.5 … … … … … … … … … 29

  30. Estimate Trunc. Hitting Times • Run mtruncated random walks to estimate truncated hitting time of each node • T = 10, m = 1,000,000 • Prune nodes with hitting times = T

  31. Add Ngram Nodes “reach the objective” “achieve the aim” “achieve the goal” … … “reach” “objective” “the” “achieve the” “the aim” 31

  32. Add “Syntax” Nodes “the objective is” “the aim is” “whose goal is” “what goal” start with article end with be start with interrogatives 32

  33. Add Not-Substring-Of Nodes “reach the objective” “reach the aim” “reach the” “objective” not-substring-of 33

  34. Feature Nodes phrase nodes p1 = 0.1 p2 p4 = 0.1 = 0.4 not-substring nodes ngram nodes p3 = 0.4 “syntax” nodes 34

  35. Re-estimate Truncated Hitting Times • Run mtruncated random walks again • Rank paraphrases in increasing order of hitting times 35

  36. Motivation Background Hitting Time Paraphraser Experiments Future Work Overview 36

  37. Europarl dataset [Koehn, MT-Summit’05] Use 6of 11 languages: English, Danish, German, Spanish, Finnish, Dutch About a million sentences per language English−Foreign phrasal alignments by giza++ [Callison-Burch, EMNLP’08] Foreign−Foreign phrasal alignments by MSR aligner Data 37

  38. SBP system [Callison-Burch, EMNLP’08] HTP with no feature node HTP with bipartite graph Comparison Systems 38

  39. NIST dataset 4 English translations per Chinese sentence 33,216 English translations Randomly selected 100 English phrases From 1-4grams in both NIST & Europarl datasets Exclude stop words, numbers, phrases containing periods and commas Evaluation Methodology 39

  40. For each phrase, randomly select a sentence from NIST dataset containing it Substituted top 1 to 10 paraphrases for phrase Methodology 40

  41. Manually evaluated resulting sentences 0: Clearly wrong; grammatically incorrect or does not preserve meaning 1: Minor grammatical errors (e.g., subject-verb disagreement; wrong tenses, etc.), or meaning largely preserved but not completely 2: Totally correct; grammatically correct and meaning is preserved Correct: 1 and 2; Wrong: 0 Two evaluators; Kappa = 0.62 (substantial agree.) Methodology 41

  42. HTP vs. SBP 0.71 0.53 12 11 10 11 12 10 2 6 4 5 6 7 4 2 3 2 5 7 1 4 3 5 3 1 6 2 4 9 1 1 5 7 8 1 2 8 3 2 1 4 3 2 1 3 1 7 5 3 2 1 8 7 4 5 3 6 4 5 6 7 9 8 3 5 6 4 2 p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p 49 50 100 51 49 1 50 50 1 50 50 51 1 1 51 49 49 51 50 49 51 49 1 51 1 51 2 1 51 2 1 2 1 2 1 100 1 1 50 100 1 1 100 2 1 100 1 1 100 2 100 1 49 100 49 2 51 49 1 51 49 49 51 2 49 51 49 42

  43. HTP vs. SBP 373 paraphrases per system 0.56 0.39 10 10 12 11 11 12 9 7 5 5 1 3 4 2 3 9 5 5 1 6 1 5 4 4 1 7 3 2 8 2 4 3 1 3 6 2 1 3 8 6 5 4 3 2 2 7 1 5 4 3 2 2 6 7 4 8 5 7 6 7 2 1 3 4 8 1 6 p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p 100 50 49 51 49 1 1 1 1 49 49 49 51 50 50 50 50 50 50 49 49 49 49 2 2 2 1 1 1 1 1 1 1 1 1 51 1 51 100 51 51 51 100 51 100 51 51 100 51 100 100 100 51 49 49 49 2 2 2 2 2 1 49 1 1 1 1 43

  44. HTP vs. SBP 0.54 10 11 12 11 12 10 3 7 1 5 7 6 7 2 6 2 3 2 6 2 6 5 1 5 8 6 7 4 5 9 5 1 3 4 3 1 4 9 6 2 3 3 2 2 4 8 5 5 4 3 1 2 4 1 7 1 3 8 4 1 2 8 1 4 5 7 3 p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p 100 49 50 49 51 50 50 50 50 51 49 49 51 49 1 1 1 1 50 1 50 49 49 49 2 2 2 1 1 1 1 1 1 1 49 1 100 1 51 51 51 100 51 51 100 51 100 100 51 51 100 1 100 49 49 49 49 2 2 2 2 2 1 1 1 1 51 483 paraphrases 44

  45. HTP vs. SBP 0.71 0.61 0.53 12 11 10 10 11 12 9 7 5 5 1 3 9 2 3 4 6 5 1 7 1 5 4 8 1 3 2 4 6 2 8 4 1 3 5 2 1 8 3 6 5 4 3 3 2 1 1 7 6 5 2 4 8 7 4 2 5 6 7 7 2 1 6 4 3 2 3 p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p 51 100 50 49 49 1 1 1 1 51 49 49 51 51 50 50 50 50 50 100 49 49 49 49 2 2 1 1 1 1 1 1 1 50 1 100 1 51 100 51 51 100 100 51 51 100 51 51 1 100 1 2 49 49 49 2 2 2 2 2 1 49 1 1 1 51 49 0.50 45

  46. HTP vs. SBP 0.54 0.43 0.39 10 11 12 11 12 10 2 2 3 6 7 1 6 3 4 5 2 7 2 6 9 2 2 3 8 3 4 5 8 1 7 1 5 1 4 9 5 2 1 2 5 4 4 3 5 2 6 1 7 6 5 4 1 7 3 1 4 5 6 7 3 8 3 3 1 8 4 p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p p 49 50 51 49 100 49 2 49 49 2 51 100 2 1 51 51 51 1 50 100 1 50 50 1 50 1 49 49 1 49 49 1 2 49 50 49 1 49 1 51 51 100 1 100 51 1 1 51 51 100 51 2 100 100 50 51 2 1 1 1 2 1 1 1 2 1 49 373 paraphrases 145 correct paraphrases 0.32 492 paraphrases 420 correct paraphrases 975 paraphrases 46

  47. Timings 47

  48. Motivation Background Hitting Time Paraphraser Experiments Future Work Overview 48

  49. Apply HTP to languages other than English Evaluate HTP impact on applications e.g., improve performance of resource-sparse machine translation systems Add more features etc. Future Work 49

  50. HTP:a paraphrase system based on random walks Good paraphrases have smaller hitting times General graph Path length > 2 Incorporate domain knowledge HTP outperforms state-of-the-art Conclusion 50

More Related