Discriminative Reordering with Chinese Grammatical Relations Features

1. Discriminative Reordering with Chinese Grammatical Relations Features Speaker: Pi-Chuan Chang Joint work with Huihsin Tseng, Dan Jurafsky, and Chris Manning

2. Motivation • Chinese and English – both SVO languages, but… • Lots of different word orders

3. VP PP PP VP Lots of different word orders –Ch [PP VP]  En [VP PP] • Ch [PP VP]  En [VP PP]

4. LCP Lots of different word orders –Localizers • Localizers • like a post-phrasal preposition • often used with temporal or locative phrases localizer

5. Lots of different word orders –The notorious “DE” DE: Ch [Complementizer Phrase that pre-modify an NP ] En [ relative clause ] DE: not always reordered Possessive: 他(he) / 的(‘s) / 院子(yard)  his yard Adjective: 红色(red) / 的/ (“DE”) / 苹果 (apple)  red apple CP NP

6. 被 (BEI) construction 把 (BA) constructionExample: Lots of different word orders –special Chinese constructions NP0 BA NP1 VP NP0 VP NP1

7. Motivation Chinese and English – lots of different word orders Previous work Chinese Syntactic Reordering for Statistical Machine Translation,(Wang et al, EMNLP 2007) Reorder Chinese sentences according to the parse structures Address most of the important reordering patterns above Disambiguating "DE" for Chinese-English Machine Translation,(Chang et al, WMT 2009) Build a “DE” classifier to mark different usages of DE

8. This work… • Instead of using pre-defined patterns, use general Chinese structures as features and learn which ones are more important • Instead of preprocessing data, put the reordering model into MERT

9. Outline • Motivation • Discriminative Reordering model • Chinese Grammatical Relations • Experiments • Conclusion

10. Outline • Motivation • Discriminative Reordering model • A model proposed by Zens and Ney (2006)Discriminative reordering models for statistical machine translation • improved by path features with grammatical relations • Chinese Grammatical Relations • Experiments • Conclusion

11. ? ? Discriminative Reordering Model(Zens and Ney 2006) In a phrase-based system: hypothesis c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 ordered reversed e1 e2 e3 e4 e5 e6 e7 For two consecutive English words, decide the order of the source Chinese words: {reversed, ordered}

12. Reordering model in (Zens and Ney 2006) Labeled examples extracted from parallel text and alignment ( i ) ( j )

13. Reordering model in (Zens and Ney 2006) Labeled examples extracted from parallel text and alignment i = 0, i’ = 1 j = 0, j’ = 6 class = ordered ( i ) ( j )

14. Reordering model in (Zens and Ney 2006) Labeled examples extracted from bi-text and alignment i = 1, i’ = 2 j = 6, j’ = 7 class = ordered ( i ) ( j )

15. Reordering model in (Zens and Ney 2006) Labeled examples extracted from bi-text and alignment i = 2, i’ = 3 j = 7, j’ = 7 No examples extracted ( i ) ( j )

16. Reordering model in (Zens and Ney 2006) Labeled examples extracted from bi-text and alignment i = 3, i’ = 4 j = 7, j’ = 5 class = reversed ( i ) ( j )

17. Reordering model in (Zens and Ney 2006) Labeled examples extracted from bi-text and alignment i = 4, i’ = 5 j = 5, j’ = 3 class = reversed ( i ) ( j )

18. Reordering model in (Zens and Ney 2006) Labeled examples extracted from bi-text and alignment i = 5, i’ = 6 j = 3, j’ = 4 class = ordered ( i ) ( j )

19. Reordering model in (Zens and Ney 2006) Labeled examples extracted from bi-text and alignment i = 6, i’ = 7 j = 4, j’ = 8 class = ordered ( i ) ( j )

20. Reordering model in (Zens and Ney 2006) • 6 labeled examples are extracted • 4 ordered • 2 reversed ( i ) ( j )

21. Reordering model in (Zens and Ney 2006) Feature extraction ( i ) ( j )

22. Reordering model in (Zens and Ney 2006) Feature extraction i = 3, i’ = 4 j = 7, j’ = 5 class = reversed ( i ) Baseline feature functions:lexical features around the example After labeled examples and features are extracted: Train a log-linear model to predict orientation classes: {ordered, reversed} ( j )

23. Improving the Orientation Model • Baseline features: lexical features • More source-side syntactic features • motivation: certain Chinese constructions are more likely to cause reordering • We propose to use grammatical relations between words as additional features.

24. Outline • Motivation • Discriminative Reordering model • Chinese Grammatical Relations • Grammatical relations between words • PATH features • Experiments • Conclusion

25. Chinese Grammatical Relations • Binary relations between words • typed dependencies nominal subject prepositional modifier nsubj localizer modifier of a preposition prep localizer object plmod lobj dobj direct object 在 里 吃 西瓜 我 [他 的 院子] I at [he ‘s yard] inside eat watermelon

26. Chinese Grammatical Relations • 45 named grammatical relations • Coverage: 91.3% of all dependencies (tested on CTB6 files 1—325) • The rest 8.7% get a generic dependency name (dep).

27. nsubj prep plmod dobj lobj 在 里 吃 西瓜 我 [他 的 院子] I at [he ‘s yard] inside eat watermelon PATH features in the reordering model

28. nsubj prep plmod dobj lobj 在 里 吃 西瓜 我 [他 的 院子] I at [he ‘s yard] inside eat watermelon PATH features in the reordering model i = 0, i’ = 1 j = 0, j’ = 6 class = ordered PATH feature = nsubj nsubj

29. nsubj prep plmod dobj lobj 在 里 吃 西瓜 我 [他 的 院子] I at [he ‘s yard] inside eat watermelon PATH features in the reordering model i = 3, i’ = 4 j = 7, j’ = 5 class = reversed PATH feature= plmod-prep-dobjR plmod prep * directionality: “R” dobj

30. Outline • Motivation • Discriminative Reordering model • Chinese Grammatical Relations • Experiments • Orientation classifier • MT experiments • Analysis • Conclusion

31. Orientation Classifier • About 9M examples are extracted from the MT training data and alignment • Binary classifier : {ordered, reversed} 45% error reduction +2.8 +7.5 86.3

32. MajorityClass Orientation Classifier • Binary classifier : {ordered, reversed} • Look at the performance on the minor class reversed PATH feature:improve accuracy, especially on “reversed” +11.5 74.8

33. Using the model in MT experiments • Use log probability of the orientation model as an extra feature in Phrase-based systems Current hypothesis c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 e1 e2 e3 e4 e5

34. Using the model in MT experiments • Use log probability of the orientation model as an extra feature in Phrase-based systems New hypothesis c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 reversed e1 e2 e3 e4 e5 e6 e7 add feature value: log P(reversed | , , , ) c4 c5 c6 e5 e6 e7

35. Using the feature in MT experiments • Use log probability of the orientation model as an extra feature in Phrase-based systems Current hypothesis c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 e1 e2 e3 e4 e5

36. Using the feature in MT experiments • Use log probability of the orientation model as an extra feature in Phrase-based systems New hypothesis c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 ordered e1 e2 e3 e4 e5 e6 e7 add feature value: log P(ordered | , , , ) c7 c8 c6 e5 e6 e7

37. MT experiments • Experiments: Exp1. Moses (+ lexicalized reordering) Exp2. (Exp1) + ReorderingLexicalFeatures (no PATH feature) Exp3. (Exp1) + ReorderingLexicalFeatures+PATH features

38. Analysis: features for “reversed” • Features highly weighted for being “reversed”: • Many of them match linguistic intuitions • Chinese [PP VP]  English [VP PP] • prep-dobjR • Chinese [CPNP]  English [NP relative clause] • a DE construction that causes reordering • rcmod (relative clause modifier) CP rcmod NP

39. Analysis: features for “reversed” • Another example for a highly “reversed” path: • det-nn : • A longer nominal modifier (with a DT) in Chinese is more likely to become a prepositional modifier in English det nn 各 阶段 产品 every level product products of all level

40. Conclusion • We presented a set of Chinese grammatical relations that describes binary relations between words • The dependency paths between Chinese words bring useful information for reordering decisions • Significant BLEU gain on three test sets:MT02 (+0.6), MT03 (+1.0) and MT05 (+0.8).

41. Software • Stanford parser • http://www-nlp.stanford.edu/software/lex-parser.shtml • English, Chinese, Arabic, and German • Typed dependencies: English and Chinese • (note: the current released version contains an older set of Chinese dependencies. Will be updated later)

42. Thank you.