1 / 18

Statistical Transfer and MEMT Activities

Statistical Transfer and MEMT Activities. Chinese-to-English Statistical Transfer MT system (Stat-XFER) Developed over past year Included in Rosetta Phase-II GNG evaluation Team: Erik Peterson and Alon Lavie Multi-Engine Machine Translation MEMT service within the cross-GALE IOD

sela
Download Presentation

Statistical Transfer and MEMT Activities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical Transfer and MEMT Activities • Chinese-to-English Statistical Transfer MT system (Stat-XFER) • Developed over past year • Included in Rosetta Phase-II GNG evaluation • Team: Erik Peterson and Alon Lavie • Multi-Engine Machine Translation • MEMT service within the cross-GALE IOD • MEMT system combination for GNG evaluation within Rosetta consortium • Team: Greg Hanneman, Shilpa Aurora, Dave Svoboda, Alon Lavie and Eric Nyberg GALE Update: Alon Lavie

  2. CMU Statistical-XFER System Truly hybrid “rule-based”/statistical system • Scaled up version of our XFER approach developed for low-resource languages (under NSF funding) • Large-coverage “clean” bilingual lexicon + syntactic transfer rules (human written + extracted from data) • XFER formalism is a Synchronous CFG + feature unification constraints • Supports morphological analysis and generation as “plug in” components • Two-stage translation process: • Build lattice of translation constituents at all grammar levels “bottom-up” • Monotonic decoder selects best combination of lattice edges • Beam-search with multiple features at both stages • Features include: LM, fragmentation, lexical probabilities, length, etc. • Optimized Log-linear combination of feature scores GALE Update: Alon Lavie

  3. Chinese-English S-XFER System • Bilingual lexicon: over 1.1 million entries (multiple resources, incl. ADSO, Wikipedia, extracted base NPs) • Manual syntactic XFER grammar:76 rules! (mostly NPs, a few PPs, and reordering of NPs/PPs within VPs) • Multiple overlapping Chinese word segmentations • English morphology generation • Uses CMU SMT-group’s Suffix-Array LM toolkit for LM • Current Performance (GALE dev-test): • NW XFER: 10.89(B)/0.4509(M) Best (UMD): 15.58(B)/0.4769(M) • NG XFER: 8.92(B)/0.4229(M) Best (UMD): 12.96(B)/0.4455(M) • In Progress: • Automatic extraction of “clean” base NPs from parallel data • Automatic learning and extraction of high-quality transfer-rules from parallel data GALE Update: Alon Lavie

  4. Recent Performance Analysis • What fraction of the time does each MT system produce the best translation (sentence-by-sentence)? • Evaluated on Chinese GALE dev-test (text) data BLEU METEOR CMU-PhraseSyntaxCombination 60 of 284 (21.1%) 41 of 284 (14.4%) IBM-smt 50 of 284 (17.6%) 49 of 284 (17.2%) IBM-ylee 64 of 284 (22.5%) 50 of 284 (17.6%) maryland-jhu-combination 71 of 284 (25.0%) 77 of 284 (27.1%) Stat-XFER 32 of 284 (11.2%) 56 of 284 (19.7%) GALE Update: Alon Lavie

  5. Translation Example • REFERENCE:When responding to whether it is possible to extend Russian fleet's stationing deadline at the Crimean peninsula, Yanukovych replied, "Without a doubt. • Stat-XFER (0.3989): In reply to whether the possibility to extend the Russian fleet stationed in Crimea Pen. left the deadline of the problem , Yanukovich replied : " of course . • IBM-ylee (0.2203): In response to the possibility to extend the deadline for the presence in Crimea peninsula , the Queen Vic said : " of course . • CMU-SMT (0.2067): In response to a possible extension of the fleet in the Crimean Peninsula stay on the issue , Yanukovych vetch replied : " of course . • maryland-hiero (0.1878): In response to the possibility of extending the mandate of the Crimean peninsula in , replied: "of course. • IBM-smt (0.1862): The answer is likely to be extended the Crimean peninsula of the presence of the problem, Yanukovych said: " Of course. • CMU-syntax (0.1639): In response to the possibility of extension of the presence in the Crimean Peninsula , replied : " of course . GALE Update: Alon Lavie

  6. Synthetic Combination MEMT Two Stage Approach: • Identify common words and phrases across the translations provided by the engines • Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation Example: • announced afghan authorities on saturday reconstituted four intergovernmental committees • The Afghan authorities on Saturday the formation of the four committees of government GALE Update: Alon Lavie

  7. Synthetic Combination MEMT Two Stage Approach: • Identify common words and phrases across the translations provided by the engines • Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation Example: • announced afghan authoritieson saturday reconstituted four intergovernmental committees • The Afghan authoritieson Saturday the formation of the four committees of government MEMT: the afghan authorities announced on Saturday the formation of four intergovernmental committees GALE Update: Alon Lavie

  8. Scoring MEMT Hypotheses • Scoring Features: • Word confidence score [0,1] based on engine confidence and reinforcement from alignments • LM score based on suffix-array 5-gram LM • Exponential “occurrence-based” long n-gram model • Select best scoring hypothesis based on optimized log-linear combination of feature scores GALE Update: Alon Lavie

  9. GALE Update: Alon Lavie

  10. MEMT – Main Activities • Preserving Source Alignments: target phrases that originate from same source word can be marked as unbreakable units (performance effects are currently under testing…) • LM experiments using CMU’s Suffix-Array LM toolkit and new LM features (work in progress) • Case Restoration: scheme for selecting the case of words in final MEMT output • Handling of varying number of MEMT input engines • Parameter Optimization for MEMT decoder • Upgrades to MEMT software infrastructure to support IOD-3 requirements: UIMA v2.0 and ActiveMQ • MEMT server is up 24/7 for ongoing IOD runs • Active participation in phase-II GNG evaluation GALE Update: Alon Lavie

  11. Recent Evaluation Results GALE Update: Alon Lavie

  12. MEMT Fresh Example • Extracted from IOD overnight run of Sunday, July 22 • IBM_MT: “defence minister announced that the Afghan security forces and Afghan joint NATO began this morning the process of wide security” • RWTH_MT:"afghan defence minister that the security forces , a joint us-afghan started today in the morning , the implementation of a wide security operation .” • CMU_MEMT:"afghan defence minister announced that the security forces and Afghan joint NATO began this morning in the process of the a wide security.” GALE Update: Alon Lavie

  13. Future Plans and New Directions • Classifiers for Hypothesis Selection • Simpler than MEMT, but perhaps more effective (given recent analysis results) • Constrained Search-spaces for MEMT • Other groups use more constrained combination spaces with good results – can we also do better? • Discriminative feature-rich LMs for MT (and MEMT) • Standard statistical LMs are not sufficiently discriminative for MT • New NSF grant (with Rebecca Hwa) to explore novel feature-rich “occurrence-based” models • M-TER • Create a fully automatic metric that approximates H-TER using the stemming and synonymy capabilities of METEOR to create “targeted” references GALE Update: Alon Lavie

  14. Translation Example • REFERENCE: I believe there is a detail that worth mentioning. • Stat-XFER (0.5799): There is a detail , I believe worth together . • maryland-hiero (0.3221): There is a one details, I think they deserve. • IBM-ylee (0.2525): There is one of the details , I think it is worth . • IBM-smt (0.2062): There is a details, I think is a way. • CMU-SMT (0.2000): One of the details , I think it is worth to join in . • CMU-syntax (0.1546): a detail , I think it is a cold front . • maryland-jhu (0.1031): One to details, I think they deserve. GALE Update: Alon Lavie

  15. Translation Example • REFERENCE: Xinhua News Agency, Zhuhai, Nov. 2 (reporters Wang Hongshan, Li Xuanliang) - Yang Liwei, Deputy Director of China Astronaut Research and Training Center and 'Space Hero', said in Zhuhai on the 1st that China has no plan to select female astronaut yet. • Stat-XFER (0.6349): The Xinhua News Agency and the Zhuhai November seconds ( reporters Wang Hongshan , li announced good ) Chinese astronaut research and training center deputy director , " aerospace hero " Yang Liwei and the first said in zhuhai , china now not yet to select woman astronaut plan . • IBM-ylee (0.5325): Zhuhai November 2 ( Xinhua News Agency , Chinese astronauts scientific training center , deputy director of the " space hero Yang Liwei said on July 1 in Zhuhai , China is still no plan and the flight , astronauts . • IBM-smt (0.5194): Xinhua Zhuhai, November 2nd (Xinhua) (Reporter,) Chinese astronauts scientific training, deputy director of the Center, " Space hero ' yang Liwei, 1st in Zhuhai, said that China has no plan of selecting astronaut. • maryland-hiero (0.5180): ZHUHAI, November 2 (Xinhua , ) deputy director of the Chinese astronauts scientific research and training center, "1" space hero Yang Liwei said in Zhuhai, China does not plan to make astronaut selection. • CMU-combination (0.4701): zhuhai , november 2nd ( xinhua ) deputy director of china's astronaut training centre , " space hero yang liwei , 1st in zhuhai , china currently has no plans of the selected astronaut . GALE Update: Alon Lavie

  16. Chinese-English Example - Before THE SCIENTISTS IN ORDER TO Øü TO CLOSE IN THE EARLY PERIOD TO GO THE THE KNOWLEDGE THE THE DISEASE IN THE CHROMOSOME HAS BEEN COMPLETED IS SCHEDULED TO ORDER Overall: -6.68742, Prob: -188.623, Rules: 1.46679, Frag: 0.4, Length: 0.729557, Words: 13,30 1 < 0 1 -13.4185: ¿Æѧ¼Ò (LEX,49119 'THE SCIENTISTS')> 43 < 1 2 -12.2635: Ϊ (LEX,12060 'IN ORDER TO')> 178 < 2 3 -22.1807: Øü (UNK,0 'Øü')> 285 < 3 4 -20: ¹Ø (VP,3 (V,474 'TO CLOSE'))> 1291 < 4 5 -19.8476: ³õÆÚ (LEX,19200 'IN THE EARLY PERIOD')> 69 < 5 6 -13.3294: ʧ (V,125852 'TO GO')> 395 < 6 7 -30: ÖÇ (NP,1 (LITERAL 'THE') (NB,1 (N,1133856 'THE KNOWLEDGE')))> 751 < 7 8 -30: Ö¢ (NP,1 (LITERAL 'THE') (NB,1 (N,1130647 'THE DISEASE')))> 128 < 8 9 -5.20685: µÄ (LEX,47352 'IN')> 864 < 9 10 -20: ȾɫÌå (NP,1 (LITERAL 'THE') (NB,1 (N,10988 'CHROMOSOME')))> 206 < 10 11 -17.5087: Íê³É (LEX,28856 'HAS BEEN COMPLETED')> 214 < 11 12 -13.3074: ¶¨ (LEX,28935 'IS SCHEDULED')> 230 < 12 13 -15.5876: Ðò (N,1102943 'TO ORDER')> GALE Update: Alon Lavie

  17. Chinese-English Example - After THE SCIENTISTS COMPLETED SEQUENCING FOR THE CHROMOSOMES WHICH RELATED TO THE INITIAL STAGE DEMENTIA Overall: -7.09009, Prob: -99.2612, Rules: 3.75244, Frag: 0, Length: 0.493383, Words: 8,14 2902 < 0 14 -99.2612: ¿Æѧ¼Ò Ϊ Øü¹Ø ³õÆÚ Ê§ÖÇÖ¢ µÄ ȾɫÌå Íê³É ¶¨Ðò (S,1 (NP,1 (LITERAL 'THE') (NB,1 (N,21601 'SCIENTISTS'))) (VP,4 (VP,1 (V,7513 'COMPLETED')(NP,2 (NB,1 (N,940881 'SEQUENCING')))) (PP,1 (PREP,5 'FOR')(NPRC,1 (NP,1 (LITERAL 'THE') (NB,1 (N,1039285 'CHROMOSOMES'))) (LITERAL 'WHICH') (VP,1 (V,18 'RELATED TO') (NPASSOC,5 (NP,1 (LITERAL 'THE') (NB,1 (N,7637 'INITIAL STAGE'))) (NP,2 (NB,1 (N,445 'DEMENTIA')))))))))> GALE Update: Alon Lavie

  18. C/E Stat-XFER Lexicon • Lexical sources and their sizes: • Named Entities from LDC 820080 lexical entries • Filtered base NPs from Parallel Corpus 148631 lexical entries • ADSO bilingual lexicon 104513 lexical entries • LDC word bilingual glossary 78173 lexical entries • Wikipedia extracted bilingual lexicon 51009 lexical entries • Phrases from Parallel Corpus 44968 lexical entries • Manual bilingual lexicon (high freq) 1149 lexical entries GALE Update: Alon Lavie

More Related