1 / 43

Session #2

Session #2. Introductions Syllabus, requirements, etc. Language in 10 minutes Lecture MT Approaches MT Evaluation MT Lab. Road Map. Multilingual Challenges for MT MT Approaches MT Evaluation. Gisting. MT Approaches MT Pyramid. Source meaning. Target meaning. Source syntax.

ilyssa
Download Presentation

Session #2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Session #2 • Introductions • Syllabus, requirements, etc. • Language in 10 minutes • Lecture • MT Approaches • MT Evaluation • MT Lab

  2. Road Map • Multilingual Challenges for MT • MT Approaches • MT Evaluation

  3. Gisting MT ApproachesMT Pyramid Source meaning Target meaning Source syntax Target syntax Source word Target word Analysis Generation

  4. MT ApproachesGisting Example Sobre la base de dichas experiencias se estableció en 1988 una metodología. Envelope her basis out speak experiences them settle at 1988 one methodology. On the basis of these experiences, a methodology was arrived at in 1988.

  5. Gisting Transfer MT ApproachesMT Pyramid Source meaning Target meaning Source syntax Target syntax Source word Target word Analysis Generation

  6. butter Y X MT ApproachesTransfer Example • Transfer Lexicon • Map SL structure to TL structure poner :subj :mod :obj  :subj :obj mantequilla en X :obj Y X puso mantequilla en Y X buttered Y

  7. Gisting Transfer Interlingua MT ApproachesMT Pyramid Source meaning Target meaning Source syntax Target syntax Source word Target word Analysis Generation

  8. MT ApproachesInterlingua Example: Lexical Conceptual Structure (Dorr, 1993)

  9. Gisting Transfer Interlingua MT ApproachesMT Pyramid Source meaning Target meaning Source syntax Target syntax Source word Target word Analysis Generation

  10. Dictionaries/Parallel Corpora Transfer Lexicons Interlingual Lexicons MT ApproachesMT Pyramid Source meaning Target meaning Source syntax Target syntax Source word Target word Analysis Generation

  11. MT ApproachesStatistical vs. Rule-based Source meaning Target meaning Source syntax Target syntax Source word Target word Analysis Generation

  12. Statistical MT Noisy Channel Model Portions from http://www.clsp.jhu.edu/ws03/preworkshop/lecture_yamada.pdf

  13. Slide based on Kevin Knight’s http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Statistical MT Automatic Word Alignment • GIZA++ • A statistical machine translation toolkit used to train word alignments. • Uses Expectation-Maximization with various constraints to bootstrap alignments Maria no dio una bofetada a la bruja verde Mary did not slap the green witch

  14. Giza Plateau and MT Pyramid • EGYPT toolkit • Giza, Giza++ • Cairo visualizer • Pharaoh • Hiero • Moses Words Words Words

  15. Statistical MT IBM Model (Word-based Model) http://www.clsp.jhu.edu/ws03/preworkshop/lecture_yamada.pdf

  16. Slide courtesy of Kevin Knight http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Phrase-Based Statistical MT Morgen fliege ich nach Kanada zur Konferenz Tomorrow I will fly to the conference In Canada • Foreign input segmented in to phrases • “phrase” is any sequence of words • Each phrase is probabilistically translated into English • P(to the conference | zur Konferenz) • P(into the meeting | zur Konferenz) • Phrase distortion • This is state-of-the-art!

  17. Slide courtesy of Kevin Knight http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Word Alignment Induced Phrases Maria no dió una bofetada a la bruja verde Mary did not slap the green witch (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green)

  18. Slide courtesy of Kevin Knight http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Word Alignment Induced Phrases Maria no dió una bofetada a la bruja verde Mary did not slap the green witch (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the)

  19. Slide courtesy of Kevin Knight http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Word Alignment Induced Phrases Maria no dió una bofetada a la bruja verde Mary did not slap the green witch (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the) (Maria no, Mary did not) (no dió una bofetada, did not slap), (dió una bofetada a la, slap the) (bruja verde, green witch)

  20. Word Alignment Induced Phrases Slide courtesy of Kevin Knight http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Maria no dió una bofetada a la bruja verde Mary did not slap the green witch (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the) (Maria no, Mary did not) (no dió una bofetada, did not slap), (dió una bofetada a la, slap the) (bruja verde, green witch)(Maria no dió una bofetada, Mary did not slap) (a la bruja verde, the green witch) …

  21. Slide courtesy of Kevin Knight http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Word Alignment Induced Phrases Maria no dió una bofetada a la bruja verde Mary did not slap the green witch (Maria, Mary) (no, did not) (slap, dió una bofetada) (la, the) (bruja, witch) (verde, green) (a la, the) (dió una bofetada a, slap the) (Maria no, Mary did not) (no dió una bofetada, did not slap), (dió una bofetada a la, slap the) (bruja verde, green witch)(Maria no dió una bofetada, Mary did not slap) (a la bruja verde, the green witch) … (Maria no dió una bofetada a la bruja verde, Mary did not slap the green witch)

  22. Phrase-Based Models • Sentence f is decomposed into J phrases f1J = f1,...,fj,...,fJ • Sentence e is decomposed into l phrases e = eI1 = e1,...,ei,...,eI. • We choose the sentence with the highest probability:

  23. Phrase-Based Models • Model the posterior probability using a log-linear combination of feature functions. • We have a set of M feature functions hm(eI1,f1J),m = 1,...,M. For each feature function, there exists a model parameter λm ,m = 1,...,M • The decision Rule is • Features cover the main components • Phrase-Translation Model • Reordering Model • Language Model

  24. Slide courtesy of Kevin Knight http://www.sims.berkeley.edu/courses/is290-2/f04/lectures/mt-lecture.ppt Advantages of Phrase-Based SMT • Many-to-many mappings can handle non-compositional phrases • Local context is very useful for disambiguating • “Interest rate” … • “Interest in”  … • The more data, the longer the learned phrases • Sometimes whole sentences

  25. MT ApproachesStatistical vs. Rule-based vs. Hybrid Source meaning Target meaning Source syntax Target syntax Source word Target word Analysis Generation

  26. Original statistical MT Word-based only Electronic dictionaries Example-based MT Phrase tables Hand-built by experts Hand-built by non-experts Learn from annotated data Learn from un-annotated data Original direct approach Syntactic Constituent Structure Typical transfer system Semantic analysis New Research Goes Here! Classic interlingual system Interlingua Shallow/ Simple Knowledge Acquisition Strategy All manual Fully automated Knowledge Representation Strategy Deep/ Complex Slide courtesy of Laurie Gerber

  27. MT ApproachesPractical Considerations • Resources Availability • Parsers and Generators • Input/Output compatability • Translation Lexicons • Word-based vs. Transfer/Interlingua • Parallel Corpora • Domain of interest • Bigger is better • Time Availability • Statistical training, resource building

  28. Road Map • Multilingual Challenges for MT • MT Approaches • MT Evaluation

  29. MT Evaluation • More art than science • Wide range of Metrics/Techniques • interface, …, scalability, …, faithfulness, ... space/time complexity, … etc. • Automatic vs. Human-based • Dumb Machines vs. Slow Humans

  30. Human-based Evaluation ExampleAccuracy Criteria

  31. Human-based Evaluation ExampleFluency Criteria

  32. Fluency vs. Accuracy FAHQ MT conMT Prof. MT Fluency Info. MT Accuracy

  33. Automatic Evaluation ExampleBleu Metric(Papineni et al 2001) • Bleu • BiLingual Evaluation Understudy • Modified n-gram precision with length penalty • Quick, inexpensive and language independent • Correlates highly with human evaluation • Bias against synonyms and inflectional variations

  34. Automatic Evaluation ExampleBleu Metric Test Sentence colorless green ideas sleep furiously Gold Standard References all dull jade ideas sleep irately drab emerald concepts sleep furiously colorless immature thoughts nap angrily

  35. Automatic Evaluation ExampleBleu Metric Test Sentence colorless green ideassleepfuriously Gold Standard References all dull jade ideassleep irately drab emerald concepts sleepfuriously colorless immature thoughts nap angrily Unigram precision = 4/5

  36. Automatic Evaluation ExampleBleu Metric Test Sentence colorless green ideas sleep furiously colorless green ideas sleep furiously colorless greenideas sleepfuriously colorless green ideassleep furiously Gold Standard References all dull jade ideassleep irately drab emerald concepts sleepfuriously colorless immature thoughts nap angrily Unigram precision = 4 / 5 = 0.8 Bigram precision = 2 / 4 = 0.5 Bleu Score = (a1 a2 …an)1/n = (0.8╳ 0.5)½ = 0.6325  63.25

  37. Automatic Evaluation ExampleMETEOR (Lavie and Agrawal 2007) • Metric for Evaluation of Translation with Explicit word Ordering • Extended Matching between translation and reference • Porter stems, wordNet synsets • Unigram Precision, Recall, parameterized F-measure • Reordering Penalty • Parameters can be tuned to optimize correlation with human judgments • Not biased against “non-statistical” MT systems

  38. Automatic Evaluation ExampleSEPIA(Habash and ElKholy 2008) • A syntactically-aware evaluation metric • (Liu and Gildea, 2005; Owczarzak et al., 2007; Giménez and Màrquez, 2007) • Uses dependency representation • MICA parser (Nasr & Rambow 2006) • 77% of all structural bigrams are surface n-grams of size 2,3,4 • Includes dependency surface span as a factor in score • long-distance dependencies should receive a greater weight than short distance dependencies • Higher degree of grammaticality?

  39. Metrics MATR Workshop • Workshop in AMTA conference 2008 • Association for Machine Translation in the Americas • Evaluating evaluation metrics • Compared 39 metrics • 7 baselines and 32 new metrics • Various measures of correlation with human judgment • Different conditions: text genre, source language, number of references, etc.

  40. Building an SMT System • Training • Input • Parallel Training Data • Monolingual Data of the Target Language • Output • Phrase Table • Language Model • Initial Translation Model parameters • Tuning • Input • Parallel Tune Data Set • Decoder • Phrase Table + Language Model + Initial Translation Model parameters • Output • Improved Translation Model parameters • Development & Testing • Input • Parallel Test Data Set • Decoder • Phrase Table + Language Model + FinalTranslation Model parameters • Output • BLEU score

  41. Training Source Target Monolingual Preprocess Create LM Alignment Phrase Extraction LM Phrase Table (PT)

  42. Tuning Tune Set Preprocess LM Decoder Weights Adjust Weights PT Output No Optimal? Yes Metric Score Done

  43. Testing Test Set LM Preprocess PT Weights Decoder Output

More Related