LING 180 SYMBSYS 138 Intro to Computer Speech and Language Processing

1. LING 180 SYMBSYS 138Intro to Computer Speech and Language Processing Lecture 10: Machine Translation (II) October 26, 2005 Dan Jurafsky Thanks to Kevin Knight for much of this material, and many slides also came from Bonnie Dorr and Christof Monz!

2. Outline for MT Week • Intro and a little history • Language Similarities and Divergences • Four main MT Approaches • Transfer • Interlingua • Direct • Statistical • Evaluation

3. How do we evaluate MT? Human • Fluency • Overall fluency • Human rating of sentences read out loud • Cohesion (Lexical chains, anaphora, ellipsis) • Hand-checking for cohesion. • Well-formedness • 5-point scale of syntactic correctness • Fidelity (same information as source?) • Hand rating of target text on 100pt scale • Clarity • Comprehensibility • Noise test • Multiple choice questionnaire • Readability • cloze

4. Evaluating MT: Problems • Asking humans to judge sentences on a 5-point scale for 10 factors takes time and $$$ (weeks or months!) • We can’t build language engineering systems if we can only evaluate them once every quarter!!!! • We need a metric that we can run every time we change our algorithm. • It would be OK if it wasn’t perfect, but just tended to correlate with the expensive human metrics, which we could still run in quarterly.

5. BiLingual Evaluation Understudy (BLEU —Papineni, 2001) http://www.research.ibm.com/people/k/kishore/RC22176.pdf • Automatic Technique, but …. • Requires the pre-existence of Human (Reference) Translations • Approach: • Produce corpus of high-quality human translations • Judge “closeness” numerically (word-error rate) • Compare n-gram matches between candidate translation and 1 or more reference translations

6. BLEU Evaluation Metric (Papineni et al, ACL-2002) Reference (human) translation:The U.S. island of Guam is maintaining a high state of alertafter theGuamairport and itsoffices both received an e-mail from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such asthe airport . • N-gram precision (score is between 0 & 1) • What percentage of machine n-grams can be found in the reference translation? • An n-gram is an sequence of n words • Not allowed to use same portion of reference translation twice (can’t cheat by typing out “the the the the the”) • Brevity penalty • Can’t just type out single word “the” (precision 1.0!) • *** Amazingly hard to “game” the system (i.e., find a way to change machine output so that BLEU goes up, but quality doesn’t) Machine translation:The American [?] international airport and its the office all receives one calls self the sand Arab rich business [?] and so on electronic mail , which sends out ; The threat will be able after public place and so on the airport to start the biochemistry attack , [?] highly alerts after the maintenance.

7. BLEU Evaluation Metric (Papineni et al, ACL-2002) Reference (human) translation:The U.S. island of Guam is maintaining a high state of alertafter theGuamairport and itsoffices both received an e-mail from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such asthe airport . • BLEU4 formula • (counts n-grams up to length 4) • exp (1.0 * log p1 + • 0.5 * log p2 + • 0.25 * log p3 + • 0.125 * log p4 – • max(words-in-reference / words-in-machine – 1, • 0) • p1 = 1-gram precision • P2 = 2-gram precision • P3 = 3-gram precision • P4 = 4-gram precision Machine translation:The American [?] international airport and its the office all receives one calls self the sand Arab rich business [?] and so on electronic mail , which sends out ; The threat will be able after public place and so on the airport to start the biochemistry attack , [?] highly alerts after the maintenance.

8. Multiple Reference Translations Reference translation 1:The U.S. island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an e-mail from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport . Reference translation 1:The U.S. island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an e-mail from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport . Reference translation 2:Guam International Airport and its offices are maintaining a high state of alert after receiving an e-mail that was from a person claiming to be the wealthy Saudi Arabian businessman Bin Laden and that threatened to launch a biological and chemical attack on the airport and other public places . Reference translation 2:Guam International Airport and its offices are maintaining a high state of alert after receiving an e-mail that was from a person claiming to be the wealthy Saudi Arabian businessman Bin Laden and that threatened to launch a biological and chemical attack on the airport and other public places . Machine translation:The American [?] international airport and its the office all receives one calls self the sand Arab rich business [?] and so on electronic mail , which sends out ; The threat will be able after public place and so on the airport to start the biochemistry attack , [?] highly alerts after the maintenance. Machine translation:The American [?] international airport and itsthe office all receives one calls self the sand Arab rich business [?] and so on electronic mail ,which sends out ; The threat will be able afterpublic place and so on theairportto start the biochemistryattack , [?] highly alerts after the maintenance. Reference translation 3:The US International Airport of Guam and its office has received an email from a self-claimed Arabian millionaire named Laden , which threatens to launch a biochemical attack on such public places as airport . Guam authority has been on alert . Reference translation 3:The US International Airport of Guam and its office has received an email from a self-claimed Arabian millionaire named Laden ,which threatens to launch a biochemical attack on such public places as airport . Guam authority has been on alert . Reference translation 4:US Guam International Airport and its office received an email from Mr. Bin Laden and other rich businessman from Saudi Arabia . They said there would be biochemistry air raid to Guam Airport and other public places . Guam needs to be in high precaution about this matter . Reference translation 4:US Guam International Airport and its office received an email from Mr. Bin Laden and other rich businessman from Saudi Arabia . They said there would be biochemistry air raid to Guam Airport and other public places . Guam needs to be in high precaution about this matter .

9. BLEU in Action 枪手被警方击毙。 (Foreign Original) the gunman was shot to death by the police . (Reference Translation) thegunmanwaspolicekill. #1woundedpolicejayaof #2thegunmanwasshotdeadbythepolice. #3thegunmanarrestedbypolicekill. #4thegunmenwerekilled. #5thegunmanwasshottodeathbythepolice. #6 gunmenwerekilledbypolice?SUB>0?SUB>0 #7 albythepolice. #8theringeriskilledbythepolice. #9policekilledthegunman. #10 green= 4-gram match (good!) red= word not matched (bad!)

10. Bleu Comparison Chinese-English Translation Example: Candidate 1: It is a guide to action which ensures that the military always obeys the commands of the party. Candidate 2: It is to insure the troops forever hearing the activity guidebook that party direct. Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party.

11. How Do We Compute Bleu Scores? • Intuition: “What percentage of words in candidate occurred in some human translation?” • Proposal: count up # of candidate translation words (unigrams) # in any reference translation, divide by the total # of words in # candidate translation • But can’t just count total # of overlapping N-grams! • Candidate: the the the the the the • Reference 1: The cat is on the mat • Solution: A reference word should be considered exhausted after a matching candidate word is identified.

12. “Modified n-gram precision” • For each word compute: (1) the maximum number of times it occurs in any single reference translation (2) number of times it occurs in the candidate translation • Instead of using count #2, use the minimum of #1 and #2, I.e. clip the counts at the max for the reference transcription • Now use that modified count. • And divide by number of candidate words.

13. Modified Unigram Precision: Candidate #1 It(1) is(1) a(1) guide(1) to(1) action(1) which(1) ensures(1) that(2) the(4) military(1) always(1) obeys(0) the commands(1) of(1) the party(1) Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. What’s the answer??? 17/18

14. Modified Unigram Precision: Candidate #2 It(1) is(1) to(1) insure(0) the(4) troops(0) forever(1) hearing(0) the activity(0) guidebook(0) that(2) party(1) direct(0) Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. What’s the answer???? 8/14

15. Modified Bigram Precision: Candidate #1 It is(1) is a(1) a guide(1) guide to(1) to action(1) action which(0) which ensures(0) ensures that(1) that the(1) the military(1) military always(0) always obeys(0) obeys the(0) the commands(0) commands of(0) of the(1) the party(1) Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. 10/17 What’s the answer????

16. Modified Bigram Precision: Candidate #2 It is(1) is to(0) to insure(0) insure the(0) the troops(0) troops forever(0) forever hearing(0) hearing the(0) the activity(0) activity guidebook(0) guidebook that(0) that party(0) party direct(0) Reference 1: It is a guide to action that ensures that themilitary will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. What’s the answer???? 1/13

17. Catching Cheaters the(2) the the the(0) the(0) the(0) the(0) Reference 1: The cat is on the mat Reference 2: There is a cat on the mat What’s the unigram answer? 2/7 What’s the bigram answer? 0/7

18. Bleu distinguishes human from machine translations

19. Bleu problems with sentence length • Candidate: of the • Solution: brevity penalty; prefers candidates translations which are same length as one of the references Reference 1: It is a guide to action that ensures that themilitary will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. Problem: modified unigram precision is 2/2, bigram 1/1!

20. Sentence length • N-gram penalizes using extra words. • Fails to enforce the proper translation length. Sentence Brevity • Multiplicative brevity penalty • Done over the entire corpus, rather than sentence-by-sentence: r/c • r = sum of the “best match lengths” • c = total length of the candidate translation

21. Brevity Penalty Ranking

22. BLEU Tends to Predict Human Judgments (variant of BLEU) slide from G. Doddington (NIST)

23. Rule-Based vs. Statistical MT • Rule-based MT: • Hand-written transfer rules • Rules can be based on lexical or structural transfer • Pro: firm grip on complex translation phenomena • Con: Often very labor-intensive -> lack of robustness • Statistical MT • Mainly word or phrase-based translations • Translation are learned from actual data • Pro: Translations are learned automatically • Con: Difficult to model complex translation phenomena

24. Parallel Corpus • Example from DE-News (8/1/1996)

25. Word-Level Alignments • Given a parallel sentence pair we can link (align) words or phrases that are translations of each other:

26. Parallel Resources • Newswire: DE-News (German-English), Hong-Kong News, Xinhua News (Chinese-English), • Government: Canadian-Hansards (French-English), Europarl (Danish, Dutch, English, Finnish, French, German, Greek, Italian, Portugese, Spanish, Swedish), UN Treaties (Russian, English, Arabic, . . . ) • Manuals: PHP, KDE, OpenOffice (all from OPUS, many languages) • Web pages: STRAND project (Philip Resnik)

27. Sentence Alignment • If document De is translation of document Df how do we find the translation for each sentence? • The n-th sentence in De is not necessarily the translation of the n-th sentence in document Df • In addition to 1:1 alignments, there are also 1:0, 0:1, 1:n, and n:1 alignments • Approximately 90% of the sentence alignments are 1:1

28. Sentence Alignment (c’ntd) • There are several sentence alignment algorithms: • Align (Gale & Church): Aligns sentences based on their character length (shorter sentences tend to have shorter translations then longer sentences). Works astonishingly well • Char-align: (Church): Aligns based on shared character sequences. Works fine for similar languages or technical domains • K-Vec (Fung & Church): Induces a translation lexicon from the parallel texts based on the distribution of foreign-English word pairs.

29. Computing Translation Probabilities • Given a parallel corpus we can estimate P(e | f) The maximum likelihood estimation of P(e | f) is: freq(e,f)/freq(f) • Way too specific to get any reasonable frequencies! Vast majority of unseen data will have zero counts! • P(e | f ) could be re-defined as: • Problem: The English words maximizing P(e | f ) might not result in a readable sentence

30. Computing Translation Probabilities (c’tnd) • We can account for adequacy: each foreign word translates into its most likely English word • We cannot guarantee that this will result in a fluent English sentence • Solution: transform P(e | f) with Bayes’ rule: P(e | f) = P(e) P(f | e) / P(f) • P(f | e) accounts for adequacy • P(e) accounts for fluency

31. Decoding • The decoder combines the evidence from P(e) and P(f | e) to find the sequence e that is the best translation: • The choice of word e’ as translation of f’ depends on the translation probability P(f’ | e’) and on the context, i.e. other English words preceding e’

32. What makes a good translation • Translators often talk about two factors we want to maximize: • Faithfulness or fidelity • How close is the meaning of the translation to the meaning of the original • (Even better: does the translation cause the reader to draw the same inferences as the original would have) • Fluency or naturalness • How natural the translation is, just considering its fluency in the target language

33. Statistical MT: Faithfulness and Fluency formalized! • Best-translation of a source sentence S: • Developed by researchers who were originally in speech recognition at IBM • Called the IBM model

34. The IBM model • Hmm, those two factors might look familiar… • Yup, it’s Bayes rule:

35. It’s also the Noisy Channel Model Slide from Christof Monz

36. Fluency: P(T) • How to measure that this sentence • That car was almost crash onto me • is less fluent than this one: • That car almost hit me. • Answer: language models (N-grams!) • For example P(hit|almost) > P(was|almost) • But can use any other more sophisticated model of grammar • Advantage: this is monolingual knowledge!

37. Faithfulness: P(S|T) • French: ça me plait [that me pleases] • English: • that pleases me - most fluent • I like it • I’ll take that one • How to quantify this? • Intuition: degree to which words in one sentence are plausible translations of words in other sentence • Product of probabilities that each word in target sentence would generate each word in source sentence.

38. Faithfulness P(S|T) • Need to know, for every target language word, probability of it mapping to every source language word. • How do we learn these probabilities? • Parallel texts! • Lots of times we have two texts that are translations of each other • If we knew which word in Source Text mapped to each word in Target Text, we could just count!

39. Faithfulness P(S|T) • Sentence alignment: • Figuring out which source language sentence maps to which target language sentence • Word alignment • Figuring out which source language word maps to which target language word

40. Big Point about Faithfulness and Fluency • Job of the faithfulness model P(S|T) is just to model “bag of words”; which words come from say English to Spanish. • P(S|T) doesn’t have to worry about internal facts about Spanish word order: that’s the job of P(T) • P(T) can do Bag generation: put the following words in order • have programming a seen never I language better -actual the hashing is since not collision-free usually the is less perfectly the of somewhat capacity table

41. P(T) and bag generation: the answer • “Usually the actual capacity of the table is somewhat less, since the hashing is not collision-free” • How about: • loves Mary John

42. A motivating example • Japanese phrase 2000nen taio • 2000nen • 2000 - highest • Y2K • 2000 years • 2000 year • Taio • Correspondence -highest • Corresponding • Equivalent • Tackle • Dealing with • Deal with P(S|T) alone prefers: 2000 Correspondence Adding P(T) might produce correct Dealing with Y2K

43. insistent Wednesday may recurred her trips to Libya tomorrow for flying Cairo 6-4 ( AFP ) - an official announced today in the Egyptian lines company for flying Tuesday is a company " insistent for flying " may resumed a consideration of a day Wednesday tomorrow her trips to Libya of Security Council decision trace international the imposed ban comment . And said the official " the institution sent a speech to Ministry of Foreign Affairs of lifting on Libya air , a situation her receiving replying are so a trip will pull to Libya a morning Wednesday " . Egyptair Has Tomorrow to Resume Its Flights to Libya Cairo 4-6 (AFP) - said an official at the Egyptian Aviation Company today that the company egyptair may resume as of tomorrow, Wednesday its flights to Libya after the International Security Council resolution to the suspension of the embargo imposed on Libya. " The official said that the company had sent a letter to the Ministry of Foreign Affairs, information on the lifting of the air embargo on Libya, where it had received a response, the first take off a trip to Libya on Wednesday morning ". Recent Progress in Statistical MT slide from C. Wayne, DARPA 2002 2003

44. Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

45. 1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

46. 1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

47. 1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

48. 1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp ???

49. 1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

50. 1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihokyorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp