Introduction to Statistical Machine Translation

Introduction to Statistical Machine Translation David Kauchak CS457 – Fall 2011 Philipp Koehn Kevin Knight Some slides adapted from CSAIL Massachusetts Institute of Technology USC/Information Sciences Institute USC/Computer Science Department

Admin • How are the projects going? • Remaining classes applications • 3 MT • General overview today • Dive into one specific implementation next time • MT lab • Other applications • Information extraction • Information retrieval • Question answering/summarization

Language translation Yo quiero Taco Bell

MT Systems Where have you seen machine translation systems?

Machine Translation The U.S. island of Guam is maintaining a high state of alert after the Guamairport and its offices both received an e-mail from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport . 美国关岛国际机场及其办公室均接获一名自称沙地阿拉伯富商拉登等发出的电子邮件，威胁将会向机场等公众地方发动生化袭击後，关岛经保持高度戒备。 The classic acid test for natural language processing. Requires capabilities in both interpretation and generation. People around the world stubbornly refuse to write everything in English.

Machine Translation 美国关岛国际机场及其办公室均接获一名自称沙地阿拉伯富商拉登等发出的电子邮件，威胁将会向机场等公众地方发动生化袭击後，关岛经保持高度戒备。 Machine translation is becoming very prevalent Even PowerPoint has translation built into it! The U.S. island of Guam is maintaining a high state of alert after the Guamairport and its offices both received an e-mail from someone calling himself the Saudi Arabian Osama bin Laden and threatening a biological/chemical attack against public places such as the airport . United States Guam International Airport and the Office received one claiming to be a wealthy Saudi Arabia an email such as Osama bin Laden, threats to the airport after biochemical attacks in public places such as Guam remain on high alert.

Which is the human? Beijing Youth Daily said that under the Ministry of Agriculture, the beef will be destroyed after tests. The Beijing Youth Daily pointed out that the seized beef would be disposed of after being examined according to advice from the Ministry of Agriculture. ?

Which is the human? Pakistan President Pervez Musharraf Wins Senate Confidence Vote Pakistani President Musharraf Won the Trust Vote in Senate and Lower House ?

Which is the human? There was not a single vote against him." No members vote against him. " ?

Warren Weaver (1947) ingcmpnqsnwfcvfpnowoktvcv huihgzsnwfvrqcffnwcwowgcnwf kowazoanv ...

Warren Weaver (1947) e e e e ingcmpnqsnwf cv fpn owoktvcv e e e hu ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv ...

Warren Weaver (1947) e e e the ingcmpnqsnwf cv fpn owoktvcv e e e hu ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv ...

Warren Weaver (1947) e he e the ingcmpnqsnwf cv fpn owoktvcv e e e t hu ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv ...

Warren Weaver (1947) e he e of the ingcmpnqsnwf cv fpn owoktvcv e e e t hu ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv ...

Warren Weaver (1947) e he e of the fof ingcmpnqsnwf cv fpn owoktvcv e f o e o oe t hu ihgzsnwfv rqcffnw cw owgcnwf ef kowazoanv ...

Warren Weaver (1947) e he e of the ingcmpnqsnwf cv fpn owoktvcv e e e t hu ihgzsnwfv rqcffnw cw owgcnwf e kowazoanv ...

Warren Weaver (1947) e he e is the sis ingcmpnqsnwf cv fpn owoktvcv e s i e i ie t hu ihgzsnwfv rqcffnw cw owgcnwf es kowazoanv ...

Warren Weaver (1947) decipherment is the analysis ingcmpnqsnwf cv fpn owoktvcv of documents written in ancient hu ihgzsnwfv rqcffnw cw owgcnwf languages ... kowazoanv ...

Warren Weaver (1947) The non-Turkish guy next to me is even deciphering Turkish! All he needs is a statistical table of letter-pair frequencies in Turkish … Can this be computerized? Collected mechanically from a Turkish body of text, or corpus

“When I look at an article in Russian, I say: this is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.” - Warren Weaver, March 1947

“When I look at an article in Russian, I say: this is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.” - Warren Weaver, March 1947 “... as to the problem of mechanical translation, I frankly am afraid that the [semantic] boundaries of words in different languages are too vague ... to make any quasi-mechanical translation scheme very hopeful.” - Norbert Wiener, April 1947

Noisy channel along the way things get messed up Some message is sent What was originally sent? We have the mutated message, but would like to recover the original

Noisy channel known system If we know something about what goes on inside here, we might be to decode/recover the message.

Noisy channel “Hi bob” I baab

Noisy channel • ‘H’s often get dropped • ‘o’s go to ‘aa’ sometimes • … “Hi bob” I baab

Noisy channel • ‘H’s often get dropped • ‘o’s go to a sometimes • … “Hi bob” I baab Hi bob

Noisy channel ? Sometime, we don’t know what goes on or we only have a rough guess… What then?

Noisy channel “Hi bob” “hello” “banana” … “I baab” “ello” “banana” … ? A data driven approach Send a bunch of data through where we know what is being sent

Noisy channel “Hi bob” “hello” “banana” … “I baab” “ello” “banana” … Use this data to train a model A data driven approach: learn a model of the types of transformations that occur

Noisy channel • source (s): what was originally sent • target (t): what did we get through the noisy channel We want a probabilistic model of the process: What is the probability of a given source sentence, given that we’ve seen target

How does a model p(s | t) help us? “….” (s) “….” (t) Decode:

Noisy channel model Bayes’ rule how likely is it to receive the target message What types of messages are likely to be sent? What do sent messages look like? What types of transformation happen?How likely are we to generate t from s?

Noisy channel model model channel modeltranslation model source model language model how does the target get “messed up” from the source? what types of things should I expect?

Applications: Speech recognition source words noise given the target audio, what were the original words? target

Applications: Speech recognition model channel modeltranslation model language model acoustic model, how words turn in to sounds what are likely words/word sequences

Applications: Spelling correction source words noise given the target text with misspellings, what were the original, correctly spelled words? baskteballcomitee target

Applications: Spelling correction model channel modeltranslation model language model mistake model, keyboard map, phonetics what are likely words/word sequences

Applications: Sentence compression source The dog runs. noise given the target uncompressed text, what was the original compressed text? The dog runs slowly through the large park. target

Applications: Sentence compression model channel modeltranslation model language model compression model, what words/constituents get deleted, rephrasing what are likely words/word sequences?

Applications: Machine translation source English text noise given the target Chinese text, what was the original English text? 美国关岛国际机场及其办公室均接获一名自称沙地阿拉伯富商拉登等发出的电子邮件，威胁将会向机场等公众地方发动生化袭击後，关岛经保持高度戒备。 target

Applications: Machine translation model channel modeltranslation model language model how do English words/phrases translate to Chinese? what are likely English words/word sequences?

Data-Driven Machine Translation Hmm, every time he sees “banco”, he either types “bank” or “bench” … but if he sees “banco de…”, he always types “bank”, never “bench”… Man, this is so boring. Translated documents

Welcome to the Chinese Room Chinese texts with English translations New Chinese Document English Translation You can teach yourself to translate Chinese using only bilingual data (without grammar books, dictionaries, any people to answer your questions…)

1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp ???

1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihok yorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

1a. ok-voon ororok sprok . 1b. at-voon bichat dat . 7a. lalok farok ororok lalok sprok izok enemok . 7b. wat jjat bichat wat dat vat eneat . 2a. ok-drubel ok-voon anok plok sprok . 2b. at-drubel at-voon pippat rrat dat . 8a. lalok brok anok plok nok . 8b. iat lat pippat rrat nnat . 3a. erok sprok izok hihok ghirok . 3b. totat dat arrat vat hilat . 9a. wiwok nok izok kantok ok-yurp . 9b. totat nnat quat oloat at-yurp . 4a. ok-voon anok drok brok jok . 4b. at-voon krat pippat sat lat . 10a. lalok mok nok yorok ghirok clok . 10b. wat nnat gat mat bat hilat . 5a. wiwok farok izok stok . 5b. totat jjat quat cat . 11a. lalok nok crrrok hihokyorok zanzanok . 11b. wat nnat arrat mat zanzanat . 6a. lalok sprok izok jok stok . 6b. wat dat krat quat cat . 12a. lalok rarok nok izok hihok mok . 12b. wat nnat forat arrat vat gat . Centauri/Arcturan [Knight, 1997] Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

Introduction to Statistical Machine Translation