1 / 53

CMSC 723 / LING 645: Intro to Computational Linguistics

CMSC 723 / LING 645: Intro to Computational Linguistics. September 8, 2004: Dorr MT (continued), MT Evaluation Prof. Bonnie J. Dorr Dr. Christof Monz TA: Adam Lee. MT Challenges: Ambiguity. Syntactic Ambiguity I saw the man on the hill with the telescope Lexical Ambiguity E: book

ondrea
Download Presentation

CMSC 723 / LING 645: Intro to Computational Linguistics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CMSC 723 / LING 645: Intro to Computational Linguistics September 8, 2004: Dorr MT (continued), MT Evaluation Prof. Bonnie J. DorrDr. Christof MonzTA: Adam Lee

  2. MT Challenges: Ambiguity • Syntactic AmbiguityI saw the man on the hill with the telescope • Lexical Ambiguity E: book S: libro, reservar • Semantic Ambiguity • Homography:ball(E) = pelota, baile(S) • Polysemy:kill(E), matar, acabar (S) • Semantic granularityesperar(S) = wait, expect, hope (E)be(E) = ser, estar(S)fish(E) = pez, pescado(S)

  3. MT Challenges: Divergences • Meaning of two translationally equivalent phrases is distributed differently in the two languages • Example: • English: [RUN INTO ROOM] • Spanish: [ENTER IN ROOM RUNNING]

  4. Divergence Frequency • 32% of sentences in UN Spanish/English Corpus (5K) • 35% of sentences in TREC El Norte Corpus (19K) • Divergence Types • Categorial (X tener hambre  X have hunger) [98%] • Conflational (X dar puñaladas a Z  X stab Z) [83%] • Structural (X entrar en Y  X enter Y) [35%] • Head Swapping (X cruzar Y nadando  X swim across Y) [8%] • Thematic (X gustar a Y  Y like X) [6%]

  5. Spanish/Arabic Divergences Divergence E/E’ (Spanish) E/E’ (Arabic) Categorial be jealous when he returns have jealousy [tener celos] upon his return [ﻋﻧﺩ ﺮﺠﻭﻋﻪ] Conflational float come again go floating [ir flotando] return [ﻋﺎﺪ] Structural enter the house seek enter in the house [entrar en la casa] search for [ﺒﺣﺙ ﻋﻦ] Head Swap run in do something quickly enter running [entrar corriendo] go-quickly in doing something [ﺍﺴﺭﻉ] Thematic I have a headache my-head hurts me [me duele la cabeza] — [Arg1 [V]] Þ [Arg1 [MotionV] Modifier(v)] “The boat floated’’ Þ “The boat went floating’’

  6. Automatic Divergence Detection (using narrowly defined divergence detection rules) LanguageDetectedHumanSampleCorpus ConfirmedSizeSize Spanish – Total 11.1% 10.5% 19K 150K Arabic – Total 31.9 12.5% 1K 28K

  7. Application of Divergence Detection: Bilingual Alignment for MT • Word-level alignments of bilingual texts are an integral part of MT models • Divergences present a great challenge to the alignment task • Common divergence types can be found in multiple language pairs, systematically identified, and resolved

  8. The Problem:Alignment & Projection I began to eat the fish Yo empecé a comer el pescado

  9. Why is this a hard problem? I run into the room Yo entro en el cuarto corriendo

  10. Divergences! English: [RUN INTO ROOM] Spanish: [ENTER IN ROOM RUNNING]

  11. Our Goal: Improved Alignment & Projection • Induce higher interannotator agreement rate • Increase the number of aligned words • Decrease multiple alignments

  12. E: I run into the room E¢: I move-in running the room S: Yo entro en el cuarto corriendo DUSTer Approach: Divergence Unraveling

  13. enter run John room into John running room Ex: John ran into the room → John entered the room running Word-Level Alignment (1): Test Setup • Divergence Detection: Categorize English sentences into one of 5 divergence types • Divergence Correction: Apply appropriate structural transformation [E → E¢]

  14. Word-Level Alignment (2): Testing Impact of Divergence Correction • Human align English and foreign sentence • Human align English¢ and foreign sentence • Compare inter-annotator agreement, unaligned units, multiple alignments

  15. Word-Level Alignment Results • Inter-Annotator Agreement: • English-Spanish: agreement increased from 80.2% to 82.9% • English-Arabic: agreement increased from 69.7% to 75.1% • Number of aligned words: • English-Spanish: aligned words increased from 82.8% to 86% • English-Arabic: aligned words increased from 61.5% to 88.1% • Multiple Alignments: • English-Spanish: number of links went from 1.35 to 1.16 • English-Arabic: number of links increased from 1.48 to 1.72

  16. Divergence Unraveling Conclusions • Divergence handling shows promise for improvement of automatic alignment • Conservative lower bound on divergence frequency • Effective solution: syntactic transformation of English • Validity of solution shown through alignment experiments

  17. How do we evaluate MT? • Human-based Metrics • Semantic Invariance • Pragmatic Invariance • Lexical Invariance • Structural Invariance • Spatial Invariance • Fluency • Accuracy • “Do you get it?” • Automatic Metrics: Bleu

  18. BiLingual Evaluation Understudy (BLEU —Papineni, 2001) http://www.research.ibm.com/people/k/kishore/RC22176.pdf • Automatic Technique, but …. • Requires the pre-existence of Human (Reference) Translations • Approach: • Produce corpus of high-quality human translations • Judge “closeness” numerically (word-error rate) • Compare n-gram matches between candidate translation and 1 or more reference translations

  19. Bleu Comparison Chinese-English Translation Example: Candidate 1: It is a guide to action which ensures that the military always obeys the commands of the party. Candidate 2: It is to insure the troops forever hearing the activity guidebook that party direct. Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party.

  20. How Do We Compute Bleu Scores? • Key Idea: A reference word should be considered exhausted after a matching candidate word is identified. • For each word compute: (1) candidate word count (2) maximum ref count • Add counts for each candidate word using the lower of the two numbers . • Divide by number of candidate words..

  21. Modified Unigram Precision: Candidate #1 It(1) is(1) a(1) guide(1) to(1) action(1) which(1) ensures(1) that(2) the(4) military(1) always(1) obeys(0) the commands(1) of(1) the party(1) Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. What’s the answer?????? 17/18

  22. Modified Unigram Precision: Candidate #2 It(1) is(1) to(1) insure(0) the(4) troops(0) forever(1) hearing(0) the activity(0) guidebook(0) that(2) party(1) direct(0) Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. What’s the answer?????? 8/14

  23. Modified Bigram Precision: Candidate #1 It is(1) is a(1) a guide(1) guide to(1) to action(1) action which(0) which ensures(0) ensures that(1) that the(1) the military(1) military always(0) always obeys(0) obeys the(0) the commands(0) commands of(0) of the(1) the party(1) Reference 1: It is a guide to action that ensures that the military will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. What’s the answer?????? 10/17

  24. Modified Bigram Precision: Candidate #2 It is(1) is to(0) to insure(0) insure the(0) the troops(0) troops forever(0) forever hearing(0) hearing the(0) the activity(0) activity guidebook(0) guidebook that(0) that party(0) party direct(0) Reference 1: It is a guide to action that ensures that themilitary will forever heed Party commands. Reference 2: It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party. What’s the answer?????? 1/13

  25. Catching Cheaters the(2) the the the(0) the(0) the(0) the(0) Reference 1: The cat is on the mat Reference 2: There is a cat on the mat What’s the unigram answer? 2/7 What’s the bigram answer? 0/7

  26. CMSC 723 / LING 645: Intro to Computational Linguistics September 8, 2004: Monz Regular Expressions and Finite State Automata (J&M 2) Prof. Bonnie J. DorrDr. Christof MonzTA: Adam Lee

  27. Regular Expressions and Finite State Automata • REs: Language for specifying text strings • Search for document containing a string • Searching for “woodchuck” • How much wood would a woodchuckchuck if a woodchuck would chuck wood? • Searching for “woodchucks” with an optional final “s” • Finite-state automata (FSA)(singular: automaton)

  28. Regular Expressions • Basic regular expression patterns • Perl-based syntax (slightly different from other notations for regular expressions) • Disjunctions /[wW]oodchuck/

  29. Regular Expressions • Ranges [A-Z] • Negations [^Ss]

  30. *+ Stephen Cole Kleene Regular Expressions • Optional characters ? ,* and + • ? (0 or 1) • /colou?r/  colororcolour • * (0 or more) • /oo*h!/  oh! or Ooh! or Ooooh! • + (1 or more) • /o+h!/  oh! or Ooh! or Ooooh! • Wild cards .- /beg.n/  begin or began or begun

  31. Regular Expressions • Anchors ^ and $ • /^[A-Z]/  “Ramallah, Palestine” • /^[^A-Z]/  “¿verdad?” “really?” • /\.$/  “It is over.” • /.$/  ? • Boundaries \b and \B • /\bon\b/  “on my way” “Monday” • /\Bon\b/  “automaton” • Disjunction | • /yours|mine/  “it is either yours or mine”

  32. Disjunction, Grouping, Precedence • Column 1 Column 2 Column 3 …How do we express this? /Column [0-9]+ */ /(Column [0-9]+ *)*/ • Precedence • Parenthesis () • Counters * + ? {} • Sequences and anchors the ^my end$ • Disjunction | • REs are greedy!

  33. Perl Commands While ($line=<STDIN>){ if ($line =~ /the/){ print “MATCH: $line”; } }

  34. Writing correct expressions Exercise: Write a Perl regular expression to match the English article “the”: /the/ /[tT]he/ /\b[tT]he\b/ /[^a-zA-Z][tT]he[^a-zA-Z]/ /(^|[^a-zA-Z])[tT]he[^a-zA-Z]/

  35. A more complex example Exercise: Write a regular expression that will match “any PC with more than 500MHz and 32 Gb of disk space for less than $1000”: /$[0-9]+/ /$[0-9]+\.[0-9][0-9]/ /\b$[0-9]+(\.[0-9][0-9])?\b/ /\b$[0-9][0-9]?[0-9]?(\.[0-9][0-9])?\b/ /\b[0-9]+ *([MG]Hz|[Mm]egahertz| [Gg]igahertz)\b/ /\b[0-9]+ *(Mb|[Mm]egabytes?)\b/ /\b[0-9](\.[0-9]+) *(Gb|[Gg]igabytes?)\b/

  36. Advanced operators

  37. Substitutions and Memory Substitute as many times as possible! • Substitutions s/colour/color/ s/colour/color/g Case insensitive matching s/colour/color/i • Memory (\1,\2, etc. refer back to matches) s/([0-9]+)/<\1>/ /the (.*)er they were, the \1er they will be/ /the (.*)er they (.*), the \1er they \2/

  38. Eliza [Weizenbaum, 1966] User: Men are all alike ELIZA: IN WHAT WAY User: They’re always bugging us about something or other ELIZA: CAN YOU THINK OF A SPECIFIC EXAMPLE? User: Well, my boyfriend made me come here ELIZA: YOUR BOYFRIEND MADE YOU COME HERE User: He says I’m depressed much of the time ELIZA: I AM SORRY TO HEAR THAT YOU ARE DEPRESSED

  39. Eliza-style regular expressions Step 1: replace first person references with second person references s/.* YOU ARE (depressed|sad) .*/I AM SORRY TO HEAR YOU ARE \1/ s/.* YOU ARE (depressed|sad) .*/WHY DO YOU THINK YOU ARE \1/ s/.* all .*/IN WHAT WAY/ s/.* always .*/CAN YOU THINK OF A SPECIFIC EXAMPLE/ s/\bI(’m| am)\b/YOU ARE/g s/\bmy\b/YOUR/g S/\bmine\b/YOURS/g Step 2: use additional regular expressions to generate replies Step 3: use scores to rank possible transformations

  40. Finite-state Automata • Finite-state automata (FSA) • Regular languages • Regular expressions

  41. baa! baaa! baaaa! baaaaa! ... /baa+!/ a b a a ! q0 q1 q2 q3 q4 finalstate state transition Finite-state Automata (Machines)

  42. q0 a b a ! b a b a a ! 0 1 2 3 4 Input Tape REJECT

  43. q4 q0 q1 q2 q3 q3 a b a a a b a a ! ! 0 1 2 3 4 Input Tape ACCEPT

  44. Finite-state Automata • Q: a finite set of N states • Q = {q0, q1, q2, q3, q4} • : a finite input alphabet of symbols •  = {a, b, !} • q0: the start state • F: the set of final states • F = {q4} • (q,i): transition function • Given state q and input symbol i, return new state q' • (q3,!)  q4

  45. State-transition Tables

  46. D-RECOGNIZE function D-RECOGNIZE (tape, machine) returns accept or rejectindex Beginning of tapecurrent-state  Initial state of machineloopif End of input has been reached thenif current-state is an accept state thenreturn acceptelsereturn rejectelsiftransition-table [current-state, tape[index]] is empty thenreturn rejectelsecurrent-state  transition-table [current-state, tape[index]]index  index + 1end

  47. ! ! b ! b ! b b a a qF Adding a failing state a b a a ! q0 q1 q2 q3 q4

  48. a b a a ! q0 q1 q2 q3 q4 = = = = qF Adding an “all else” arc

  49. Languages and Automata • Can use FSA as a generator as well as a recognizer • Formal language L: defined by machine M that both generates and recognizes all and only the strings of that language. • L(M) = {baa!, baaa!, baaaa!, …} • Regular languages vs. non-regular languages

  50. Languages and Automata • Deterministic vs. Non-deterministic FSAs • Epsilon () transitions

More Related