1 / 25

MT Evaluation CA446 Week 9

Andy Way School of Computing Dublin City University, Dublin 9, Ireland away@computing.dcu.ie. MT Evaluation CA446 Week 9. MT Evaluation. Source only! Manual: Subjective Sentence Error Rates Correct/Incorrect Error categorization Objective Usage Testing. Automatic:

mary
Download Presentation

MT Evaluation CA446 Week 9

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Andy Way School of Computing Dublin City University, Dublin 9, Ireland away@computing.dcu.ie MT EvaluationCA446 Week 9

  2. MT Evaluation • Source only! • Manual: • Subjective Sentence Error Rates • Correct/Incorrect • Error categorization • Objective Usage Testing • Automatic: • Exact Match (SER), WER, BLEU, NIST etc.

  3. Online (RB)MT Systems • E.g. Babelfish, Logomedia, etc. Are better than you might think! • Especially now we have Google Translate and Bing Translator …

  4. MT Evaluation by Humans • Human MT evaluation types: • Ranking • Rating: • Fluency and Adequacy scored by (subjective) human judges • Need both Fluency and Adequacy: Source Sentence: Le chat entre dans la chambre. Adequate Fluent translation: The cat enters the room. Adequate Disfluent translation: The cat enters in the bedroom. Fluent Inadequate translation: My Granny plays the piano. Disfluent Inadequate translation: piano Granny the plays My.

  5. MT Evaluation by Humans • Problems: • Inter-rater (and intra-rater) agreement • Very expensive (w.r.t time and money) • Not always available • Can’t help day-to-day system development

  6. Automatic Machine Translation Evaluation • Objective • Inspired by the Word Error Rate metric used by ASR research • Measuring the “closeness” between the MT hypothesis and human reference translations • Precision: n-gram precision • Recall: • Against the best matched reference • Approximated by brevity penalty • Cheap, fast • Highly correlated with human evaluations • MT research has greatly benefited from automatic evaluations • Typical metrics: BLEU, NIST, F-Score, Meteor, TER

  7. BLEU Evaluation Metric Reference (human) translation: The US island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an e-mail from someone calling himself Osama Bin Laden and threatening a biological/chemical attack against the airport. • N-gram precision (score between 0 & 1) • what % of machine n-grams (a sequence of words) can be found in the reference translation? • Brevity Penalty • Can’t just type out single word “the’’ (precision 1.0!) NB, Extremely hard to trick the system, i.e. find a way to change MT output so that BLEU score increases, but quality doesn’t. Machine translation: The American [?] International airport and its the office a [?] receives one calls self the sand Arab rich business [?] and so on electronic mail, which sends out; The threat will be able after the maintenance at the airport.

  8. More Reference Translations are Better Reference translation 1: The US island of Guam is maintaining a high state of alert after the Guam airport and its offices both received an e-mail from someone calling himself Osama Bin Laden and threatening a biological/ chemical attack against the airport. Reference translation 2: Guam International Airport and its offices are maintaining a high state of alert after receiving an e-mail that was from a person claiming to be the rich Saudi Arabian businessman Osama Bin Laden and that threatened to launch a biological and chemical attack on the airport. Machine translation: The American [?] International airport and its the office a [?] receives one calls self the sand Arab rich business [?] and so on electronic mail , which sends out; The threat will be able after the maintenance at the airport to start the biochemistryattack. Reference translation 3: The US International Airport of Guam and its office has received an email from a self-claimed Arabian millionaire named Laden , which threatens to launch a biochemical attack on airport. Guam authority has been on alert. Reference translation 4: US Guam International Airport and its offices received an email from Mr. Bin Laden and other rich businessmen from Saudi Arabia. They said there would be biochemistry air raid to Guam Airport. Guam needs to be in high precaution about this matter.

  9. BLEU in action • Reference Translation: The gunman was shot to death by the police . • The gunman was shotkill . • Woundedpolicejaya of • The gunman was shotdeadby the police . • The gunmanarrestedby policekill . • Thegunmenwere killed . • The gunman was shot to death by the police . • Theringer is killedby the police . • Policekilledthe gunman . • Green = 4-gram match (good!) Red = unmatched word (bad!)

  10. BLEU Metrics • Proposed by IBM’s SMT group (Papineni et al, ACL-2002) • Widely used in MT evaluations • DARPA TIDES MT evaluation (www.darpa.mil/ipto/programs/tides/strategy.htm) • IWSLT evaluation (www.slt.atr.co.jp/IWSLT2004/) • TC-Star (www.tc-star.org/) • BLEU Metric: • Pn: Modified n-gram precision • Geometric mean of p1, p2,..pn • BP: Brevity penalty (c=length of MT hypothesis, r=length of reference) • Usually, N=4 and wn=1/N.

  11. An Example • MT Hypothesis: The gunman was shot dead by police . • Ref 1: The gunman was shot to death by the police . • Ref 2: The gunman was shot to death by the police . • Ref 3: Police killed the gunman . • Ref 4: The gunman was shot dead by the police . • Precision: p1=1.0(8/8) p2=0.86(6/7) p3=0.67(4/6) p4=0.6 (3/5) • Brevity Penalty: c=8, r=9, BP=0.8825 • Final Score:

  12. Sample BLEU Performance • Reference: George Bush will often take a holiday in Crawford Texas George Bush will often take a holiday in Crawford Texas (1.000) Bush will often holiday in Texas (0.4611) Bush will often holiday in Crawford Texas (0.6363) George Bush will often holiday in Crawford Texas (0.7490) George Bush will not often vacation in Texas (0.4491) George Bush will not often take a holiday in Crawford Texas (0.9129)

  13. Sample BLEU Performance • Reference: George Bush will often take a holiday in Crawford Texas George Bush will often take a holiday in Crawford Texas (1.000) Bush will often holiday in Texas (0.4611) Bush will often holiday in Crawford Texas (0.6363) George Bush will often holiday in Crawford Texas (0.7490) George Bush will not often vacation in Texas (0.4491) George Bush will not often take a holiday in Crawford Texas (0.9129) NB, BLEU was never designed to be used at the segment-level, but rather at the document-level ….

  14. Content of ‘gold standard’ matters! • Which is better? • George Bush often takes a holiday in Crawford Texas. • Holiday often Bush a takes George in Crawford Texas. What would BLEU say (assume max. bigrams important)? What if human reference was: The President frequently makes his vacation in Crawford Texas.

  15. Content of ‘gold standard’ matters! (2) • Sometimes, the reference translation is impossible for any MT system (current or future) to match: • From Canadian Hansards: • Again, this was voted down by the Liberal majority => • Malheureusement, encore une fois, la majorité libérale l’ a rejeté • [Unfortunately, still one time, the majority liberal it has rejected] • Of course, human translators are quite entitled to do this sort of thing, and do so all the time …

  16. Correlation between BLEU score and Training Set Size? BLEU score Experiments by Philipp Koehn No. sentence pairs used in training

  17. An Improvement? • (cf. Hovy & Ravichandran MT-Summit 2003) • Ref: The President frequently makes his vacation in Crawford Texas. • MT1: George Bush often takes a holiday in Crawford Texas • (0.2627 BLEU 4-gram) • MT2: Holiday often Bush a takes George in Crawford Texas • (0.2627 BLEU 4-gram) • POS only: • Ref: DT NNP RB VBZ PRP$ NN IN NNP NNP • MT1: NNP NNP RB VBZ DT NN IN NNP NNP (0.5411) • MT2: NN RB NNP DT VBZ NNP IN NNP NNP (0.3217) • (Words + POS)/2: • MT1: (0.4020) • MT2: (0.2966)

  18. Amendments to BLEU • NIST • Weights more heavily those n-grams that are more informative (i.e. rarer ones) • Less punitive brevity penalty measure • Pro: more sensitive than BLEU • Con: information gain for >unigram not meaningful, i.e. 80% of NIST score comes from unigram matches • Modified BLEU (Zhang 2004) • More balanced contribution from different n-grams

  19. Are Two MT Systems Different? Zhang & Vogel (TMI-04): Compared two MT systems’ performance using confidence intervals, i.e. from a population of BLEU/M-BLEU/NIST scores per test set of translations (Chinese-English), measure accuracy (sensitivity, consistency) of these scores • M-Bleu and NIST have more discriminative power than Bleu • Automatic metrics have pretty high correlations with the human ranking • Human judges like system E (Syntactic system) more than B (Statistical system), but automatic metrics do not

  20. Problems with BLEU • It can be easy to look good (cf. output from current ‘state-of-the-art’ SMT systems) • Not currently very sensitive to global syntactic structure (disputable) • Doesn’t care about nature of untranslated words: • gave it to Bush • gave it at Bush • gave it to rhododendron • As MT improves, BLEU won’t be ‘good enough’ …

  21. Problems with using BLEU • Not designed to test individual sentences • Not meant to compare different MT systems Extremely useful tool for system developers! Q: what/who is evaluation for? cf. [Callison-Burch et al., EACL-06]

  22. Newer Evaluation Metrics • P&R (GTM: Turian et al., MT-Summit 03) • RED (Akiba et al., MT-Summit 01) [based on edit distance, cf. WER/PER …] • ORANGE (Lin & Och, COLING-04) • Classification by Learning (Kulesza & Shieber, TMI-04) • Meteor (Banerjee & Lavie, ACL-05)

  23. Other Places to Look • BLEU/NIST: www.nist.gov/speech/tests/mt/resources/scoring.htm • GTM: nlp.cs.nyu.edu/GTM/ • EAGLES: www.issco.unige.ch/ewg95/ewg95.html • FEMTI: www.isi.edu/natural-language/mteval/ • www.redbrick.dcu.ie/~grover/mteval/mteval.html • MT Summit/LREC workshops etc etc … => MT Evaluation is (one of) the flavour(s) of the month …

  24. Is MT-Eval for people who can’t do MT? • I used to say so (somewhat mischievously), but some groups that have come up with MT-Eval metrics include: • Aachen (Ney) • Google (Och) • CMU (Lavie, Vogel) • NYU (Melamed) • Edinburgh (Koehn)

  25. Is MT-Eval for people who can’t do MT? • I used to say so (somewhat mischievously), but some groups that have come up with MT-Eval metrics include: • Aachen (Ney) • Google (Och) • CMU (Lavie, Vogel) • NYU (Melamed) • Edinburgh (Koehn) • DCU (Way)

More Related