1 / 14

MADCAT Evaluation Approach: Operational Accuracy of MT Applied to OCR

MADCAT Evaluation Approach: Operational Accuracy of MT Applied to OCR. Caitlin Christianson, Bonnie Dorr*, Joseph Olive Defense Advanced Research Projects Agency and * Florida IHMC & University of Maryland LREC-2014 MTE Workshop. Approved for Public Release, Distribution Unlimited.

derica
Download Presentation

MADCAT Evaluation Approach: Operational Accuracy of MT Applied to OCR

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MADCAT Evaluation Approach: Operational Accuracy of MT Applied to OCR Caitlin Christianson, Bonnie Dorr*, Joseph Olive Defense Advanced Research Projects Agency and *Florida IHMC & University of Maryland LREC-2014 MTE Workshop Approved for Public Release, Distribution Unlimited

  2. The Question What accuracy is necessary for a given application? Editable: Publishable with human editing Gistable: Suitable for human readers to determine basic meaning Triageable: Suitable for determining mission relevance. Approved for Public Release, Distribution Unlimited

  3. Experiment Setup Stimuli Preparation Test Translation 4 Intel Analysts MT Editor Correction List Task Judgments:A=Can you describe the topic and determine relevance?B=Can you summarize meaning and pull key facts out?C=Is it publishable with minor editing and are portions usable in reports? ........ . He told that it is expected to pointed out that he expects a firm intervention by Washingtonfrom Washington’s side to stop the that deal, saying that those missiles could be used in the shooting down of the cargo aircraft, which was carrying Randomly chosen corrections from list وستنقل الطائرة القادمة من جدة حجاجا ليبيين الى بلدهم وستعبر الاجواء المصرية الخميس Scores Translator Doc 1 - . . Doc 6 A . . Doc 8 A,B . . Doc 12 A,B,C 90% 50% 55% HT 95% Approved for Public Release, Distribution Unlimited

  4. Experiment Results Triageable = 65% (Arabic) (55% for Spanish) (Arabic) Gistable = 75% (70% for Spanish) Task Judgments:A=Can you describe the topic and determine relevance?B=Can you summarize meaning and pull key facts out?C=Is it publishable with minor editing and are portions usable in reports? (Arabic) (85% for Spanish) Editable = 85% Scores This study led to the birth of Human-mediated Translation Error Rate (HTER) Doc 1 - . . Doc 6 A . . Doc 8 A,B . . Doc 12 A,B,C Approved for Public Release, Distribution Unlimited

  5. No. of errors Accuracy = 1 No. of words HTER-Based MT Accuracy Evaluation Foreign Language Text & Speech GALE Machine Translation Engine Translators GALE Machine Translation Evaluators Human Editors who conduct comparison Adjudicator Which is right? Can it be ambiguous? Is it an idiom? Gold Standard Translation Approved for Public Release, Distribution Unlimited

  6. Corrected machine translation MT Accuracy Measurement Example Human-Translated Reference The statement said that “your brothers in the military wing of the Al-Qaeda Jihad Organization in Mesopotamia carried out an assassination of one of the criminal tyrants in the city of Baquba.” Corrected machine translation The statement said thattheyour brothers in the military wing to regulate Al Jihad base in the country had carried out the assassination of one of the criminals in the city of penalty. 1 error Machine translation The statement said that the brothers in the military wing to regulate Al Jihad base in the country had carried out the assassination of one of the criminals in the city of penalty. Corrected machine translation The statement said that theyour brothers in the military wing to regulateof the Al Jihad base in the country had carried out the assassination of one of the criminals in the city of penalty. 5 errors Corrected machine translation The statement said that theyour brothers in the militarywing to regulate of the Al QaedaJihad base in the country had carried out the assassination of one of the criminals in the city of penalty. 6 errors Corrected machine translation The statement said that the your brothers in the military wing to regulateof the Al Qaeda Jihadorganizationbaseinthe country Mesopotamia had carried out the assassination of one of the criminal tyrantsin the city of penalty Baquba. 11 errors in 33 words (67% accuracy) Approved for Public Release, Distribution Unlimited

  7. Operational Accuracy of MT Applied to OCR Translation Accuracy on Program-Generated Documents Translation Accuracy on Field-Collected Documents Translation Accuracy (HTER) Translation Accuracy (HTER) Percentage of Documents Percentage of Documents Editable Gistable Triageable Approved for Public Release, Distribution Unlimited

  8. What are the Take-Aways? • Evaluation of MT Accuracy depends on operational context and user requirements for task completion. • HTER is an effective (meaning-based) means of evaluating OCR extrinsically vs. standard (shallow) transcription error rates. • Determining performance levels on data with operational characteristics is crucial for determining usefulness for end users. • True usefulness derives from relating accuracy judgments to user-relevant utility levels. Approved for Public Release, Distribution Unlimited

  9. BACK UPS Approved for Public Release, Distribution Unlimited

  10. Human Translation He pointed out that he expects a firm intervention from Washington’s side to stop that deal, saying that those missiles could be used in the shooting down of the cargo aircrafts that carry supplies for the forces in Iraq. Approved for Public Release, Distribution Unlimited

  11. Machine Translation He told that it is expected to firm intervention by Washington to stop the deal, saying that those missiles could be used in the shooting down of cargo aircraft, which was carrying supplies of forces in Iraq. Approved for Public Release, Distribution Unlimited

  12. 100% Corrected Machine Translation He told that it is expected to pointed out that he expects a firm intervention by Washington from Washington’s side to stop the that deal, saying that those missiles could be used in the shooting down of the cargo aircraft, which was carrying s that carry supplies of for the forces in Iraq. Approved for Public Release, Distribution Unlimited

  13. 50% Translation He told that it is expected to firm intervention by Washington from Washington’s side to stop the that deal, saying that those missiles could be used in the shooting down of the cargo aircraft, which was carrying s that carry supplies of for the forces in Iraq. Approved for Public Release, Distribution Unlimited

  14. More Refined Grading System Approved for Public Release, Distribution Unlimited

More Related