1 / 16

Quantitative Evaluation of Machine Translation Systems: Sentence Level

Quantitative Evaluation of Machine Translation Systems: Sentence Level. Palmira Marrafa António Ribeiro. Outline. Motivation ISO Characteristics to Measure Draft Proposals Future Work. Motivation. Evaluate the Quality of Translated Sentences Assumption

dian
Download Presentation

Quantitative Evaluation of Machine Translation Systems: Sentence Level

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quantitative Evaluation of Machine Translation Systems: Sentence Level Palmira Marrafa António Ribeiro

  2. Outline • Motivation • ISO Characteristics to Measure • Draft Proposals • Future Work

  3. Motivation • Evaluate the Quality of Translated Sentences • Assumption • It is possible to quantify the quality of Translations • Design Evaluation Measures

  4. ISO Characteristics to Measure 2.2 System external characteristics 1 Functionality 2 Accuracy ...

  5. ISO Characteristics to Measure 2 Individual sentence level 1 Morphology 2 Syntax (sentence and phrase structure) 3 Types of errors 3 Lexical errors 4 Syntax errors 5 Stylistic errors

  6. Characteristics to Measure • Types of Errors • Lexicon • Syntax • Morphology

  7. MT Systems Used • Systems • S1 • S2

  8. Lexicon • « «Lexical» errors refer words or phrases that are inappropriate … » • Example • Input: Foot-and-mouth disease • Output: Febre aftosa • Systran: *Doenca de pé-e-boca • Intertran: *Pé-e-boca doenca

  9. Lexicon: Evaluation • Collocations • Fixed Expressions • Semi-fixed Expressions

  10. Lexicon: Evaluation • 4-point scale • 0 wrong (Doenca de pé-e-boca) • 1 marginal (Febre dos pés e da boca) • 2 inadequate (Doenca aftosa) • 3 correct (Febre aftosa)

  11. Lexicon: Evaluation • Scores • Average of individual word scores • Order to be included • Foot-and-mouth disease vs • Disease of foot-and-mouth

  12. Syntax • ««Syntax» is concerned with grammaticality» • Previous work: • DARPA: 5-point scale

  13. Syntax: Evaluation • Comprehensive typology of Errors in • NPs • PPs • VPs • Predication Level • e.g. subject verb agreement

  14. Syntax: Evaluation • NPs • Specifiers • todos os estudantes • all *the students • Modifiers • Co-occurrence Restrictions • Order • ...

  15. Syntax: Evaluation • VPs • Tense • Mode • Aspect • Modality (should vs might) • Negation • Complements order • Preposition Selection • ...

  16. Future Work • Generalise across the Categories • to build the metrics • Assign scores to each error • Test the metrics

More Related