Towards Application of User-Tailored Machine Translation in Localization

1. Towards Application of User-Tailored Machine Translation in Localization Andrejs Vasiļjevs, Raivis Skadiņš, Inguna Skadiņa TILDE JEC 2011, Luxembourg October14, 2011

2. MT in Localization Work • The localization process is generally related to the translation and cultural adaptation of software, video games, and websites, and less frequently to any written translation • Translation Memory is commonly used • Although there are number of MT use-cases in practical localization process, it is not yet widely adapted

3. Previous Work • Evaluation with keyboard-monitoring program and Choice Network Analysis for measuring the effort involved in post-editing MT output (O´Brien, 2005) • Productivity tests have been performed in translation and localization industry settings at Microsoft (Schmidtke, 2008): • SMT system of Microsoft Research trained on MS tech domain for 3 languages for Office Online 2007 localization task: Spanish, French and German • By applying MT to all new words on average 5-10% productivity was gained

4. Previous Work – Adobe • In Adobe two experiments were performed (Flournoy and Duran, 2009): • Small test set of 800-2000 words was machine translated and post-edited • Then, based on the positive results, about 200,000 words of new text were localized • The rule-based MT was used for translation into Russian (PROMT) and SMT for Spanish and French (Language Weaver) • Productivity increase between 22% and 51%

5. Previous Work – Autodesk • Evaluation of Autodesk Moses SMT system (Plitt and Masselot, 2010): • Translation from English to French, Italian, German and Spanish with three translators for each language pair • To measure translation time special workbench was designed to capture keyboard and pause times for each sentence • MT allowed translators to improve their throughput on average by 74% • Varying increase in productivity: from 20% to 131% • Optimum throughput has been reached for sentences of around 25 words in length

6. LetsMT! project • User-driven cloud-based MT factory, based on open-source MT tools • Services for data collection, MT generation, customization and running of variety of user-tailored MT systems • Application in localization among the key usage scenarios • Strong synergy with FP7 project ACCURAT to advance data-driven machine translation for under-resourced languages and domains

7. Partnership with Complementing Competencies • Tilde (Project Coordinator) - Latvia • University of Edinburgh - UK • University of Zagreb - Croatia • Copenhagen University - Denmark • Uppsala University - Sweden • Moravia – Czech Republic • SemLab – Netherlands

8. LetsMT! Main Features • Onlinecollaborative platform for MT buildingfromuser-provideddata • Repositoryof parallelandmonolingualcorporaforMT generation • Automated training of SMT systems from specified collections of data • Userscanspecify particular training data collections and build customised MT engines from these collections • Userscanalsouse LetsMT! platform for tailoring MT system to their needs from their non-public data

9. User requirement analysis • 21 interviews have been conducted with localization/translation agencies. • Only 7 of the respondents replied that they use fully automatic MT systems in their translation practice and only one LSP organization employs MT as the primary translation method.

10. User survey

11. User survey

13. LetsMT! architecture • Multitier architecture • User interface (webpage UI, web service API) • Application Logic • Resource Repository(stores MT training data and trained models) • High-performance Computing Cluster(executes all computationally heavy tasks: SMT training, MT service, Processing and aligning of training data etc.)

14. LetsMT! architecture • Webpage interfacewhere users can • See, upload and manage corpora • See, train and manage user tailored SMT system • Translate texts and files • Web service API • Integration with CAT tools • Integration in web pages and web browsers

15. LetsMT! architecture • Resource Repository • Stores SMT training data • Supports different formats – TMX, XLIFF, PDF, DOC, plain text • Converts to unified format • Performs format conversions and alignment

16. Based on Moses SMT toolkit • Training tasks are managed with Moses Experiment Management System • Training tasks are executed in HPC cluster (Oracle Grid Engine) • Hosted in Web Services infrastructure which provides easy access to on demand computing resources • New Moses features: • incremental training, • distributed language models, • interpolated language models for domain adaptation • randomized language models to train using huge corpora • translation of formatted texts, • running Moses decoder in a server mode

21. Bilingual corpora for the English-Latvian system

22. Latvian monolingual corpora

23. The MT System • Tilde English-Latvian MT • Tools: Moses, Giza++, SRILM, Tilde Latvian morphological analyzer • Factoredphrase-based SMT system: • Surface form  Surface form, Morphology tag • 2 Language models: • (1) 5-gram surface form • (2) 7-gram morphology tag

24. The MT system • Development and evaluation data • Development - 1000 sentences • Evaluation – 500 sentences • Balanced • BLEU score: 35.0

25. Evaluate original / assign Translator and Editor Import into SDL TradosAnalyze against TMs Translateusing translation suggestions for TMs Evaluate translation quality / Edit (optional) Fix errors Hand over to customer Localization workflow at Tilde

26. Evaluate original / assign Translator and Editor Analyze against TMs Translateusing translation suggestions for TMsand MT Evaluate translation quality / Edit Fix errors Ready translation MT Integration into Localization Workflow MT translate new sentences

27. Evaluation of Productivity • Key interest of localization industry is to increase productivity of translation process while maintaining required quality level • Productivity was measured as the translation output of an average translator in words per hour • 5 translators participated in evaluation including both experienced and new translators

28. Evaluation of Quality • Performed by human editors as part of their regular QA process • Result of translation process was evaluated, editors did not know was or was not MT applied to assist translator • Comparison to reference is not part of this evaluation • Tilde standard QA assessment form was used covering the following text quality areas: • Accuracy • Spelling and grammar • Style • Terminology

29. Error Score

30. QA Evaluation Form

31. QA Grades Tilde Localization QA assessment applied in the evaluation

32. Evaluation data • 54 documents in IT domain • 950-1050 adjusted words in each document • Each document was split in half: • the first part was translated using suggestions from TM only • the second half was translated using suggestions from both TM and MT

33. Integration of MT in SDL Trados

34. Evaluation Results • Average translation productivity: • Baseline with TM only: 550 w/h • With TM and MT: 731 w/h 32.9% productivity increase • High variability in individual performance with productivity increase by 64% to decrease by 5%. • Increase of error score from 20.2 to 28.6 points but still at the level “GOOD” (<30 points)

35. Average error scores by error types Biggest quality degradation in language quality due to increase of grammatical errors.

36. Conclusions • The results of our experiment clearly demonstrate that it is feasible to integrate the current state of the art SMT systems for highly inflected languages into the localization process. • Quality of translation decreases in all error categories still degradation is not critical and the result is acceptable for production purposes. • Significant performance differences for individual translators hints to the role of human factors such as training, work procedures, personal inclinations etc. • Extended experiments are planned (I) involving more translators, (II) translating texts in different domains and (III) in other language pairs.

37. Application in localization workflow

38. Thank you! Let's MT! The research within the LetsMT! project leading to these results has received funding from the ICT Policy Support Programme (ICT PSP), Theme 5 – Multilingual web, grant agreement no 250456