Developing a multilingual text analysis engine - does using Unicode solve all the issues?

1. Developing a multilingual text analysis engine - does using Unicode solve all the issues? Dr. Brian O’Donovan, IBM Ireland, Sept. 2002

2. Agenda • What is IBM LanguageWare? • Where/How is it Used? • Our Experiences of Conversion to Unicode • Benefits Accruing • Challenges to be Overcome • Word Identification problems • Future work

3. What is IBM LanguageWare? • A suite of tools to assist in linguistic analysis • Initially developed by IBM USA, but now developed by a Globally distributed team • Used in internal and external products • Available to OEMs as a toolkit

4. Where are We? • Developers • North Carolina, USA • Dublin, Ireland • Helsinki, Finland • Taipei, Taiwan • Yamato, Japan • Seoul, Korea • Major Customers • Various Locations, USA • Böblingen, Germany

5. Afrikaans Arabic Catalan Simplified Chinese Traditional Chinese Czech Danish Dutch (x2) English x4 regions & x2 domains Finnish French (x2) German (x3) Greek Hebrew Hungarian Icelandic Italian Japanese Korean Norwegian (x2) Polish Portuguese (x2) Russian Spanish Swedish Tamil Thai Turkish 28 (or 30+) Languages Supported

6. Lexical Analysis Hierarchy Summarization Grammar check Words = spell check, morphology, etc.

7. Where/How is it Used? • Word Processors • spell aid, grammar checker • Search Engines & Data Mining • Extracting Lemmatized form • Query Expansion (e.g. Synonyms) • Assisting in Taxonomy Generation • Translation Memory • Identification of linguistic units • NLP tools use as first stage of analysis • Machine Translation (limited) • Speech recognition (not currently)

8. Conversion to Unicode • Previous version used different code pages for each language • In fact a collection of different engines • New version has a single engine for all languages using Unicode (not all languages have been converted yet) • Provided many benefits • Raised some challenges

9. Benefits of Unicode • No more conversion tables • Standardized on utf-16 • Big-Endian Dictionaries • Converted to platform byte order on load • Possible to deal properly with multilingual text • Able to utilize ICU utilities • Can edit/view all test cases on one machine

10. Issues with Unicode • Lots of work • Converting dictionaries is not trivial • Changing code would be huge (we were rewriting anyway) • Large code page • Some ambiguous representations & orthographic rules • ICU Tokenisation not always to our liking

11. How we dealt with large code page • Our architecture is based upon Finite State Transducers (FSTs) • In FSTs you frequently need to build tables with entries for every possible next character • for single byte code page => 256 entries • for double byte code page => 64K entries • If not careful, dictionaries might grow to 256 times their previous size • We deal with this by building a character transition table per dictionary • Each UNICODE character appearing in the dictionary is assigned a code by which it is referenced within the dictionary • This is not a private code page • The table is dynamically created when building the dictionary and will map differently each time • When looking up dictionary each character is first mapped through the table • If any characters are not in mapping table => we won't find a match in the dictionary • New characters are automatically added to the mapping table once words with the characters are added to the dictionary • Typically European languages require fewer than 100 characters • e.g. punctuation symbols do not appear in the dictionary

12. FSA Node English Dictionary a b c Z

13. FSA Node Japanese Dictionary か 山 川 部

14. Representation Issues • Even with utf-16 as a universal standard representation issues can arise • The letter ë is represented as 0x00EB • But could be e=0x0065 followed by umlaut=0x0308 • Arabic letter Heh (ه) = E5 in Windows code page • In UNICODE 0x0647 • But it has different forms e.g. ههه • Unicode defines 4 more code points for presentation forms • isolated=0xFEE9, final=0xFEEA, initial=0xFEEB and medial=0xFEEC

15. Arabic Shape Bitmaps (Heh) • Heh varies with position • Isolated • Initial • Medial • Final

16. Arabic Shape Bitmaps (Seen) • Seen varies much less • Isolated • Initial • Medial • Final

17. Orthographic Variation • A word may appear differently in a sample text than in the dictionary. • All languages have different rules about what is allowed/required. • Some languages have simple rules • e.g. English casing rules • a lowercase dictionary word can appear in text as titlecase or uppercase • a titlecase dictionary word could appear in text as uppercase • an uppercase or mixed case dictionary word must appear in text exactly the same.

18. Other languages have more complex rules • In France letters lose their accent when capitalized • e.g. é is capitalized as E not É • but this rule does not apply in French Canada • So être becomes Etre in Paris but Être in Montreal • In German capitalization can change the letter count • 'ß' is sometimes capitalized as 'SS' • In English we optionally drop accents • e.g. the name Zoë can be written Zoe in English • But in German dropped umlauts are represented by a following e • e.g. Böblingen can be written Boeblingen

19. Vowel dropping in Semitic languages • Semitic languages such as Arabic and Hebrew, have an orthographic rule that short vowels may be dropped. • Imagine that Arabic has the words hat, hit, hut • They will be written in dictionary as • Hat = هَت • Hut = هُت • Hit = هِت • In any piece of text the string هت could be any of these words

20. Arabic examples in Large font • Hat = هَت • Hut = هُت • Hit = هِت • Ht = هت

21. Inflection Variants walk walked walking Surface Forms walk Walk WALK walked Walked WALKED walking Walking WALKING Root Word walk Model of How Orthographic & Representation variation handled Orthography & Representation Inflection

22. Inflection Variants passé Surface Forms passé Passe passe‌´ Passé Passe‌‌´ Passe PASSÉ PASSE PASSE‌‌́´ Root Word passé Model of How Orthographic & Representation variation handled Orthography & Representation Inflection

23. Recognize talk l k a t 4 3 2 0 1

24. Recognize walk or talk w l k a t 4 3 2 0 1

25. Recognize walk, walked, walking talk, talked or talking d 6 w 5 e l k a t 4 3 2 0 1 i g n 9 7 8

26. Recognize all variants of walk or talk d 6 w 5 e l k a t 4 3 2 0 1 i g W n 9 7 a 8 T D 15 14 E K L A 13 12 11 I 10 G N 18 16 17

27. Recognize all variants of wálk or tálk d 6 w 5 e k l á 4 t 2 3 1 0 i a g n 19 9 7 8 á W T D 15 a 14 E K L 13 12 11 I G Á N 18 10 16 17

28. Word Breaking • ICU break iterators can only provide us with a first pass of the word segmentation • Significant improvements in ICU 2.2 • Sometimes the word segmentation is not obvious • multi word expressions • compound words • languages with no spaces

29. Multiword Expressions • Word break is not always obvious • French pommes de terre • as 3 words = apples of the ground • as 1 word = potatoes • French l'intelligence artificielle • really 2 words le and intelligence artificielle • Sometimes the word break is ambigous • English red tape • as 2 words = tape with red color • as 1 word = synonym of bureaucracy

30. Decompounding • German and related languages allow speakers to generate their own compound words by combining component words • typically a series of nouns and/or adjectives • ICU considers these one word • but to analyse them properly we need to break it into its constituent words and look these up in the dictionary • Can be computationally expensive • We can also sometimes get ambiguous results

31. Ambiguous Decompounding - example 1 • Wachstube • Option 1 = wax tube • Wachs = wax • Tube = tube • Option 2 = guard room • Wache = guard • Stube = room • Option 3 = awake room • Wach = awake • Stube = room

32. Ambiguous Decompounding - example 2 • Hochschullehrer = University Lecturer • 3 component words • Hoch = High • Schule = school • Lehrer = teacher • Could index under • Hochschullehrer = Univesity Lecturer • Hochschule = High School & Lehrer = Teacher • Hoch = Higher & Shullehrer = school teacher • However Schullehrer (=schoolteacher) is a lexicalised compound • The two words are so often used together that native speakers no consider them to be a single word • Hoch & Schullehrer is only valid decomposition • In fact most German Speakers would regard Hochschullehrer as a single lexical word because it is commonly used

33. Ambiguous Decompounding - example 2 • Neuroschaltungsverstärkung • Neuro = neuronal • Schaltungs = circuit • verstärkung = amplification • Could be Neuroschaltungs|verstärkung • interpreted as an amplification of a "neuronal circuit" • Could also be Neuro|schaltungsverstärkung • interpreted as a kind of circuit amplification which is done by neuronal technology

34. Languages with no spaces • Some languages (e.g. Chinese, Japanese, Thai) do not put spaces between words • Therefore, it is difficult to figure out where the word breaks are • Sometimes there are multiple possible word segmentations. • Sometimes the choice of segmentation can change the meaning • For these languages we use ICU break iterator to find "unambiguous word break" (e.g. punctuation symbol, line break, character type change) • Looking in the dictionary we find all possible word combinations within this text sequence • Using statistical techniques we figure out which word sequence is most likely

35. 此 = this (ci) 路 = road (lu) 不 = no/not (bu) 通 = through (tong) 此路不通 = cul-de-sac (cilubutong) 行 = walk (xing) 得 = get (de) 不得 = forbidden (bu-de) 在 = be (ci) 在此 = here (zaici) 小 = small (xiao) 便 = convenience (bian) 小便 = urinate (xiaobian) 此路不通行 此路不通行不得在此小便 Interp 1 此 /路 /不 /通 //行不得 /在此 /小便 No through way for pedestrians. Urination forbidden here. Interp 2 此路不通 行不 / 得 在此 / 小便 Cul-de-sac. Walking Forbidden. Urinate here. Chinese Segmentation Example 此路不通行不得在此小便

36. 此 = this (ci) 路 = road (lu) 不 = no/not (bu) 通 = through (tong) 此路不通 = cul-de-sac (cilubutong) 行 = walk (xing) Interp 1 此 /路 /不 /通 //行 This road not through walk No through way for pedestrians. Chinese Segmentation Example 此路不通行

37. 此 = this (ci) 路 = road (lu) 不 = no/not (bu) 通 = through (tong) 此路不通 = cul-de-sac (cilubutong) 行 = walk (xing) 得 = get (de) 不得 = forbidden (bu-de) 在 = be (ci) 在此 = here (zaici) 小 = small (xiao) 便 = convenience (bian) 小便 = urinate (xiaobian) Interp 1 此 /路 /不 /通 //行不得 /在此 /小便 This road not through walkForbidden here urinate No through way for pedestrians. Urination forbidden here. Chinese Segmentation Example 此路不通行/不得在此小便

38. 此 = this (ci) 路 = road (lu) 不 = no/not (bu) 通 = through (tong) 此路不通 = cul-de-sac (cilubutong) 行 = walk (xing) 得 = get (de) 不得 = forbidden (bu-de) 在 = be (ci) 在此 = here (zaici) 小 = small (xiao) 便 = convenience (bian) 小便 = urinate (xiaobian) Interp 1 此 /路 /不 /通 //行不得 /在此 /小便 No through way for walkers. Urination forbidden here. Interp 2 此路不通 = cul-de-sac 行不 / 得 = walk not 在此 / 小便 = here urinate Cul-de-sac. Walking Forbidden. Urinate here. Chinese Segmentation Example 此路不通行不得在此小便

39. Future Challenges • Complete move of all languages to new architecture • Improving quality and breadth of linguistic data • more languages • more words (better user dictionary support) • richer relationships (e.g. part of, type of etc.) • Increasing Accuracy of analysis • Part of speech disambiguation • Ranking of parse results • Increasing performance speed • Latest version exceeds 2.5 Giga Char/hour on standard PC • Some customers say it is not fast enough, but this only allows 1.5 micro seconds/char • Available through ICU API