Finite-state automata and Morphology

1. Finite-state automata and Morphology

2. Outline • Morphology • What is it? • Why do we need it? • How do we model it? • Computational Model: Finite-state transducer

3. Structure of Words • What are words? • Orthographic tokens separated by white space. • In some languages the distinction between words and sentences is less clear. • Chinese, Japanese: no white space between words • nowhitespace  no white space/no whites pace/now hit esp ace • Turkish: words could represent a complete “sentence” • Eg: uygarlastiramadiklarimizdanmissinizcasina • “(behaving) as if you are among those whom we could not civilize” • Morphology: the structure of words • Basic elements: morphemes • Morphological Rules: how to combine morphemes. • Syntax: the structure of sentences • Rules for ordering words in a sentence

4. Morphology and Syntax • Interplay between syntax and morphology • How much information does a language allow to be packed in a word, and how easy is it to unpack. • More information  less rigid syntax  more free word order • Eg: Hindi: John likes Mary – all six orders are possible, due to rich morphological information.

5. Why Study Morphology? • Morphology provides • systematic rules for forming new words in a language. • can be used to verify if a word is legitimate in a language. • efficient storage methods. • improving lexical coverage of a system. • group words into classes. • Applications • Improving recall in search applications • Try “fish” as a query in a search engine • Text-to-speech synthesis • category of a word determines its pronunciation • Parsing • Morphological information eliminates spurious parses for a sentence

6. Structure of Words • Morphology: The study of how words are composed from smaller, meaning-bearing units (morphemes) • Stems: children, undoubtedly, • Affixes(prefixes, suffixes, circumfixes, infixes) • Immaterial, Trying, Gesagt, Absobl**dylutely • Different ways of creating words • Concatenative • Adding strings to the stems (examples above) • Non-concatenative • Infixation (e.g. Tagalog: hingi (order)  humingi) • Templatic (e.g. Arabic root-and-pattern) morphological systems • Triconsonant CCC expands into words using templates that represent more semantic information. • Hebrew: lmd (study)  lamad (he studied), limed (he taught), lumad (he was taught) • Stacking of affixes in English is limited (three or four); but in Turkish nine or ten affixes can be stacked. • Agglutinative languages

7. Different Classifications • Sets of words display (almost) regular patterns. Open class words: Noun, Verb, Adverb, Adjectives Closed class words: Pronoun, Preposition, Conjunction, Determiners. • Kinds of morphology Inflectional Word stem + grammatical morpheme results in a word of the same class. Morpheme serves a syntactic function: grammatical agreement, case marker -s for plural on nouns, -ed for past tense on verbs Only Nouns and Verbs inflect in English for grammatical reasons In French, adjectives agree (match on gender and number) with nouns they modify. Derivational Word stem + grammatical morpheme results in a different class of word. Nominalization: -ation suffix makes verbs into nouns (e.g. Computerize  computerization -ly suffix makes adjectives into adverbs (e.g. beautiful beautifully)

8. Inflectional Morphology • Word stem + grammatical morpheme • Morpheme serves a syntactic function: grammatical agreement, case marker • -s for plural on nouns, -ed for past tense on verbs • Results in word of the same class as the stem • (bat  bats; man  man’s; jump  jumps, jumped, jumping) • Noun morphology: • Regular nouns: bat bats, fish  fishes • Irregular nouns: spelling changes (mouse mice; ox  oxen) • Verb morphology: • Regular verbs: jump jumps (-s form), jumped (past/ -ed participle), jumping (-ing participle) • Irregular verbs: eat eats (-s form), ate (past), eaten (-ed participle), eating (-ing participle) • Irregular forms have to be stored. • Language learners typically make errors on irregular forms. • hit hitted

9. Inflectional Morphology • Nominal morphology • Plural forms • s ores • Irregular forms (goose/geese) • Mass vs. count nouns (fish/fish,email or emails?) • Possessives (cat’s, cats’) • Verbal inflection • Main verbs (sleep, like, fear) verbs relatively regular • -s, ing, ed • And productive: Emailed, instant-messaged, faxed, homered • But some are not regular: eat/ate/eaten, catch/caught/caught • Primary (be, have, do) and modal verbs (can, will, must) often irregular and not productive • Be: am/is/are/were/was/been/being • Irregular verbs few (~250) but frequently occurring • Irregularity a consequence of frequency? • So….English inflectional morphology is fairly easy to model….with some special cases...

10. (English) Derivational Morphology • Word stem+ grammatical morpheme • Usually produces word ofdifferentclass • More complicated than inflectional • E.g. verbs --> nouns • -ize verbs  -ation nouns • generalize, realize  generalization, realization • E.g.: verbs, nouns  adjectives • embrace, pity embraceable, pitiable • care, wit  careless, witless • E.g.: adjective  adverb • happy  happily • But “rules” have many exceptions • Less productive: *evidence-less, *concern-less, *go-able, *sleep-able • Meanings of derived terms harder to predict by rule • clueless, careless, nerveless

11. How do humans represent words? • Hypotheses: • Full listing hypothesis: words listed • Minimum redundancy hypothesis: morphemes listed • Words are formed using rules. • Experimental evidence: • Primingexperiments (Does seeing/hearing one word facilitate recognition of another?) suggest neither • Regularly inflected forms prime stem but not derived forms • But spoken derived words can prime stems if they are semantically close (e.g. government/govern but not department/depart) • Speech errors suggest affixes must be represented separately in the mental lexicon • easy enoughly

12. Finite-State Morphological Parsing • Morphological Parsing: • takes : root=take, category=V, person=3, number=sg • cities : root=city, category=N, number=plural • ships: a. root=ship, category=N, number=plural b. root=ship, category=V, person=3, number=sg • What does a parser need? • Lexicon: root forms of the words in a language • Morphological Rules: morpheme ordering rules • Orthographic Rules: spelling change rules • We will represent each of these components as finite-state automata. • Parsing versus Recognition • Recognition: yes/no • Parsing: derivation structure • indecipherable [in [[de [cipher]] able]]

13. Lexicon as an FSA • Words can be represented as paths in a FSA with characters as transition symbols. • Also called a “trie” structure List of words: bat, cat, mat, sat c a t b m s

14. q1 q2 q0 Morphological Recognition reg-n plural (-s) • Check if a word is a valid word in the language. • For nouns: cats, geese, goose • For adjectives: Derivational morphology • Adj-root1: clear, happy, real (clearly) • Adj-root2: big, red (~bigly) q0 q1 q2 irreg-pl-n irreg-sg-n adj-root1 un- -er, -ly, -est adj-root1 q5 q3 q4 -er, -est  adj-root2

15. Morphotactic Rules • An automaton that encodes valid English word forming rules. • Koskenniemi’s two-level morphology • Lexical string:stem and morphemes • Surface string:spelling of the derived word • Pair up lexical string with the surface string. • (N,bat,pl; bats) • (V,bat,3sg; bats) • Align the strings at the character level • N:eb:b a:at:t pl:s • V:e b:b a:at:t 3sg:s • Relation that maps characters to other characters.

16. Finite-State Transducers • Finite State Acceptors represent regular sets. • Finite State Transducers represent regular relations. • Relation • If A and B are two regular sets; relation R ⊆ A x B • Example: {(x,y) | x ∊ a*, y ∊ b*} • FSTs can be considered as • Translators (Hello:Ciao) • Parser/generators (Hello:HowmayI help you?) • As well as Kimmo-style morphological parsing

17. a:o b:m a:o a:o !:? q0 q1 q2 q3 q4 Finite State Transducers – formally speaking • FST is a 5-tuple consisting of • Q: set of states {q0,q1,q2,q3,q4} • : an alphabet of complex symbols, each an i:o pair s.t. i  I (an input alphabet) and o  O (an output alphabet) and  ⊆ I x O • q0: a start state • F: a set of final states in Q {q4} • (q,i:o): a transition function mapping Q x  to Q

18. More FST examples • FSTs are functions, • Compose (L1,L2) = {(x,y) | (x,z) ∊ L1 and (z,y) ∊ L2} • 3((q1,q2),(x,y)) = (q3,q4) if ∃z 1(q1,(x,z)) = q3 and 2(q2,(z,y)) = q4 • Inverse(L) = {(x,y) | (y,x) ∊ L} • FirstProjection(L) = {x | (x,y) ∊ L} • SecondProjection(L) = {y | (x,y) ∊ L} • Speech Recognition as FST composition • A: Acoustic  phones • Lex: phones  Words • Gram: Words  Words • Speech Recognition: A ● Lex ● Gram • Machine Translation as FST composition • LexTrans: SourceWords  TargetWords • LexReorder: UnOrderedTargetWords  OrderedTargetWords • Weighted FSTs allow for ranking of the generated outputs.

19. FST for a 2-level Lexicon c:c a:a t:t q3 q0 q1 q2 • E.g. • Nominal Morphology g:g e:e q4 q5 q6 q7 e:o s:s e:o +PL:^s# q4 reg-n +SG:# +N: irreg-n-sg q0 q2 q5 q7 +SG:# irreg-n-pl q6 +PL:#

20. Orthographic Rules • Define additional FSTs to implement spelling rules • Gemination: consonant doubling (beg begging), • Elision: ‘e’ deletion(make  making), • Epenthesis: ‘e’ insertion (fox  foxes), etc. • Y replacement: ‘y’ changes to ‘ie’ before ‘ed’ (trytries) • I spelling: ‘I’ changes to ‘y’ before a vowel (lielying) • Rewrite rule notation: a b / p _ d • Examples: Rule: a  b/a _ b • aaabaab  aabbabb (Sequential; there are other control strategies: parallel, feedback). • Spelling rules: e e / x ^ _ s (eg. fox^s  foxes) • Multiple spelling rules might apply for a given word. • Eg: spy^s  spies (yi, ee) • All spelling rules execute in parallel. • Rewrite rules can be compiled into FSTs.

21. FST-based Morphological Parsing • Remember FSTs can be composed. • Lexicon : List of stems • Morphotactics: Rules for adding morphemes • Orthographic Rules: Spelling Rules • Morphology = Lexicon ● Morphotactics ● Orhtographics • All morphological analyses generated for a given word, • …but in a context a preference ordering on the analysis is needed. • Weighted FSTs to the rescue.

22. Porter Stemmer • Lexicon free stemmer; heuristic rules for deriving the stem. • Rules to rewrite the suffix of a word. • “ational”  “ate” (eg. relational  relate) • “-ing”  e (eg. motoring  motor) • Details of the rules in Appendix B in the book. • Purported to improve recall in IR engines.

23. Role of Morphology in Machine Translation • Every MT system contains a bilingual lexicon • Bilingual lexicon: a table mapping the source language token to target language token(s). • Two options: • 1. Full-form lexicon • every word form of the source token is paired with the target token • large table if the vocabulary is large for morphologically rich languages • 2. Root-form lexicon • pairing of stems from the two languages • Reduces the size of the lexicon • requires morphological analysis for source language • bats  (bat, V, 3sg) (bat, N, pl) • morphological generation for target language • (bat, V, 3sg)  bats • Unknown words: words not covered in the bilingual lexicon • - with morphology, one can guess the syntactic function • Compounding of words: • English: simple juxtaposition (“car seat”), some times (“seaweed”) • German: fusion is more common • (“Dampfschiffahrtsgesellschaft”  steamship company

24. Readings • For this class • HS: pages 15-16, 81-85 • JM: chapter on Morphology • For next class • Machine Translation divergences (Dorr 94) • Example based Machine Translation (Somers 99)