Finite-State Methods

Finite-State Methods 600.465 - Intro to NLP - J. Eisner

c a e Finite state acceptors (FSAs) • Things you may know about FSAs: • Equivalence to regexps • Union, Kleene *, concat, intersect, complement, reversal • Determinization, minimization • Pumping, Myhill-Nerode Defines the language a? c* = {a, ac, acc, accc, …,, c, cc, ccc, …} 600.465 - Intro to NLP - J. Eisner

n-gram models not good enough • Want to model grammaticality • A “training” sentence known to be grammatical: BOS mouse traps catch mouse traps EOS • Resulting trigram model has to overgeneralize: • allows sentences with 0 verbsBOS mouse traps EOS • allows sentences with 2 or more verbsBOS mouse traps catch mouse traps catch mouse traps catch mouse traps EOS • Can’t remember whether it’s in subject or object(i.e., whether it’s gotten to the verb yet) trigram model must allow these trigrams 600.465 - Intro to NLP - J. Eisner

Noun Noun Noun Verb Noun preverbal states(still need a verb to reach final state) postverbal states(verbs no longerallowed) Finite-state models can “get it” • Want to model grammaticalityBOS mouse traps catch mouse traps EOS • Finite-state can capture the generalization here: Noun+ Verb Noun+ Allows arbitrarily long NPs (just keep looping around for another Noun modifier). Still, never forgets whether it’s preverbal or postverbal! (Unlike 50-gram model) 600.465 - Intro to NLP - J. Eisner

How powerful are regexps / FSAs? • More powerful than n-gram models • The hidden state may “remember” arbitrary past context • With k states, can remember which of k “types” of context it’s in • Equivalent to HMMs • In both cases, you observe a sequence and it is “explained” by a hidden path of states. The FSA states are like HMM tags. • Appropriate for phonology and morphology Word = Syllable+ = (Onset Nucleus Coda?)+ = (C+ V+ C*)+ = ( (b|d|f|…)+ (a|e|i|o|u)+ (b|d|f|…)* )+ 600.465 - Intro to NLP - J. Eisner

finite-state can handle this pattern (can you write the regexp?) but not this pattern,which requires a CFG How powerful are regexps / FSAs? • But less powerful than CFGs / pushdown automata • Can’t do recursive center-embedding • Hmm, humans have trouble processing those constructions too … • This is the rat that ate the malt. • This is the malt that the rat ate. • This is the cat that bit the rat that ate the malt. • This is the malt that the rat that the cat bit ate. • This is the dog that chased the cat that bit the rat that ate the malt. • This is the malt that [the rat that [the cat that [the dog chased] bit] ate]. 600.465 - Intro to NLP - J. Eisner

Noun Noun S = S = NP Verb NP converting to FSA copies the NP twice  Noun Verb Noun duplicatedstructure duplicatedstructure Noun NP = Noun How powerful are regexps / FSAs? • But less powerful than CFGs / pushdown automata • More important: Less explanatory than CFGs • An CFG without recursive center-embedding can be converted into an equivalent FSA – but the FSA will usually be far larger • Because FSAs can’t reuse the same phrase type in different places more elegant – usingnonterminals like thisis equivalent to a CFG 600.465 - Intro to NLP - J. Eisner

We’ve already used FSAs this way … CFG with regular expression on the right-hand side: X  (A | B) G H (P | Q) NP  (Det | ) Adj* N So each nonterminal has a finite-state automaton, giving a “recursive transition network (RTN)” A P X  G H B Q Adj Det NP  Automaton state replaces dotted rule (X  A G . H P) N Adj N 600.465 - Intro to NLP - J. Eisner 600.465 - Intro to NLP - J. Eisner 8

CFG / RTN bigram modelof words or tags(first-order Markov Model) Hidden Markov Model of words and tags together?? Verb Det Start Prep Adj Adj Noun Det Stop NP  N Adj N But where can we put our weights? 600.465 - Intro to NLP - J. Eisner

slide courtesy of L. Karttunen (modified) Network Wordlist c l e a r clear clever ear ever fat father e v e compile f a t h Another useful FSA … FSM 17728 states, 37100 arcs 0.6 sec /usr/dict/words 25K words 206K chars 600.465 - Intro to NLP - J. Eisner

slide courtesy of L. Karttunen (modified) Wordlist clear 0 clever 1 ear 2 ever 3 fat 4 father 5 Weights are useful here too! Network c/0 l/0 e/0 a/0 r/0 e/2 v/1 e/0 compile f/4 a/0 t/0 h/1 Computes a perfect hash! Sum the weights along a word’s accepting path. 600.465 - Intro to NLP - J. Eisner

slide courtesy of L. Karttunen (modified) Wordlist c/0 l/0 e/0 a/0 r/0 clear 0 clever 1 ear 2 ever 3 fat 4 father 5 e/2 v/1 e/0 f/4 a/0 t/0 h/1 Example: Weighted acceptor Network c l e a r 2 6 1 2 1 2 e v e compile f a t h 2 1 2 2 • Compute number of paths from each state (Q: how?) • Successor states partition the path set • Use offsets of successor states as arc weights • Q: Would this work for an arbitrary numbering of the words? A: recursively, like DFS 600.465 - Intro to NLP - J. Eisner

the problem of morphology (“word shape”) - an area of linguistics Example: Unweighted transducer VP [head=vouloir,...] V[head=vouloir,tense=Present,num=SG, person=P3] ... veut 600.465 - Intro to NLP - J. Eisner

slide courtesy of L. Karttunen (modified) vouloir +Pres +Sing + P3 Finite-state transducer veut canonical form inflection codes v o u l o i r +Pres +Sing +P3 v e u t inflected form Example: Unweighted transducer VP [head=vouloir,...] V[head=vouloir,tense=Present,num=SG, person=P3] ... veut the relevant path 600.465 - Intro to NLP - J. Eisner

slide courtesy of L. Karttunen vouloir +Pres +Sing + P3 Finite-state transducer veut canonical form inflection codes v o u l o i r +Pres +Sing +P3 v e u t inflected form Example: Unweighted transducer • Bidirectional: generation or analysis • Compact and fast • Xerox sells for about 20 languages including English, German, Dutch, French, Italian, Spanish, Portuguese, Finnish, Russian, Turkish, Japanese, ... • Research systems for many other languages, including Arabic, Malay the relevant path 600.465 - Intro to NLP - J. Eisner

What is a function? • Formally, a set of <input, output> pairswhere each input ∈ domain, output ∈ co-domain. • Moreover, ∀x ∈ domain, ∃ exactly one y such that <x,y> ∈ the function. • square: int int= { <0,0>, <1,1>, <-1,1>, <2,4>, <-2,4>, <3,9>, <-3,9>, ... } • domain(square) = {0, 1, -1, 2, -2, 3, -3, …} • range(square) = {0, 1, 4, 9, ...} domain co-domain 600.465 - Intro to NLP - J. Eisner

What is a relation? • square: int int= { <0,0>, <1,1>, <-1,1>, <2,4>, <-2,4>, <3,9>, <-3,9>, ... } • inverse(square): int  int= { <0,0>, <1,1>, <1,-1>, <4,2>, <4,-2>, <9,3>, <9,-3>, ... } • Is inverse(square) a function? • 0↦ {0} 9↦ {3,-3} 7 ↦ {} -1 ↦ {} • No - we need a more general notion! • A relation is any set of<input, output> pairs 600.465 - Intro to NLP - J. Eisner

Regular Relation (of strings) • Relation: like a function, but multiple outputs ok • Regular: finite-state • Transducer: automaton w/ outputs • b ? a ? • aaaaa ? a:e b:b {b} {} a:a a:c {ac, aca, acab, acabc} ?:c b:b ?:b • Invertible? • Closed under composition? ?:a b:e 600.465 - Intro to NLP - J. Eisner

Regular Relation (of strings) • Can weight the arcs:  vs.  • b {b} a {} • aaaaa {ac, aca, acab, acabc} • How to find best outputs? • For aaaaa? • For all inputs at once? a:e b:b a:a a:c ?:c b:b ?:b ?:a b:e 600.465 - Intro to NLP - J. Eisner

Acceptors (FSAs) Transducers (FSTs) c c:z a a:x Unweighted e e:y c:z/.7 c/.7 a:x/.5 a/.5 Weighted .3 .3 e:y/.5 e/.5 Function from strings to ... {false, true} strings numbers (string, num) pairs 600.465 - Intro to NLP - J. Eisner

Sample functions Acceptors (FSAs) Transducers (FSTs) {false, true} strings Grammatical? Markup Correction Translation Unweighted numbers (string, num) pairs Good markups Good corrections Good translations How grammatical? Better, how probable? Weighted 600.465 - Intro to NLP - J. Eisner

Terminology (acceptors) Regular language defines recognizes compiles into Regexp FSA implements accepts matches (or generates) matches String 600.465 - Intro to NLP - J. Eisner

Terminology (transducers) Regular relation defines recognizes compiles into Regexp FST implements accepts matches (or, transducesone string of the pair intothe other) (or generates) matches ? String pair 600.465 - Intro to NLP - J. Eisner

3 views of a context-free rule (randsent) • generation (production): S  NP VP • parsing (comprehension): S  NP VP • verification (checking): S = NP VP (parse) v o u l o i r +Pres +Sing +P3 v e u t Perspectives on a Transducer • Remember these CFG perspectives: • Similarly, 3 views of a transducer: • Given 0 strings, generate a new string pair (by picking a path) • Given one string (upper or lower), transduce it to the other kind • Given two strings (upper & lower), decide whether to accept the pair FST just defines the regular relation (mathematical object: set of pairs). What’s “input” and “output” depends on what one asks about the relation.The 0, 1, or 2 given string(s) constrain which paths you can use. 600.465 - Intro to NLP - J. Eisner

abcd abcd f g Functions ab?d 600.465 - Intro to NLP - J. Eisner

Functions ab?d abcd Function composition: f  g [first f, then g – intuitive notation, but opposite of the traditional math notation] Like the Unix pipe: cat x | f | g > y Example: Pass the input through a sequence of ciphers 600.465 - Intro to NLP - J. Eisner

g 3 4 abcd 2 2 abed abed 8 6 abd abjd ... From Functions to Relations f ab?d abcd 600.465 - Intro to NLP - J. Eisner

4 2 8 From Functions to Relations 3 abcd ab?d 2 2 abed Relation composition: f  g 6 abd ... 600.465 - Intro to NLP - J. Eisner

3+4 2+2 2+8 From Functions to Relations abcd ab?d abed Relation composition: f  g abd ... 600.465 - Intro to NLP - J. Eisner

2+2 From Functions to Relations ab?d abed Pick min-cost or max-prob output Often in NLP, all of the functions or relations involved can be described as finite-state machines, and manipulated using standard algorithms. 600.465 - Intro to NLP - J. Eisner

g 3 4 abcd 2 2 abed abed 8 6 abd abjd ... Inverting Relations f ab?d abcd 600.465 - Intro to NLP - J. Eisner

Inverting Relations g-1 f-1 3 4 abcd ab?d abcd 2 2 abed abed 8 6 abd abjd ... 600.465 - Intro to NLP - J. Eisner

3+4 2+2 2+8 Inverting Relations abcd ab?d abed (f  g)-1 = g-1 f-1 abd ... 600.465 - Intro to NLP - J. Eisner

slide courtesy of L. Karttunen (modified) c l e a r e v e f a t h Lexical Transducer (a single FST) Lexicon FSA composition Regular Expressions for Rules ComposedRule FSTs Compiler b i g +Adj +Comp g e b i g r one path Building a lexical transducer big | clear | clever | ear | fat | ... Regular Expression Lexicon 600.465 - Intro to NLP - J. Eisner

slide courtesy of L. Karttunen (modified) être+IndP +SG + P1 suivre+IndP+SG+P1 suivre+IndP+SG+P2 suivre+Imp+SG + P2 Weighted version of transducer: Assigns a weight to each string pair “upper language” 4 payer+IndP+SG+P1 19 20 12 Weighted French Transducer 50 3 suis paie “lower language” paye 600.465 - Intro to NLP - J. Eisner

Finite-State Methods