1 / 36

Finite-State Methods

Finite-State Methods. c. a. e. Finite state acceptors (FSAs). Things you may know about FSAs: Equivalence to regexps Union, Kleene *, concat, intersect, complement, reversal Determinization, minimization Pumping, Myhill-Nerode. Defines the language a? c*

tress
Download Presentation

Finite-State Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Finite-State Methods 600.465 - Intro to NLP - J. Eisner

  2. c a e Finite state acceptors (FSAs) • Things you may know about FSAs: • Equivalence to regexps • Union, Kleene *, concat, intersect, complement, reversal • Determinization, minimization • Pumping, Myhill-Nerode Defines the language a? c* = {a, ac, acc, accc, …,, c, cc, ccc, …} 600.465 - Intro to NLP - J. Eisner

  3. n-gram models not good enough • Want to model grammaticality • A “training” sentence known to be grammatical: BOS mouse traps catch mouse traps EOS • Resulting trigram model has to overgeneralize: • allows sentences with 0 verbsBOS mouse traps EOS • allows sentences with 2 or more verbsBOS mouse traps catch mouse traps catch mouse traps catch mouse traps EOS • Can’t remember whether it’s in subject or object(i.e., whether it’s gotten to the verb yet) trigram model must allow these trigrams 600.465 - Intro to NLP - J. Eisner

  4. Noun Noun Noun Verb Noun preverbal states(still need a verb to reach final state) postverbal states(verbs no longerallowed) Finite-state models can “get it” • Want to model grammaticalityBOS mouse traps catch mouse traps EOS • Finite-state can capture the generalization here: Noun+ Verb Noun+ Allows arbitrarily long NPs (just keep looping around for another Noun modifier). Still, never forgets whether it’s preverbal or postverbal! (Unlike 50-gram model) 600.465 - Intro to NLP - J. Eisner

  5. How powerful are regexps / FSAs? • More powerful than n-gram models • The hidden state may “remember” arbitrary past context • With k states, can remember which of k “types” of context it’s in • Equivalent to HMMs • In both cases, you observe a sequence and it is “explained” by a hidden path of states. The FSA states are like HMM tags. • Appropriate for phonology and morphology Word = Syllable+ = (Onset Nucleus Coda?)+ = (C+ V+ C*)+ = ( (b|d|f|…)+ (a|e|i|o|u)+ (b|d|f|…)* )+ 600.465 - Intro to NLP - J. Eisner

  6. finite-state can handle this pattern (can you write the regexp?) but not this pattern,which requires a CFG How powerful are regexps / FSAs? • But less powerful than CFGs / pushdown automata • Can’t do recursive center-embedding • Hmm, humans have trouble processing those constructions too … • This is the rat that ate the malt. • This is the malt that the rat ate. • This is the cat that bit the rat that ate the malt. • This is the malt that the rat that the cat bit ate. • This is the dog that chased the cat that bit the rat that ate the malt. • This is the malt that [the rat that [the cat that [the dog chased] bit] ate]. 600.465 - Intro to NLP - J. Eisner

  7. Noun Noun S = S = NP Verb NP converting to FSA copies the NP twice  Noun Verb Noun duplicatedstructure duplicatedstructure Noun NP = Noun How powerful are regexps / FSAs? • But less powerful than CFGs / pushdown automata • More important: Less explanatory than CFGs • An CFG without recursive center-embedding can be converted into an equivalent FSA – but the FSA will usually be far larger • Because FSAs can’t reuse the same phrase type in different places more elegant – usingnonterminals like thisis equivalent to a CFG 600.465 - Intro to NLP - J. Eisner

  8. We’ve already used FSAs this way … CFG with regular expression on the right-hand side: X  (A | B) G H (P | Q) NP  (Det | ) Adj* N So each nonterminal has a finite-state automaton, giving a “recursive transition network (RTN)” A P X  G H B Q Adj Det NP  Automaton state replaces dotted rule (X  A G . H P) N Adj N 600.465 - Intro to NLP - J. Eisner 600.465 - Intro to NLP - J. Eisner 8

  9. We’ve already used FSAs once .. NP rules from the WSJ grammar become a single DFA ADJP NP  DT DFA NP ADJ ADJP  P ADJP NP ADJP ADJP JJ JJ NN NNS | ADJP DT NN | ADJP JJ NN | ADJP JJ NN NNS | ADJP JJ NNS | ADJP NN | ADJP NN NN | ADJP NN NNS | ADJP NNS | ADJP NPR | ADJP NPRS | DT | DT ADJP | DT ADJP , JJ NN | DT ADJP ADJP NN | DT ADJP JJ JJ NN | DT ADJP JJ NN | DT ADJP JJ NN NN etc. regular expression 600.465 - Intro to NLP - J. Eisner 600.465 - Intro to NLP - J. Eisner 9

  10. CFG / RTN bigram modelof words or tags(first-order Markov Model) Hidden Markov Model of words and tags together?? Verb Det Start Prep Adj Adj Noun Det Stop NP  N Adj N But where can we put our weights? 600.465 - Intro to NLP - J. Eisner

  11. slide courtesy of L. Karttunen (modified) Network Wordlist c l e a r clear clever ear ever fat father e v e compile f a t h Another useful FSA … FSM 17728 states, 37100 arcs 0.6 sec /usr/dict/words 25K words 206K chars 600.465 - Intro to NLP - J. Eisner

  12. slide courtesy of L. Karttunen (modified) Wordlist clear 0 clever 1 ear 2 ever 3 fat 4 father 5 Weights are useful here too! Network c/0 l/0 e/0 a/0 r/0 e/2 v/1 e/0 compile f/4 a/0 t/0 h/1 Computes a perfect hash! 600.465 - Intro to NLP - J. Eisner

  13. slide courtesy of L. Karttunen (modified) Wordlist c/0 l/0 e/0 a/0 r/0 clear 0 clever 1 ear 2 ever 3 fat 4 father 5 e/2 v/1 e/0 f/4 a/0 t/0 h/1 Example: Weighted acceptor Network c l e a r 2 6 1 2 1 2 e v e compile f a t h 2 1 2 2 • Compute number of paths from each state (Q: how?) • Successor states partition the path set • Use offsets of successor states as arc weights • Q: Would this work for an arbitrary numbering of the words? A: recursively, like DFS 600.465 - Intro to NLP - J. Eisner

  14. the problem of morphology (“word shape”) - an area of linguistics Example: Unweighted transducer VP [head=vouloir,...] V[head=vouloir,tense=Present,num=SG, person=P3] ... veut 600.465 - Intro to NLP - J. Eisner

  15. slide courtesy of L. Karttunen (modified) vouloir +Pres +Sing + P3 Finite-state transducer veut canonical form inflection codes v o u l o i r +Pres +Sing +P3 v e u t inflected form Example: Unweighted transducer VP [head=vouloir,...] V[head=vouloir,tense=Present,num=SG, person=P3] ... veut the relevant path 600.465 - Intro to NLP - J. Eisner

  16. slide courtesy of L. Karttunen vouloir +Pres +Sing + P3 Finite-state transducer veut canonical form inflection codes v o u l o i r +Pres +Sing +P3 v e u t inflected form Example: Unweighted transducer • Bidirectional: generation or analysis • Compact and fast • Xerox sells for about 20 languges including English, German, Dutch, French, Italian, Spanish, Portuguese, Finnish, Russian, Turkish, Japanese, ... • Research systems for many other languages, including Arabic, Malay the relevant path 600.465 - Intro to NLP - J. Eisner

  17. position in upper string 0 1 2 3 4 5 l:e a:e r:e a:e c:e 0 1 2 3 4 a:c l:c c:c a:c r:c e:c e:c e:c e:c e:c e:c l:e a:e r:e a:e c:e l:a c:a a:a r:a a:a e:a e:a e:a e:a e:a e:a l:e a:e r:e a:e c:e position in lower string l:c c:c a:c r:c a:c e:c e:c e:c e:c e:c e:c l:e a:e r:e a:e c:e l:a c:a a:a a:a r:a e:a e:a e:a e:a e:a e:a l:e a:e r:e a:e c:e Example: Weighted Transducer Edit distance:Cost of best path relatingthese two strings? 600.465 - Intro to NLP - J. Eisner

  18. Regular Relation (of strings) • Relation: like a function, but multiple outputs ok • Regular: finite-state • Transducer: automaton w/ outputs • b ? a ? • aaaaa ? a:e b:b {b} {} a:a a:c {ac, aca, acab, acabc} ?:c b:b ?:b • Invertible? • Closed under composition? ?:a b:e 600.465 - Intro to NLP - J. Eisner

  19. Regular Relation (of strings) • Can weight the arcs:  vs.  • b {b} a {} • aaaaa {ac, aca, acab, acabc} • How to find best outputs? • For aaaaa? • For all inputs at once? a:e b:b a:a a:c ?:c b:b ?:b ?:a b:e 600.465 - Intro to NLP - J. Eisner

  20. Acceptors (FSAs) Transducers (FSTs) c c:z a a:x Unweighted e e:y c:z/.7 c/.7 a:x/.5 a/.5 Weighted .3 .3 e:y/.5 e/.5 Function from strings to ... {false, true} strings numbers (string, num) pairs 600.465 - Intro to NLP - J. Eisner

  21. Sample functions Acceptors (FSAs) Transducers (FSTs) {false, true} strings Grammatical? Markup Correction Translation Unweighted numbers (string, num) pairs How grammatical? Better, how likely? Good markups Good corrections Good translations Weighted 600.465 - Intro to NLP - J. Eisner

  22. Terminology (acceptors) Regular language defines recognizes compiles into Regexp FSA implements accepts matches (or generates) matches String 600.465 - Intro to NLP - J. Eisner

  23. Terminology (transducers) Regular relation defines recognizes compiles into Regexp FST implements accepts matches (or, transducesone string of the pair intothe other) (or generates) matches ? String pair 600.465 - Intro to NLP - J. Eisner

  24. 3 views of a context-free rule (randsent) • generation (production): S  NP VP • parsing (comprehension): S  NP VP • verification (checking): S = NP VP (parse) v o u l o i r +Pres +Sing +P3 v e u t Perspectives on a Transducer • Remember these CFG perspectives: • Similarly, 3 views of a transducer: • Given 0 strings, generate a new string pair (by picking a path) • Given one string (upper or lower), transduce it to the other kind • Given two strings (upper & lower), decide whether to accept the pair FST just defines the regular relation (mathematical object: set of pairs). What’s “input” and “output” depends on what one asks about the relation.The 0, 1, or 2 given string(s) constrain which paths you can use. 600.465 - Intro to NLP - J. Eisner

  25. abcd abcd f g Functions ab?d 600.465 - Intro to NLP - J. Eisner

  26. Functions ab?d abcd Function composition: f  g [first f, then g – intuitive notation, but opposite of the traditional math notation] 600.465 - Intro to NLP - J. Eisner

  27. g 3 4 abgd 2 2 abed abed 8 6 abd abjd ... From Functions to Relations f ab?d abcd 600.465 - Intro to NLP - J. Eisner

  28. 4 2 8 From Functions to Relations 3 ab?d abgd 2 2 abed Relation composition: f  g 6 abd ... 600.465 - Intro to NLP - J. Eisner

  29. 3+4 2+2 2+8 From Functions to Relations ab?d abgd abed Relation composition: f  g abd ... 600.465 - Intro to NLP - J. Eisner

  30. 2+2 From Functions to Relations ab?d abed Pick min-cost or max-prob output Often in NLP, all of the functions or relations involved can be described as finite-state machines, and manipulated using standard algorithms. 600.465 - Intro to NLP - J. Eisner

  31. g 3 4 abgd 2 2 abed abed 8 6 abd abjd ... Inverting Relations f ab?d abcd 600.465 - Intro to NLP - J. Eisner

  32. Inverting Relations g-1 f-1 3 4 ab?d abcd abgd 2 2 abed abed 8 6 abd abjd ... 600.465 - Intro to NLP - J. Eisner

  33. 3+4 2+2 2+8 Inverting Relations ab?d abgd abed (f  g)-1 = g-1 f-1 abd ... 600.465 - Intro to NLP - J. Eisner

  34. slide courtesy of L. Karttunen (modified) c l e a r e v e f a t h Lexical Transducer (a single FST) Lexicon FSA composition Regular Expressions for Rules ComposedRule FSTs Compiler b i g +Adj +Comp g e b i g r one path Building a lexical transducer big | clear | clever | ear | fat | ... Regular Expression Lexicon 600.465 - Intro to NLP - J. Eisner

  35. slide courtesy of L. Karttunen (modified) c l e a r e v e f a t h Lexicon FSA Building a lexical transducer • Actually, the lexicon must contain elements likebig +Adj +Comp • So write it as a more complicated expression:(big | clear | clever | fat | ...) +Adj ( | +Comp | +Sup)  adjectives | (ear | father | ...) +Noun (+Sing | +Pl)  nouns | ...  ... • Q: Why do we need a lexicon at all? big | clear | clever | ear | fat | ... Regular Expression Lexicon 600.465 - Intro to NLP - J. Eisner

  36. slide courtesy of L. Karttunen (modified) être+IndP +SG + P1 suivre+IndP+SG+P1 suivre+IndP+SG+P2 suivre+Imp+SG + P2 Weighted version of transducer: Assigns a weight to each string pair “upper language” 4 payer+IndP+SG+P1 19 20 12 Weighted French Transducer 50 3 suis paie “lower language” paye 600.465 - Intro to NLP - J. Eisner

More Related