1 / 38

CPSC 503 Computational Linguistics

This lecture discusses the use of the noisy channel model for spelling correction, including the application of Bayes' rule and the estimation of error probabilities. It also introduces the minimum edit distance algorithm for computing the number and type of errors between two strings.

feckstein
Download Presentation

CPSC 503 Computational Linguistics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CPSC 503Computational Linguistics Lecture 4 Giuseppe Carenini CPSC503 Winter 2008

  2. Knowledge-Formalisms Map(including probabilistic formalisms) State Machines (and prob. versions) (Finite State Automata,Finite State Transducers, Markov Models) Morphology Logical formalisms (First-Order Logics) Syntax Rule systems (and prob. versions) (e.g., (Prob.) Context-Free Grammars) Semantics Pragmatics Discourse and Dialogue AI planners CPSC503 Winter 2008

  3. Today Sep 17 • Dealing with spelling errors • Noisy channel model • Bayes rule applied to Noisy channel model (single and multiple spelling errors) • Start n-grams models: Language Models (LM) CPSC503 Winter 2008

  4. Background knowledge • Morphological analysis • P(x) (prob. distribution) • joint p(x,y) • conditional p(x|y) • Bayes rule • Chain rule CPSC503 Winter 2008

  5. funn -> funny, fun, ... Find the most likely correct word • trust funn • a lot of funn …in this context Is it an impossible (or very unlikely) word in this context? • .. a wild big. Spelling: the problem(s) Correction Detection Non-word isolated Non-word context Real-word isolated ?! Find the most likely substitution word in this context Real-word context CPSC503 Winter 2008

  6. Spelling: Data • 05% -3% - 38% • 80% of misspelled words, single error • Insertion (toy -> tony) • deletion (tuna -> tua) • substitution (tone -> tony) • transposition (length -> legnth) • Types of errors • Typographic (more common, user knows the correct spelling… the -> rhe) • Cognitive (user doesn’t know…… piece -> peace) CPSC503 Winter 2008

  7. noisy signal signal signal Noisy Channel • An influential metaphor in language processing is the noisy channel model • Special case of Bayesian classification CPSC503 Winter 2008

  8. Bayes and the Noisy Channel: Spelling Non-word isolated Goal: Find the most likely word given some observed (misspelled) word CPSC503 Winter 2008

  9. Problem • P(w|O) is hard/impossible to get (why?) P(wine|winw)= CPSC503 Winter 2008

  10. likelihood prior Solution • Apply Bayes Rule • Simplify CPSC503 Winter 2008

  11. Always verify… Estimate of prior P(w) (Easy) smoothing CPSC503 Winter 2008

  12. Estimate of P(O|w) is feasible(Kernighan et. al ’90) • For one-error misspelling: • Estimate the probability of each possible error type • e.g., insert aafter c, substitute fwith h • P(O|w) equal to the probability of the error that generated O from w • e.g., P( cbat| cat) = P(insert b after c) CPSC503 Winter 2008

  13. Estimate P(error type) Large corpus compute confusion matrices (e.g substitution: sub[x,y]) and count matrix #Times b was incorrectly used for a a b c ……… ……… a Count(a)= # of a in corpus ……… b 5 ……… c 8 15 d 8 … ……… ……… ……… CPSC503 Winter 2008

  14. Corpus: Example … On 16 January, he sais [sub[i,y] 3] that because of astronaut safety tha [del[a,t] 4] would be no more space shuttle missions to miantain [tran[a,i] 2] and upgrade the orbiting telescope…….. CPSC503 Winter 2008

  15. (2) For all the wicompute: word prior Probability of the error generating O from w1 Final Method single error (1) Given O, collect all the wi that could have generated O by one error. E.g., O=acress=> w1 = actress (t deletion), w2 = across (sub o with e), … … (3) Sort and display top-n to user CPSC503 Winter 2008

  16. Example: O = acress 1988 AP newswire corpus 44 million words _ _ _ _ _ …stellar and versatile acress whose… CPSC503 Winter 2008

  17. Evaluation “correct” system 0 1 2 other CPSC503 Winter 2008

  18. Corpora: issues to remember • Zero counts in the corpus: Just because an event didn’t happen in the corpus doesn’t mean it won’t happen e.g., cress has not really zero probability • Getting a corpus that matches the actual use. • e.g., Kids don’t misspell the same way that adults do CPSC503 Winter 2008

  19. Multiple Spelling Errors • (BEFORE) Given O collect all the wi that could have generated O by one error……. • (NOW) Given O collect all the wi that could have generated O by 1..k errors General Solution: How to compute # and type of errors “between” O and wi? CPSC503 Winter 2008

  20. Minimum Edit Distance • Def. Minimum number of edit operations (insertion, deletion and substitution) needed to transform one string into another. gumbo gumb gum gam w delete o delete b substitute u by a O CPSC503 Winter 2008

  21. Minimum Edit Distance Algorithm • Dynamic programming (very common technique in NLP) • High level description: • Fills in a matrix of partial comparisons • Value of a cell computed as “simple” function of surrounding cells • Output: not only number of edit operations but also sequence of operations CPSC503 Winter 2008

  22. target j i source i-1 , j i-1, j-1 update z x sub or equal del i , j-1 ? y ins Minimum Edit Distance Algorithm Details del-cost =1 sub-cost=2 ins-cost=1 ed[i,j] = min distance between first i chars of the source and first j chars of the target MIN(z+1,y+1, x + (2 or 0)) CPSC503 Winter 2008

  23. target j i source i-1 , j i-1, j-1 update z x sub or equal del i , j-1 ? y ins Minimum Edit Distance Algorithm Details del-cost =1 sub-cost=2 ins-cost=1 ed[i,j] = min distance between first i chars of the source and first j chars of the target MIN(z+1,y+1, x + (2 or 0)) CPSC503 Winter 2008

  24. Min edit distance and alignment See demo CPSC503 Winter 2008

  25. (2) For all the wi compute: word prior Probability of the errors generating O from wi Final Method multiple errors (1) Given O, for each wi compute: mei=min-edit distance(wi,O) if mei<k save corresponding edit operations in EdOpi (3) Sort and display top-n to user CPSC503 Winter 2008

  26. funn -> funny, funnel... Find the most likely correct word • trust funn • a lot of funn …in this context Is it an impossible (or very unlikely) word in this context? • .. a wild big. Spelling: the problem(s) Correction Detection Non-word isolated Non-word context Real-word isolated ?! Find the most likely sub word in this context Real-word context CPSC503 Winter 2008

  27. Real Word Spelling Errors • Collect a set of common sets of confusions: C={C1 ..Cn} e.g.,{(Their/they’re/there), (To/too/two), (Weather/whether), (lave, have)..} • Whenever c’  Ciis encountered • Compute the probability of the sentence in which it appears • Substitute all cCi(c ≠ c’) and compute the probability of the resulting sentence • Choose the higher one CPSC503 Winter 2008

  28. Want to play with Spelling Correction: minimal noisy channel model implementation • (Python) http://www.norvig.com/spell-correct.html • By the way Peter Norvig is Director of Research at Google Inc. CPSC503 Winter 2008

  29. Key Transition • Up to this point we’ve mostly been discussing words in isolation • Now we’re switching to sequences of words • And we’re going to worry about assigning probabilities to sequences of words CPSC503 Winter 2008

  30. Knowledge-Formalisms Map(including probabilistic formalisms) State Machines (and prob. versions) (Finite State Automata,Finite State Transducers, Markov Models) Morphology Logical formalisms (First-Order Logics) Syntax Rule systems (and prob. versions) (e.g., (Prob.) Context-Free Grammars) Semantics Pragmatics Discourse and Dialogue AI planners CPSC503 Winter 2008

  31. Only Spelling? • Assign a probability to a sentence • Part-of-speech tagging • Word-sense disambiguation • Probabilistic Parsing • Predict the next word • Speech recognition • Hand-writing recognition • Augmentative communication for the disabled AB Impossible to estimate  CPSC503 Winter 2008

  32. Chain Rule: Decompose: apply chain rule Applied to a word sequence from position 1 to n: CPSC503 Winter 2008

  33. Example • Sequence “The big red dog barks” • P(The big red dog barks)= P(The) * P(big|the) * P(red|the big)* P(dog|the big red)* P(barks|the big red dog) Note - P(The) is better expressed as: P(The|<Beginning of sentence>) written as P(The|<S>) CPSC503 Winter 2008

  34. Not a satisfying solution  Even for small n (e.g., 6) we would need a far too large corpus to estimate: Markov Assumption: the entire prefix history isn’t necessary. unigram bigram trigram CPSC503 Winter 2008

  35. Prob of a sentence: N-Grams unigram bigram trigram CPSC503 Winter 2008

  36. Bigram<s>The big red dog barks • P(The big red dog barks)= • P(The|<S>) * • P(big|the) * • P(red|big)* • P(dog|red)* • P(barks|dog) Trigram? CPSC503 Winter 2008

  37. Estimates for N-Grams bigram ..in general CPSC503 Winter 2008

  38. Next Time • Finish N-Grams (Chp. 4) • Model Evaluation (sec. 4.4) • No smoothing 4.5-4.7 • Start Hidden Markov-Model Assignment 1 is due CPSC503 Winter 2008

More Related