Language Modeling

1. Language Modeling

2. Roadmap (for next two classes) • Review LM evaluation metrics • Entropy • Perplexity • Smoothing • Good-Turing • Backoff and Interpolation • Absolute Discounting • Kneser-Ney

3. Language Model Evaluation Metrics

4. Applications

5. Entropy and perplexity • Entropy – measure information content, in bits • is message length with ideal code • Use if you want to measure in bits! • Cross entropy – measure ability of trained model to compactly represent test data • Average logprob of test data • Perplexity – measure average branching factor

6. Entropy and perplexity • Entropy – measure information content, in bits • is message length with ideal code • Use if you want to measure in bits! • Cross entropy – measure ability of trained model to compactly represent test data • Average logprob of test data • Perplexity – measure average branching factor

7. Entropy and perplexity • Entropy – measure information content, in bits • is message length with ideal code • Use if you want to measure in bits! • Cross entropy – measure ability of trained model to compactly represent test data • Average logprob of test data • Perplexity – measure average branching factor

8. Entropy and perplexity • Entropy – measure information content, in bits • is message length with ideal code • Use if you want to measure in bits! • Cross entropy – measure ability of trained model to compactly represent test data • Average logprob of test data • Perplexity – measure average branching factor

9. Language model perplexity • Recipe: • Train a language model on training data • Get negative logprobs of test data, compute average • Exponentiate! • Perplexity correlates rather well with: • Speech recognition error rates • MT quality metrics • LM Perplexities for word-based models are normally between say 50 and 1000 • Need to drop perplexity by a significant fraction (not absolute amount) to make a visible impact

10. Parameter estimation • What is it?

11. Parameter estimation • Model form is fixed (coin unigrams, word bigrams, …) • We have observations • H HH T T H T H H • Want to find the parameters • Maximum Likelihood Estimation – pick the parameters that assign the most probability to our training data • c(H) = 6; c(T) = 3 • P(H) = 6 / 9 = 2 / 3; P(T) = 3 / 9 = 1 / 3 • MLE picks parameters best for training data… • …but these don’t generalize well to test data – zeros!

12. Parameter estimation • Model form is fixed (coin unigrams, word bigrams, …) • We have observations • H HH T T H T H H • Want to find the parameters • Maximum Likelihood Estimation – pick the parameters that assign the most probability to our training data • c(H) = 6; c(T) = 3 • P(H) = 6 / 9 = 2 / 3; P(T) = 3 / 9 = 1 / 3 • MLE picks parameters best for training data… • …but these don’t generalize well to test data – zeros!

13. Parameter estimation • Model form is fixed (coin unigrams, word bigrams, …) • We have observations • H HH T T H T H H • Want to find the parameters • Maximum Likelihood Estimation – pick the parameters that assign the most probability to our training data • c(H) = 6; c(T) = 3 • P(H) = 6 / 9 = 2 / 3; P(T) = 3 / 9 = 1 / 3 • MLE picks parameters best for training data… • …but these don’t generalize well to test data – zeros!

14. Smoothing • Take mass from seen events, give to unseen events • Robin Hood for probability models • MLE at one end of the spectrum; uniform distribution the other • Need to pick a happy medium, and yet maintain a distribution

15. Smoothing techniques • Laplace • Good-Turing • Backoff • Mixtures • Interpolation • Kneser-Ney

16. Laplace • From MLE: • To Laplace:

17. Good-Turing Smoothing • New idea: Use counts of things you have seen to estimate those you haven’t

18. Good-Turing Josh Goodman Intuition • Imagine you are fishing • There are 8 species: carp, perch, whitefish, trout, salmon, eel, catfish, bass • You have caught • 10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish • How likely is it that the next fish caught is from a new species (one not seen in our previous catch)? Slide adapted from Josh Goodman, Dan Jurafsky

19. Good-Turing Josh Goodman Intuition • Imagine you are fishing • There are 8 species: carp, perch, whitefish, trout, salmon, eel, catfish, bass • You have caught • 10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish • How likely is it that the next fish caught is from a new species (one not seen in our previous catch)? • 3/18 • Assuming so, how likely is it that next species is trout? Slide adapted from Josh Goodman, Dan Jurafsky

20. Good-Turing Josh Goodman Intuition • Imagine you are fishing • There are 8 species: carp, perch, whitefish, trout, salmon, eel, catfish, bass • You have caught • 10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish • How likely is it that the next fish caught is from a new species (one not seen in our previous catch)? • 3/18 • Assuming so, how likely is it that next species is trout? • Must be less than 1/18 Slide adapted from Josh Goodman, Dan Jurafsky

21. Some more hypotheticals How likely is it to find a new fish in each of these places?

22. Good-Turing Smoothing • New idea: Use counts of things you have seen to estimate those you haven’t

23. Good-Turing Smoothing • New idea: Use counts of things you have seen to estimate those you haven’t • Good-Turing approach: Use frequency of singletons to re-estimate frequency of zero-count n-grams

24. Good-Turing Smoothing • New idea: Use counts of things you have seen to estimate those you haven’t • Good-Turing approach: Use frequency of singletons to re-estimate frequency of zero-count n-grams • Notation: Nc is the frequency of frequency c • Number of ngrams which appear c times • N0: # ngrams of count 0; N1: # of ngrams of count 1

25. Good-Turing Smoothing • Estimate probability of things which occur c times with the probability of things which occur c+1 times • Discounted counts: steal mass from seen cases to provide for the unseen: • MLE • GT

26. GT Fish Example

27. Enough about the fish…how does this relate to language? • Name some linguistic situations where the number of new words would differ

28. Enough about the fish…how does this relate to language? • Name some linguistic situations where the number of new words would differ • Different languages: • Chinese has almost no morphology • Turkish has a lot of morphology • Lots of new words in Turkish!

29. Enough about the fish…how does this relate to language? • Name some linguistic situations where the number of new words would differ • Different languages: • Chinese has almost no morphology • Turkish has a lot of morphology • Lots of new words in Turkish! • Different domains: • Airplane maintenance manuals: controlled vocabulary • Random web posts: uncontrolled vocab

30. Bigram Frequencies of Frequencies and GT Re-estimates

31. Good-Turing Smoothing • N-gram counts to conditional probability • Use c* from GT estimate

32. Additional Issues in Good-Turing • General approach: • Estimate of c* for Nc depends on N c+1 • What if Nc+1 = 0? • More zero count problems • Not uncommon: e.g. fish example, no 4s

33. Modifications • Simple Good-Turing • Compute Nc bins, then smooth Nc to replace zeroes • Fit linear regression in log space • log(Nc) = a +b log(c) • What about large c’s? • Should be reliable • Assume c*=c if c is large, e.g c > k (Katz: k =5) • Typically combined with other approaches

34. Backoff and Interpolation • Another really useful source of knowledge • If we are estimating: • trigram p(z|x,y) • but count(xyz) is zero • Use info from:

35. Backoff and Interpolation • Another really useful source of knowledge • If we are estimating: • trigram p(z|x,y) • but count(xyz) is zero • Use info from: • Bigram p(z|y) • Or even: • Unigram p(z)

36. Backoff and Interpolation • Another really useful source of knowledge • If we are estimating: • trigram p(z|x,y) • but count(xyz) is zero • Use info from: • Bigram p(z|y) • Or even: • Unigram p(z) • How to combine this trigram, bigram, unigram info in a valid fashion?

37. Backoffvs. Interpolation • Backoff: use trigram if you have it, otherwise bigram, otherwise unigram

38. Backoffvs. Interpolation • Backoff: use trigram if you have it, otherwise bigram, otherwise unigram • Interpolation: always mix all three

39. Backoff • Bigram distribution • But could be zero… • What if we fell back (or “backed off”) to a unigram distribution? • Also could be zero…

40. Backoff • What’s wrong with this distribution? • Doesn’t sum to one! • Need to steal mass…

41. Backoff

42. Mixtures • Given distributions and • Pick any number between and • is a distribution • (Laplace is a mixture!)

43. Interpolation • Simple interpolation • Or, pick interpolation value based on context • Intuition: Higher weight on more frequent n-grams

44. How to Set the Lambdas? • Use a held-out, or development, corpus • Choose lambdas which maximize the probability of some held-out data • I.e. fix the N-gram probabilities • Then search for lambda values • That when plugged into previous equation • Give largest probability for held-out set • Can use EM to do this search

45. Kneser-Ney Smoothing • Most commonly used modern smoothing technique • Intuition: improving backoff • I can’t see without my reading…… • Compare P(Francisco|reading) vs P(glasses|reading)

46. Kneser-Ney Smoothing • Most commonly used modern smoothing technique • Intuition: improving backoff • I can’t see without my reading…… • Compare P(Francisco|reading) vs P(glasses|reading) • P(Francisco|reading) backs off to P(Francisco)

47. Kneser-Ney Smoothing • Most commonly used modern smoothing technique • Intuition: improving backoff • I can’t see without my reading…… • Compare P(Francisco|reading) vs P(glasses|reading) • P(Francisco|reading) backs off to P(Francisco) • P(glasses|reading) > 0 • High unigram frequency of Francisco > P(glasses|reading)

48. Kneser-Ney Smoothing • Most commonly used modern smoothing technique • Intuition: improving backoff • I can’t see without my reading…… • Compare P(Francisco|reading) vs P(glasses|reading) • P(Francisco|reading) backs off to P(Francisco) • P(glasses|reading) > 0 • High unigram frequency of Francisco > P(glasses|reading) • However, Francisco appears in few contexts, glasses many

49. Kneser-Ney Smoothing • Most commonly used modern smoothing technique • Intuition: improving backoff • I can’t see without my reading…… • Compare P(Francisco|reading) vs P(glasses|reading) • P(Francisco|reading) backs off to P(Francisco) • P(glasses|reading) > 0 • High unigram frequency of Francisco > P(glasses|reading) • However, Francisco appears in few contexts, glasses many • Interpolate based on # of contexts • Words seen in more contexts, more likely to appear in others

50. Kneser-Ney Smoothing: bigrams • Modeling diversity of contexts • So