1 / 60

Maximum Entropy Model

Maximum Entropy Model. LING 572 Fei Xia 02/07-02/09/06. History. The concept of Maximum Entropy can be traced back along multiple threads to Biblical times. Introduced to NLP area by Berger et. Al. (1996). Used in many NLP tasks: MT, Tagging, Parsing, PP attachment, LM, …. Outline.

lita
Download Presentation

Maximum Entropy Model

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Maximum Entropy Model LING 572 Fei Xia 02/07-02/09/06

  2. History • The concept of Maximum Entropy can be traced back along multiple threads to Biblical times. • Introduced to NLP area by Berger et. Al. (1996). • Used in many NLP tasks: MT, Tagging, Parsing, PP attachment, LM, …

  3. Outline • Modeling: Intuition, basic concepts, … • Parameter training • Feature selection • Case study

  4. Reference papers • (Ratnaparkhi, 1997) • (Ratnaparkhi, 1996) • (Berger et. al., 1996) • (Klein and Manning, 2003)  Different notations.

  5. Modeling

  6. The basic idea • Goal: estimate p • Choose p with maximum entropy (or “uncertainty”) subject to the constraints (or “evidence”).

  7. Setting • From training data, collect (a, b) pairs: • a: thing to be predicted (e.g., a class in a classification problem) • b: the context • Ex: POS tagging: • a=NN • b=the words in a window and previous two tags • Learn the prob of each (a, b): p(a, b)

  8. Features in POS tagging(Ratnaparkhi, 1996) context (a.k.a. history) allowable classes

  9. Maximum Entropy • Why maximum entropy? • Maximize entropy = Minimize commitment • Model all that is known and assume nothing about what is unknown. • Model all that is known: satisfy a set of constraints that must hold • Assume nothing about what is unknown: choose the most “uniform” distribution  choose the one with maximum entropy

  10. Ex1: Coin-flip example(Klein & Manning 2003) • Toss a coin: p(H)=p1, p(T)=p2. • Constraint: p1 + p2 = 1 • Question: what’s your estimation of p=(p1, p2)? • Answer: choose the p that maximizes H(p) H p1 p1=0.3

  11. Coin-flip example (cont) H p1 + p2 = 1 p2 p1 p1+p2=1.0, p1=0.3

  12. Ex2: An MT example(Berger et. al., 1996) Possible translation for the word “in” is: Constraint: Intuitive answer:

  13. An MT example (cont) Constraints: Intuitive answer:

  14. An MT example (cont) Constraints: Intuitive answer: ??

  15. Ex3: POS tagging(Klein and Manning, 2003)

  16. Ex3 (cont)

  17. Ex4: overlapping features(Klein and Manning, 2003)

  18. Modeling the problem • Objective function: H(p) • Goal: Among all the distributions that satisfy the constraints, choose the one, p*, that maximizes H(p). • Question: How to represent constraints?

  19. Features • Feature (a.k.a. feature function, Indicator function) is a binary-valued function on events: • A: the set of possible classes (e.g., tags in POS tagging) • B: space of contexts (e.g., neighboring words/ tags in POS tagging) • Ex:

  20. Some notations Finite training sample of events: Observed probability of x in S: The model p’s probability of x: The jth feature: Observed expectation of : (empirical count of ) Model expectation of :

  21. Constraints • Model’s feature expectation = observed feature expectation • How to calculate ?

  22. Training data  observed events

  23. Restating the problem The task: find p* s.t. where Objective function: -H(p) Constraints: Add a feature

  24. Questions • Is P empty? • Does p* exist? • Is p* unique? • What is the form of p*? • How to find p*?

  25. Theorem: if then Furthermore, p* is unique. What is the form of p*?(Ratnaparkhi, 1997)

  26. Using Lagrangian multipliers Minimize A(p):

  27. Two equivalent forms

  28. Relation to Maximum Likelihood The log-likelihood of the empirical distribution as predicted by a model q is defined as Theorem: if then Furthermore, p* is unique.

  29. Summary (so far) Goal: find p* in P, which maximizes H(p). It can be proved that when p* exists it is unique. The model p* in P with maximum entropy is the model in Q that maximizes the likelihood of the training sample

  30. Summary (cont) • Adding constraints (features): (Klein and Manning, 2003) • Lower maximum entropy • Raise maximum likelihood of data • Bring the distribution further from uniform • Bring the distribution closer to data

  31. Parameter estimation

  32. Algorithms • Generalized Iterative Scaling (GIS): (Darroch and Ratcliff, 1972) • Improved Iterative Scaling (IIS): (Della Pietra et al., 1995)

  33. GIS: setup Requirements for running GIS: • Obey form of model and constraints: • An additional constraint: Let Add a new feature fk+1:

  34. GIS algorithm • Compute dj, j=1, …, k+1 • Initialize (any values, e.g., 0) • Repeat until converge • For each j • Compute • Update where

  35. Approximation for calculating feature expectation

  36. Properties of GIS • L(p(n+1)) >= L(p(n)) • The sequence is guaranteed to converge to p*. • The converge can be very slow. • The running time of each iteration is O(NPA): • N: the training set size • P: the number of classes • A: the average number of features that are active for a given event (a, b).

  37. IIS algorithm • Compute dj, j=1, …, k+1 and • Initialize (any values, e.g., 0) • Repeat until converge • For each j • Let be the solution to • Update

  38. Calculating If Then GIS is the same as IIS Else must be calcuated numerically.

  39. Feature selection

  40. Feature selection • Throw in many features and let the machine select the weights • Manually specify feature templates • Problem: too many features • An alternative: greedy algorithm • Start with an empty set S • Add a feature at each iteration

  41. Notation With the feature set S: After adding a feature: The gain in the log-likelihood of the training data:

  42. Feature selection algorithm(Berger et al., 1996) • Start with S being empty; thus ps is uniform. • Repeat until the gain is small enough • For each candidate feature f • Computer the model using IIS • Calculate the log-likelihood gain • Choose the feature with maximal gain, and add it to S  Problem: too expensive

  43. Approximating gains(Berger et. al., 1996) • Instead of recalculating all the weights, calculate only the weight of the new feature.

  44. Training a MaxEnt Model Scenario #1: • Define features templates • Create the feature set • Determine the optimum feature weights via GIS or IIS Scenario #2: • Define feature templates • Create candidate feature set S • At every iteration, choose the feature from S (with max gain) and determine its weight (or choose top-n features and their weights).

  45. Case study

  46. POS tagging(Ratnaparkhi, 1996) • Notation variation: • fj(a, b): a: class, b: context • fj(hi, ti): h: history for ith word, t: tag for ith word • History: • Training data: • Treat it as a list of (hi, ti) pairs. • How many pairs are there?

  47. Using a MaxEnt Model • Modeling: • Training: • Define features templates • Create the feature set • Determine the optimum feature weights via GIS or IIS • Decoding:

  48. Modeling

  49. Training step 1: define feature templates History hi Tag ti

  50. Step 2: Create feature set • Collect all the features from the training data • Throw away features that appear less than 10 times

More Related