Natural Language Processing COMPSCI 423/723

1. Natural Language ProcessingCOMPSCI 423/723 Rohit Kate

2. Classification for NLP:Naïve Bayes ModelMaximum Entropy Model • Some of these slides have been adapted from Raymond Mooney’s slides from his NLP and Machine Learning courses at UT Austin. • Referenes: • - Sections 6.6 & 6.7 from Jurafsy & Martin book • Naïve Bayes portions from Word Sense Disambiguation chapters in Jurafsy & Martin and Manning & Schutze books • Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression by Tom Mitchell http://www.cs.cmu.edu/~tom/NewChapters.html • A Maximum Entropy approach to Natural Language Processing • by Adam L. Berger, Stephen A. Della Pietra and  Vincent J. Della Pietra • Computational Linguistics, Vol. 22, No. 1. (1996), pp. 39-71.

3. Naïve Bayes Model

4. Classification in NLP • Several NLP problems can be formulated as classification problems, a few examples: • Information Extraction • Given an entity, is it a person name or not? • Given two protein names, does the sentence say they interact or not? • Word Sense Disambiguation • I am out of money. I am going to the bank. • Document Classification • Given a document, which category does it belong to? • Sentiment Analysis • Given a passage (e.g. product or movie review), is it saying positive things or negative things? • Textual Entailment • Given two sentences, can the second sentence be inferred from the first?

5. Classification • Usually the classification output variable is denoted by Y and the input variables by Xs Y: {river bank, money bank, verb bank} X1: Previous word X2: Next word X3: Part-of-speech of previous word X4: Part-of-speech of next word • Xs are usually called features in NLP • Coming up with good feature sets for NLP problems is a skill: feature engineering • Requires linguistic insights • Grasp of the theory behind the classification method

6. Probabilistic Classification • Often it is useful to know the probabilities of different output values and not just the best output value • To have confidence in the output (0.9-0.1 vs 0.6-0.4) • These probabilities may be useful for the next stage of NLP processing • Conditional probability: P(Y|X1,X2,..,Xn)

7. Probabilistic Classification • If the joint probability distribution P(Y,X1,X2,..,Xn) is given then the conditional probability distribution can be easily estimated

8. Estimating Conditional Probabilities Similarly estimate P(Y|X1,X2) for the remaining values.

9. Estimating Joint Probability Distributions Not Easy :-( • Assuming Y and all Xi are binary, we need 2n+1 - 1 entries (parameters) to specify the joint probability distribution • This is impossible to accurately estimate from a reasonably-sized training set • Note that P(Y|X1,X2,..,Xn) requires fewer entries (2n-1), why? But they are still too many for even small size of n

10. Estimating Joint Probability Distributions • Simplification assumptions are made about the joint probability distribution to reduce the number of parameters to estimate • Let the random variables be nodes of a graph, there are two major types of simplifications, they are represented as • Directed probabilistic graphical models • Simplest: Naïve Bayes model • More complex: Hidden Markov Model (HMM) • Undirected probabilistic graphical models • Simplest: Maximum entropy model • More complex: Conditional Random Field (CRFs)

11. Directed Graphical Models • Simplification assumption: Some random variables are conditionally independent of others given values for some other random variables

12. Conditional Independence • Two random variables A and B are conditionally independent given C if P(AПB|C) = P(A|C)P(B|C) Rain and Thunder are not independent (given there was Rain, it increases the probability that there was Thunder). But given that there was Lightning (or no Lighting) they are independent. P(Rain^Thunder|Lightning) = P(Rain|Lightning)P(Thunder|Lightning)

13. Directed Graphical Models • Also known as Bayesian networks • Simplification assumption: Some random variables are conditionally independent of others given values for some other random variables • Simplest directed graphical model: Naïve Bayes • Naïve Bayes assumption: The features are conditionally independent given the category

14. Naïve Bayes Assumption • Features are conditionally independent given the category • How do we estimate P(Y|X1,X2,..,Xn) from this? • Recall the Bayes’ theorem: Lets us calculate P(B|A) in terms of P(A|B)

15. Bayes’ Theorem • Simple proof from definition of conditional probability: (Def. cond. prob.) (Def. cond. prob.) QED:

16. Naïve Bayes assumption Naïve Bayes Model From Bayes’ Theorem Computing marginals and definition of conditional probability

17. Naïve Bayes Model • Only need to estimate P(Y) and P(Xi|Y) for all i, that with the naïve Bayes assumption specifies the entire joint probability distribution • Assuming all Y and Xis are binary, only 2n+1 parameters instead of 2n+1-1 parameters: a dramatic reduction Lightning Y P(Y) Rain Thunder X1 P(X1|Y) X2 P(X2|Y) X3 ……. P(X3|Y) Xn P(Xn|Y) Directed graphical model representation

18. Naïve Bayes Example P(Label|Size,Color,Shape) Test Instance: <medium ,red, circle>

19. Naïve Bayes Example Test Instance: <medium ,red, circle> P(positive |medium,red,circle) = P(positive)*P(medium | positive)*P(red | positive)*P(circle | positive) / P(medium,red,cirlce) 0.5 * 0.1 * 0.9 * 0.9 = 0.0405 / P(medium,red,circle) = 0.0405 / 0.0495 = 0.8181 P(negative |medium,red,circle) = P(negative)*P(medium | negative)*P(red | negative)*P(circle | negative) / P(medium,red,cirlce) 0.5 * 0.2 * 0.3 * 0.3 = 0.009 / P(medium,red,circle) = 0.009 / 0.0495 = 0.1818

20. Estimating Probabilities • Normally, probabilities are estimated based on observed frequencies in the training data. • If D contains nkexamples in category yk, and nijk of these nk examples have the jth value for feature Xi, xij, then: • However, estimating such probabilities from small training sets is error-prone. • If due only to chance, a rare feature, Xi, is always false in the training data, yk :P(Xi=true | Y=yk) = 0. • If Xi=true then occurs in a test example, X, the result is that yk: P(X | Y=yk) = 0 and yk: P(Y=yk | X) = 0

21. Test Instance X: <medium, red, circle> P(positive | X) = 0.5 * 0.0 * 1.0 * 1.0 / P(X) = 0 P(negative | X) = 0.5 * 0.0 * 0.5 * 0.5 / P(X) = 0 Probability Estimation Example

22. Smoothing • To account for estimation from small samples, probability estimates are adjusted or smoothed. • Laplace smoothing using an m-estimate assumes that each feature is given a prior probability, p, that is assumed to have been previously observed in a “virtual” sample of size m. • For binary features, p is simply assumed to be 0.5.

23. Laplace Smothing Example • Assume training set contains 10 positive examples: • 4: small • 0: medium • 6: large • Estimate parameters as follows (if m=1, p=1/3) • P(small | positive) = (4 + 1/3) / (10 + 1) = 0.394 • P(medium | positive) = (0 + 1/3) / (10 + 1) = 0.03 • P(large | positive) = (6 + 1/3) / (10 + 1) = 0.576 • P(small or medium or large | positive) = 1.0

24. Naïve Bayes Model is a Generative Model • Models the joint probability distribution P(Y,X1,X2,..,Xn) using P(Y) and P(Xi|Y) • An assumed generative process: First generate Y according to P(Y) then generate X1,X2,..,Xn independently according to P(X1|Y), P(X2|Y), .., P(Xn|Y) respectively

25. Size Color Shape Naïve Bayes Generative Model neg pos pos neg pos neg pos Category lg circ circ red red med blue tri sm blue sqr sm tri med lg grn tri circ circ grn red med grn red med circ tri circ lg blue sqr circ lg lg sm red sm blue sqr red lg med circ grn sm sm tri sqr blue red Size Color Shape Negative Positive

26. ?? ?? Size Color Shape Naïve Bayes Inference Problem lg red circ neg pos pos neg pos neg pos Category lg circ circ red red med blue tri sm blue sqr sm tri med lg grn tri circ circ grn red med grn red med circ tri circ lg blue sqr circ lg lg sm red sm blue sqr red lg med circ grn sm sm tri sqr blue red Size Color Shape Negative Positive

27. Some Comments on Naïve Bayes Model • Tends to work well despite strong (or naïve) assumption of conditional independence • Experiments show it to be quite competitive with other classification methods on standard UCI datasets • Although it does not produce accurate probability estimates when its independence assumptions are violated, it may still pick the correct maximum-probability class in many cases

28. Maximum Entropy Model

29. Maximum Entropy Models • Very popular in NLP • Several ways to look at them: • Exponential or log-linear classifiers or multinomial logical regression • Assume a parametric form for conditional distribution • Maximize entropy of the joint distribution given the constraints • Discriminative models instead of generative (directly estimates P(Y|X1,..,Xn) instead of via P(Y,X1,..,Xn))

30. Linear Regression • Classification: Predict a discrete values • Regression: Predict a real value • Linear Regression: Predict a real value using a linear combination of inputs Y = W0 + W1*X1 + W2*X2 + … + Wn*Xn Ws are the weights associated with the features Xs Example: price = 16550 - 4900*(# vague adjectives)

31. Estimating Weights in Linear Regression • Find the Ws that minimize the sum-squared error for the given M training examples • Statistical packages are available that solve this fast

32. Logistic Regression • But we are interested in probabilistic classification, that is in predicting P(Y|X1,..,Xn) • Can we modify linear regression to do that? • Nothing constrains it to be between [0,1] which is required for a legal probability • Predict odds (assume Y is binary) instead of the probability

33. Logistic Regression • But LHS lies between 0 and infinity, RHS could be between -infinity to infinity • Take log of LHS (known as logit function) Logistic function

34. Logistic Regression as a Log-Linear Model Logistic regression is basically a linear model, which is demonstrated by taking logs

35. Logistic Regression Training Weights are set during training to maximize the conditional data likelihood : where D is the set of training examples and Yd and Xiddenote, respectively, the values of Y and Xi for example d. • Equivalently viewed as maximizing the conditional log likelihood (CLL)

36. Logistic Regression Training Use standard gradient descent to find the parameters (weights) that optimize the CLL objective function Many other more advanced training methods are available Conjugate gradient Generalized Iterative Scaling (GIS) Improved Iterative Scaling (IIS) Limited-memory quasi-Newton (L-BFGS) Packages are available that do these

37. Preventing Overfitting in Logistic Regression To prevent overfitting, one can use regularization (smoothing) by penalizing large weights by changing the training objective: Where λ is a constant that determines the amount of smoothing • This can be shown to be equivalent to assuming a Guassian prior for W with zero mean and a variance related to 1/λ.

38. Generative vs. Discriminative Models Generative models and are not directly designed to maximize the performance of classification. They model the complete joint distribution P(Y,X1,...Xn). But a generative model can also be used to perform any other inference task, e.g. P(X1 | X2, …Xn, Y) “Jack of all trades, master of none.” Discriminative models are specifically designed and trained to maximize performance of classification. They only model the conditional distribution P(Y | X1, …Xn). By focusing on modeling the conditional distribution, they generally perform better on classification than generative models when given a reasonable amount of training data. Master of one trade: Classification P(Y|X1,.. Xn) 38

39. Multinomial Logistic Regression (Maximum Entropy or MaxEnt) • So far Y was binary, a generalization if Y takes multiple values (classes) • Make weights dependent on the class c: Wci instead of Wi Normalization term (Z) so that probabilities sum to 1

40. Multinomial Logistic Regression (Maximum Entropy or MaxEnt) • Usually features take binary values in NLP • Introduce indicator functions (0 or 1 output) that depend on the input and output class • Call X as input, features are fi(c,x)

41. A Small MaxEnt Example • Word Sense Disambiguation: Y: {river bank, money bank, verb bank} X: Entire Sentence Features: f1(river bank,X) = 1 if “river” is in the sentence, 0 otherwise f2(river bank,X) = 1 if “water” is in the sentence, 0 otherwise f3(money bank,X) = 1 if “money” is in the sentence, 0 otherwise f4(money bank,X) = 1 if “deposit” is in the sentence, 0 otherwise f5(verb bank,X) = 1 if previous word was “to”, 0 otherwise • Obtain examples of feature values and Y from annotated training data • Compute weights Wci to maximize the conditional log-likelihood of the training data • For a test example, predict Y using MaxEnt equation

42. Why is it Called Maximum Entropy Model? • Entropy of a random variable Y: • The more uniform distribution, the higher is the entropy • It can be shown that standard training for logistic regression gives the distribution with maximum entropy that is empirically consistent with the training data

43. Undirected Graphical Model • Also called Markov Network, Random Field • Undirected graph over a set of random variables, where an edge represents a dependency • The Markov blanket of a node, X, in a Markov Net is the set of its neighbors in the graph (nodes that have an edge connecting to X) • Every node in a Markov Net is conditionally independent of every other node given its Markov blanket • Simplest Markov Network: MaxEnt model

44. Relation with Naïve Bayes Y Naïve Bayes … Xn X2 X1 Generative Discriminative Conditional Y Logistic Regression … Xn X2 X1

45. Simplification Assumption for MaxEnt • The probability P(Y|X1..Xn) can be factored as: • Note there is no product term that has two or more Xis

46. Naïve Bayes and MaxEnt • Naïve Bayes can be extended to work with continuous inputs X (like logistic regression) • Both make the conditional independence assumption • MaxEnt is not rigidly tied with it because it tries to maximize the conditional likelihood of the data even when the data disobeys the assumption • It has been observed that with scarce training data Naïve Bayes performs better and with sufficient data MaxEnt performs better

47. Classification in General • Several other classifiers are also available: perceptron, neural networks, support vector machines, k-nearest neighbors, decision trees… • Naïve Bayes and MaxEnt are based on probabilities • Can’t handle combination of features as features • If right features are engineered they work very well • Are widely used for tasks other than NLP tasks • All this was for one label classification (there was only one Y), extensions to handle multi-label classifications, e.g. sequence labeling with HMMs or CRFs

48. HW 2 • Write Naïve Bayes (P(Y|f1,f2,f3,f4,f5)) and MaxEnt (P(Y|X)) equations for the example shown on slide #41.

49. References for Next Class • Chapter 5 (part-of-speech tagging) of Jurafsky & Martin book; Chapter 10 of Manning and Schutze book • An Introduction to Conditional Random Fields for Relational Learning By Charles Sutton and Andrew McCallum, Book chapter in Introduction to Statistical Relational Learning. Edited by Lise Getoor and Ben Taskar. MIT Press. 2006