Word sense disambiguation 2 Instructor: Rada Mihalcea Note: Some of the material in this slide set was adapted from a

1. Word sense disambiguation (2) Instructor: Rada Mihalcea Note: Some of the material in this slide set was adapted from a tutorial given by Rada Mihalcea & Ted Pedersen at ACL 2005

2. What is Supervised Learning? Collect a set of examples that illustrate the various possible classifications or outcomes of an event. Identify patterns in the examples associated with each particular class of the event. Generalize those patterns into rules. Apply the rules to classify a new event.

3. Learn from these examples :�when do I go to the store?�

4. Learn from these examples :�when do I go to the store?�

5. Task Definition: Supervised WSD Supervised WSD: Class of methods that induces a classifier from manually sense-tagged text using machine learning techniques. Resources Sense Tagged Text Dictionary (implicit source of sense inventory) Syntactic Analysis (POS tagger, Chunker, Parser, �) Scope Typically one target word per context Part of speech of target word resolved Lends itself to �targeted word� formulation Reduces WSD to a classification problem where a target word is assigned the most appropriate sense from a given set of possibilities based on the context in which it occurs

6. Sense Tagged Text

7. Two Bags of Words (Co-occurrences in the �window of context�)

8. Simple Supervised Approach Given a sentence S containing �bank�: For each word Wi in S If Wi is in FINANCIAL_BANK_BAG then Sense_1 = Sense_1 + 1; If Wi is in RIVER_BANK_BAG then Sense_2 = Sense_2 + 1; If Sense_1 > Sense_2 then print �Financial� else if Sense_2 > Sense_1 then print �River� else print �Can�t Decide�;

9. Supervised Methodology Create a sample of training data where a given target word is manually annotated with a sense from a predetermined set of possibilities. One tagged word per instance/lexical sample disambiguation Select a set of features with which to represent context. co-occurrences, collocations, POS tags, verb-obj relations, etc... Convert sense-tagged training instances to feature vectors. Apply a machine learning algorithm to induce a classifier. Form � structure or relation among features Parameters � strength of feature interactions Convert a held out sample of test data into feature vectors. �correct� sense tags are known but not used Apply classifier to test instances to assign a sense tag.

10. From Text to Feature Vectors My/pronoun grandfather/noun used/verb to/prep fish/verb along/adv the/det banks/SHORE of/prep the/det Mississippi/noun River/noun. (S1) The/det bank/FINANCE issued/verb a/det check/noun for/prep the/det amount/noun of/prep interest/noun. (S2)

11. Supervised Learning Algorithms Once data is converted to feature vector form, any supervised learning algorithm can be used. Many have been applied to WSD with good results: Support Vector Machines Nearest Neighbor Classifiers Decision Trees Decision Lists Na�ve Bayesian Classifiers Perceptrons Neural Networks Graphical Models Log Linear Models

12. Na�ve Bayesian Classifier Na�ve Bayesian Classifier well known in Machine Learning community for good performance across a range of tasks (e.g., Domingos and Pazzani, 1997) �Word Sense Disambiguation is no exception Assumes conditional independence among features, given the sense of a word. The form of the model is assumed, but parameters are estimated from training instances When applied to WSD, features are often �a bag of words� that come from the training data Usually thousands of binary features that indicate if a word is present in the context of the target word (or not)

13. Bayesian Inference Given observed features, what is most likely sense? Estimate probability of observed features given sense Estimate unconditional probability of sense Unconditional probability of features is a normalizing term, doesn�t affect sense classification

14. Na�ve Bayesian Model

15. The Na�ve Bayesian Classifier Given 2,000 instances of �bank�, 1,500 for bank/1 (financial sense) and 500 for bank/2 (river sense) P(S=1) = 1,500/2000 = .75 P(S=2) = 500/2,000 = .25 Given �credit� occurs 200 times with bank/1 and 4 times with bank/2. P(F1=�credit�) = 204/2000 = .102 P(F1=�credit�|S=1) = 200/1,500 = .133 P(F1=�credit�|S=2) = 4/500 = .008 Given a test instance that has one feature �credit� P(S=1|F1=�credit�) = .133*.75/.102 = .978 P(S=2|F1=�credit�) = .008*.25/.102 = .020

16. Comparative Results (Leacock, et. al. 1993) compared Na�ve Bayes with a Neural Network and a Context Vector approach when disambiguating six senses of line� (Mooney, 1996) compared Na�ve Bayes with a Neural Network, Decision Tree/List Learners, Disjunctive and Conjunctive Normal Form learners, and a perceptron when disambiguating six senses of line� (Pedersen, 1998) compared Na�ve Bayes with Decision Tree, Rule Based Learner, Probabilistic Model, etc. when disambiguating line and 12 other words� �All found that Na�ve Bayesian Classifier performed as well as any of the other methods!

17. Decision Lists and Trees Very widely used in Machine Learning. Decision trees used very early for WSD research (e.g., Kelly and Stone, 1975; Black, 1988). Represent disambiguation problem as a series of questions (presence of feature) that reveal the sense of a word. List decides between two senses after one positive answer Tree allows for decision among multiple senses after a series of answers Uses a smaller, more refined set of features than �bag of words� and Na�ve Bayes. More descriptive and easier to interpret.

18. Decision List for WSD (Yarowsky, 1994) Identify collocational features from sense tagged data. Word immediately to the left or right of target : I have my bank/1 statement. The river bank/2 is muddy. Pair of words to immediate left or right of target : The world�s richest bank/1 is here in New York. The river bank/2 is muddy. Words found within k positions to left or right of target, where k is often 10-50 : My credit is just horrible because my bank/1 has made several mistakes with my account and the balance is very low.

19. Building the Decision List Sort order of collocation tests using log of conditional probabilities. Words most indicative of one sense (and not the other) will be ranked highly.

20. Computing DL score Given 2,000 instances of �bank�, 1,500 for bank/1 (financial sense) and 500 for bank/2 (river sense) P(S=1) = 1,500/2,000 = .75 P(S=2) = 500/2,000 = .25 Given �credit� occurs 200 times with bank/1 and 4 times with bank/2. P(F1=�credit�) = 204/2,000 = .102 P(F1=�credit�|S=1) = 200/1,500 = .133 P(F1=�credit�|S=2) = 4/500 = .008 From Bayes Rule� P(S=1|F1=�credit�) = .133*.75/.102 = .978 P(S=2|F1=�credit�) = .008*.25/.102 = .020 DL Score = abs (log (.978/.020)) = 3.89

21. Using the Decision List Sort DL-score, go through test instance looking for matching feature. First match reveals sense�

22. Using the Decision List

23. Learning a Decision Tree Identify the feature that most �cleanly� divides the training data into the known senses. �Cleanly� measured by information gain or gain ratio. Create subsets of training data according to feature values. Find another feature that most cleanly divides a subset of the training data. Continue until each subset of training data is �pure� or as clean as possible. Well known decision tree learning algorithms include ID3 and C4.5 (Quillian, 1986, 1993) In Senseval-1, a modified decision list (which supported some conditional branching) was most accurate for English Lexical Sample task (Yarowsky, 2000)

24. Supervised WSD with Individual Classifiers Many supervised Machine Learning algorithms have been applied to Word Sense Disambiguation, most work reasonably well. (Witten and Frank, 2000) is a great intro. to supervised learning. Features tend to differentiate among methods more than the learning algorithms. Good sets of features tend to include: Co-occurrences or keywords (global) Collocations (local) Bigrams (local and global) Part of speech (local) Predicate-argument relations Verb-object, subject-verb, Heads of Noun and Verb Phrases

25. Convergence of Results Accuracy of different systems applied to the same data tends to converge on a particular value, no one system shockingly better than another. Senseval-1, a number of systems in range of 74-78% accuracy for English Lexical Sample task. Senseval-2, a number of systems in range of 61-64% accuracy for English Lexical Sample task. Senseval-3, a number of systems in range of 70-73% accuracy for English Lexical Sample task� What to do next?

26. Ensembles of Classifiers Classifier error has two components (Bias and Variance) Some algorithms (e.g., decision trees) try and build a representation of the training data � Low Bias/High Variance Others (e.g., Na�ve Bayes) assume a parametric form and don�t represent the training data � High Bias/Low Variance Combining classifiers with different bias variance characteristics can lead to improved overall accuracy �Bagging� a decision tree can smooth out the effect of small variations in the training data (Breiman, 1996) Sample with replacement from the training data to learn multiple decision trees. Outliers in training data will tend to be obscured/eliminated.

27. Ensemble Considerations Must choose different learning algorithms with significantly different bias/variance characteristics. Na�ve Bayesian Classifier versus Decision Tree Must choose feature representations that yield significantly different (independent?) views of the training data. Lexical versus syntactic features Must choose how to combine classifiers. Simple Majority Voting Averaging of probabilities across multiple classifier output Maximum Entropy combination (e.g., Klein, et. al., 2002)

28. Ensemble Results (Pedersen, 2000) achieved state of art for interest and line data using ensemble of Na�ve Bayesian Classifiers. Many Na�ve Bayesian Classifiers trained on varying sized windows of context / bags of words. Classifiers combined by a weighted vote (Florian and Yarowsky, 2002) achieved state of the art for Senseval-1 and Senseval-2 data using combination of six classifiers. Rich set of collocational and syntactic features. Combined via linear combination of top three classifiers. Many Senseval-2 and Senseval-3 systems employed ensemble methods.

29. Task Definition: Minimally supervised WSD Supervised WSD = learning sense classifiers starting with annotated data Minimally supervised WSD = learning sense classifiers from annotated data, with minimal human supervision Examples Automatically bootstrap a corpus starting with a few human annotated examples Use monosemous relatives / dictionary definitions to automatically construct sense tagged data Rely on Web-users + active learning for corpus annotation

30. Bootstrapping WSD Classifiers Build sense classifiers with little training data Expand applicability of supervised WSD Bootstrapping approaches Co-training Self-training Yarowsky algorithm Long term goal � all words WSD Co-training / self-training are known to improve over basic classifiers Explore their applicability to the problem of WSDLong term goal � all words WSD Co-training / self-training are known to improve over basic classifiers Explore their applicability to the problem of WSD

31. Bootstrapping Recipe Ingredients (Some) labeled data (Large amounts of) unlabeled data (One or more) basic classifiers Output Classifier that improves over the basic classifiers

33. Co-training / Self-training 1. Create a pool of examples U' choose P random examples from U 2. Loop for I iterations Train Ci on L and label U' Select G most confident examples and add to L maintain distribution in L Refill U' with examples from U keep U' at constant size P

34. (Blum and Mitchell 1998) Two classifiers independent views [independence condition can be relaxed] Co-training in Natural Language Learning Statistical parsing (Sarkar 2001) Co-reference resolution (Ng and Cardie 2003) Part of speech tagging (Clark, Curran and Osborne 2003) ... Co-training

35. Self-training (Nigam and Ghani 2000) One single classifier Retrain on its own output Self-training for Natural Language Learning Part of speech tagging (Clark, Curran and Osborne 2003) Co-reference resolution (Ng and Cardie 2003) several classifiers through bagging