Complementarity of Lexical and Simple Syntactic Features: The SyntaLex Approach to S ENSEVAL -3

1. Complementarity of Lexical and Simple Syntactic Features:The SyntaLex Approach to SENSEVAL-3 Saif Mohammad Ted Pedersen University of Toronto, Toronto University of Minnesota, Duluth http://www.cs.toronto.edu/~smm http://www.d.umn.edu/~tpederse

2. Supervised Learning: Decision Trees • Decision Trees used as classifiers. • One classifier for each task (word). • Assign sense by asking a series of questions. • Questions correspond to features of the instance. • Weka’s C4.5 algorithm employed.

3. Decision Tree Feature 1 ? 1 0 Feature 2 ? Feature 4? 0 1 0 1 Feature 4 ? SENSE 3 Feature 2 ? SENSE 1 0 1 0 1 SENSE 1 SENSE 4 SENSE 3 Feature 3 ? 0 1 SENSE 2 SENSE 3

4. Why Decision Trees? • Context is a rich source of discrete features. • The learned model likely meaningful. • May provide insight into the interaction of features. • Drawbacks – training data fragmentation. • Pedersen[2001]*: Choosing the right features is of greater significance than the learning algorithm itself. * “A Decision Tree of Bigrams is an Accurate Predictor of Word Sense” Pedersen, T., In the Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL-01), June 2-7, 2001, Pittsburgh, PA.

5. Features • Represent context with lexical and simple syntactic features. • Lexical Features • Bigrams of words • Syntactic Features • Parts of Speech

6. Lexical Features: Bigrams • Bigrams • Two-word sequences in text. The interest rate is low Bigrams: the interest, interest rate, rate is, is low • Ngram Statistics Package*: Identify statistically significant bigrams. * “The Design, Implementation and Use of the Ngram Statistics Package”, Banerjee, S. and Pedersen, T., Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, 2003, February, Mexico City.

7. Part-of-Speech Features • A word used in different senses is likely to have different sets of PoS tags around it. Why did Jack turn/VB against/IN his/PRP$ team/NN Why did Jack turn/VB left/NN at/IN the/DT crossing • Individual word PoS: P-2, P-1, P0, P1, P2. • P1 = JJ implies that the word to the right of the target word is an adjective.

8. Part-of-Speech Tagging • Brill Tagger – widely used tool. • Certain target words mis-tagged to completely different parts of speech. • Pre-tagging is the act of manually assigning tags to selected words in a text prior to tagging. • Brill tagger does not guarantee pre-tagging. • BrillPatch* guarantees pre-tagging. * “Guaranteed Pre-Tagging for the Brill Tagger”, Mohammad, S. and Pedersen, T., In Proceedings of Fourth International Conference of Intelligent Systems and Text Processing, February 2003, Mexico.

9. SyntaLex Entries • SyntaLex-1 • Parts of Speech of a narrow window of words around the target word. P-1 = <Tag>orP0 = <Tag>or P1 = <Tag>. • SyntaLex-2 • Parts of Speech: broader window P-2 = <Tag>orP-1 = <Tag>orP0 = <Tag>or P1 = <Tag>orP2 = <Tag>. <Tag> represents a Part of Speech.

10. SyntaLex Entries (continued) • SyntaLex-3 • Individual classifiers based on bigrams and parts of speech (narrow context) learned. • Given a test instance, both classifiers assign probabilities to every sense. • Sense with the highest sum chosen. • SyntaLex-4 • Single unified decision tree. • Either bigrams or part of speech features (narrow context) may exist at a particular node.

11. Fine-Grained Test Results

12. Fine-Grained Test Results

13. Coarse-Grained Test Results

14. Observations • SyntaLex-1 (narrow context) just as good as SyntaLex-2 (broader context). • Low training data per task ratio. • Weak indicators (P-2 and P2) overwhelmed. • Results* on Senseval-1, Senseval-2, line, hard, serve and interest data back the hypothesis. * "Combining Lexical and Syntactic Features for Supervised Word Sense Disambiguation", Mohammad, S. and Pedersen, T., In Proceedings of the Eighth Conference on Natural Language Learning at HLT-NAACL, May 2004, Boston.

15. Observations (continued) • Pedersen’s entry (Duluth-ELSS*) • Only lexical features • Fine grained accuracy: 61.7%. • Comparable to SyntaLex-2 (61.8%). • Use of both kinds of features (SyntaLex-3 & 4) has provided some improvement. * "The Duluth Lexical Sample Systems in Senseval-3", Pedersen, T., In Proceedings of the Eighth Conference on Natural Language Learning at HLT-NAACL, May 2004, Boston.

16. Thoughts… • Lexical and syntactic features perform comparably. • Do they get the same instances right ? • How much are the individual feature sets redundant. • Are there instances correctly disambiguated by one feature set and not by the other ? • How much are the individual feature sets complementary. • Can we do better than the results obtained by a simple ensemble? • Is the effort to combine lexical and syntactic features justified?

17. Measures • Baseline Ensemble:accuracy of a hypothetical ensemble which predicts the sense correctly only if both individual feature sets do so. • Quantifies redundancy amongst feature sets. • SENSEVAL-3, with SyntaLex: 52.9% (fine grained). • Optimal Ensemble: accuracy of a hypothetical ensemble which predicts the sense correctly if either of the individual feature sets do so. • Difference with individual accuracies quantifies complementarity. • SENSEVAL-3, with SyntaLex: 72.1% (fine grained).

18. Conclusions • Significant amount of complementarity across the studied lexical and syntactic features. • Combination of the two justified. • We show that bigram and part of speech features can achieve state-of-the-art results. • How best to capitalize on the complementarity still an open issue. • Part of speech of words immediately adjacent to target word more suitable than that of larger contexts.

19. Software • SyntaLex • A system to do WSD using lexical and syntactic features. • Weka’s decision tree learning algorithm is utilized. • posSenseval • Assigns Part-of-Speech to any data in Senseval-2 data format. • Outputs tagged data in Senseval-2 data format. • Employs the Brill Tagger. • BrillPatch • Patch to the Brill Tagger to guarantee pre-tagging. http://www.d.umn.edu/~tpederse/code.html Thank You.