Classification-based Contextual Preferences

1. Classification-based Contextual Preferences Shachar Mirkin July 2011 TextInfer, Edinburgh Joint work with: Ido Dagan, Lili Kotlerman, Idan Szpektor

2. Context matching in inference • Motivation • Task definition

3. Motivation: Inference & ambiguity • H The USaccepts a large number of foreignersevery year • TThe USwelcomes hundreds of thousands of aliensyearly alien  foreigner welcome  accept If it’s any consolation, dear, our alien abduction insurance is finally going to pay off

4. Addressing ambiguity H The US accepts a large number of foreigners every year TThe US welcomes hundreds of thousands of aliens yearly • WSD • Direct sense matching • Dagan et al., 2006: Binary task • Avoiding intermediate representation of meanings • Suggested for synonyms • Lexical Substitution (McCarthy & Navigli, 2009) T1The US welcomes hundreds of thousands of foreigners yearly ? = 10123254 10123254 match?

5. Context matching • ContextMatching generalizes sense matching • Does aliens in T match the meaning of outer-space? • Does ‘children acquire English’ match X acquire Y  X learn Y? • Expected output: • Yes / No • Score / probability – quantifying the match degree • Both can be provided by classifiers

6. Name-based Text Categorization* * This isn’t the task, just the setting

7. Name-based Text Categorization (TC) • Unsupervised setting of TC • The category name is the only input: space, medicine, religion State of the art(Barak et al., 2009) - Vector-spaceIR approach: • Query: category name • Expanded with its entailing terms • A document must include a (direct or indirect) match to the category h t shuttle  space t r

8. A closer look at Context Matching • Context matching aspects • Directionality

9. Contextual matches • Contextual (mis)matches in Name-based TC: • t – h • H:(outer) space • T: The server ran out of disk space • r – h • r: room  space • Invalid for the category • t – r • r: star space is a valid rule for space • T: Elle Magazine accused of 'whitening' Bollywood star's skin r t h match?

10. Context matching directionality • Different roles for t, h & r • t should match the meaning of h and r • r should match the meaning of h r t h • H The US accepts a large number of foreignersevery year • TIf it’s any consolation, dear, our alien abduction insurance is finally going to pay off  ? • alien in T should match the meaning of foreigner in H • and not the other way around

11. r CP Contextual Preferences t h (Szpektor et al., 2008) • CP: the context matching framework • The 3 matches • Directionality • Prior work • Concrete context matching models in Szpektor et al.

12. The Contextual Preferences (CP) framework • Each inference object has contextual representation, CP • In inference operations, CPs should also be matched • Two contextual information types • Variables selectional preferences (CPv) • Topical / global (CPg) X acquire Y cpv(X) = {child}cpv(Y) = {English} cpg() = {learning, grammar} cpg(r), cpv(r) Pantel et al., 2007Pennacchiotti et al., 2007Downey et al., 2007 r Dagan et al., 2006 Connor and Roth, 2007 h t cpg(h), cpv(h) cpg(t), cpv(t) Patwardhan and Riloff, 2007 Harabagiu et al., 2003 (QA) Barak et al., 2009 (TC) • Prior work addressed specific aspects of context matching

13. Szpektor et al.’s implementation of CP cpg(r), cpv(r) Lin sim. of instantiations x score of comparing preferred NE r Lin Similarity of instantiations h t cpg(h), cpv(h) cpg(t), cpv(t) Cosine similarity of LSA vectors • Our goal: implementation of CP with a model which is: • Unified • Directional • Incorporates various context information types • Classification-based approach

14. A classification-based scheme for CP • Classifiers for context matching • Classifiers for all CP context matching aspects

15. Using classifiers to represent & assess context r • A classifier to identify valid contexts of h • Trained with typical valid & invalid contexts of h • Given a match t: • Ch is applied on the context of t • is it a valid context for the intended meaning of h? Millions of Americans followed the space shuttle Atlantis' final mission Elle Magazine accused of 'whitening' Bollywood star's skin t t h h: (outer) space .. Bollywood star's skin .. Ch Ch:space Ch:space(..Bollywood <t> ‘s skin..) Ch(t)

16. Classification-based scheme: addressing all CP matches • A classifier to identify valid contexts of the hypothesis h • Applied to t • Applied to the rule r • A classifier to identify valid contexts of the rule r • Applied to t • Classifiers in prior work addressed t-r match (Kauchak and Barzilay, 2006; Dagan et al., 2006; Connor and Roth, 2007) star space r Cr(t) Ch(r) h: (outer) space t h .. Bollywood star's skin .. Ch(t) Ch Cr

17. A self-supervised context model • A concrete model for the classification-based scheme

18. A self-supervised context model • Unsupervised setting, no labeled examples • Self-supervised: Training examples automatically obtained • By querying the TC training set • Challenge: Constructing queries that correctly retrieve valid or invalid contexts • Key principles of solution: • Add disambiguation information (only) when needed • Gradual process: • Most accurate (least ambiguous) queries first, less accurate follow

19. Self-supervised model: acquiring positive examples Acquiring positive examples (simplified): • Start with monosemous terms and • Gradually add polysemous ones • Use “context words” to resolve ambiguity Ch:space • “outer space” • (“outer space” OR space) • AND(infinite OR science OR “scientific discipline” OR . . . ) • A similar process for Cr

20. Self-supervised model: acquiring negative examples Acquiring negative examples: • Randomly (Kauchak and Barzilay, 2006) • Like positive examples, but using cohyponyms • basketball forh:hockey ; islamfor h:christianity • Semantically similar to positive • Help classifiers achieve better discrimination

21. Self-supervised model: applying the classifiers • Ch & Cr classify matches in the text → Ch(t), Cr(t) • The document provides the context for t • Ch also classifies each rule → Ch(r) • What is the context to classify?

22. Applying Ch to rules r: check:n  hockey • Can be interpreted as a domain-specific rule priors Sample k texts with the rule’s entailing term Apply Ch to each match Set Ch(r) to the ratio of positive classifications Ch(r) = 0.25

23. Experiments and results

24. Experimental Setting • Following IR approach of state of the art Name-based TC • Goal: improve TC performance by verifying matches • Integrating within TC: • Modify vector entries with 3 scheme outputs [See paper] • 2 TC datasets • Entailment rules: WordNet, Wikipedia (Shnarch et al., 2009) • SVM, linear kernel • Standard WSD features • Classifiers’ scores (and not binary decisions)

25. Baselines Barak et al., 2009: • Barakno-context • Cosine similarity between document & category vectors • Baseline for not using any context model • Barakfull • State of the art for Name-based TC • LSA as t-h context model

26. Results • Accuracy of the classification decisions • Recall: relative to the potential recall of the rule-set

27. Conclusions • Quick summary • Discussion

28. Summary • 3 contextual aspects need to be considered in inference • Prior work • addressed them partially • used symmetric models • provided a different method for each aspect • Classification-based scheme for context in inference • Complete, unified, directional • Outperforming state of the art for Name-based TC r t h

29. Future work and discussion Thank you! • Future Work • Apply to other applications • Address to more complex hypotheses • And unknown terms (for which we didn’t train classifiers in advance) • Scaling • Attractive when hypotheses are known in advance • Feasible when hypothesis are given online? • Possible directions • Instant methods for training classifiers • Training classifiers for entire rule sets in advance • Global classifiers • …