Stance-based Argument Mining – Modeling Implicit Argumentation Using Stance

1. Stance-based Argument Mining – Modeling Implicit Argumentation Using Stance Michael Wojatzki & Torsten Zesch

2. Argumentation in Social Media “I am against Atheism!” “I am a Christian” “I believe in hell” #JesusOrHell Michael Wojatzki & Torsten Zesch

3. Application-Driven Argument Mining • No evidence for religion • there is no god • there was no Christ • religious freedom • liberal beliefs • religious beliefs • life after death • belief in God • Christianity • Islam • conservative beliefs Atheism? Michael Wojatzki & Torsten Zesch

4. Application-Driven Argument Mining • No evidence for religion (100) • there is no god (20) • there was no Christ (80) • religious freedom (50) • liberal beliefs (50) (200) • religious beliefs (250) • life after death (50) • belief in God (200) • Christianity (150) • Islam (50) • conservative beliefs (50) (300) Atheism? Michael Wojatzki & Torsten Zesch

5. Argumentation • Argument := claim + optional supporting structures e.g. premises (Palau and Moens, 2009; Peldszus and Stede, 2013, Green et al. 2014) As the Bible says that infidels are going to hell, I am against atheism! Premise Claim Michael Wojatzki & Torsten Zesch

6. Argumentation in Social Media -Implicitness implicit Bible: infidels are going to hell! I am against atheism Claim Premise Michael Wojatzki & Torsten Zesch

7. Argumentation in Social Media - Noise & Implicitness implicit #JesusOrHell I am against atheism ? Claim Michael Wojatzki & Torsten Zesch

8. Stance-based Model ⊖ Atheism • Stance: • in favor or against a target(=topic/subject) • w.r.t. a given debate • Debate Stance: • stance towards the targetof the debate • often implicit, has to be inferred • Explicit Stance: • explicit stance-taking towards some target • aids inference of debate stance Christianity⨁ Michael Wojatzki & Torsten Zesch

9. Inspecting Stances ⊖ ⊖ Atheism Atheism ⊖ Atheism Christianity⨁ Existence of Hell ⨁ As the Bible says that infidels are going to hell, I am against atheism! Christianity⨁ Existence of Hell ⨁ implicit I am against atheism Bible: infidels are going to hell! Christianity⨁ Existence of Hell ⨁ implicit #JesusOrHell I am against atheism Michael Wojatzki & Torsten Zesch

10. Iceberg Metaphor Supernatural Power ⨁ ⊖ Atheism Explicitness ⊖ Atheism Debate Stance observable Supernatural Power ⨁ Explicit Stance Implicitness God will judge those infidels! unobservable Michael Wojatzki & Torsten Zesch

11. Evaluation of the Model • How reliable can we annotate it? • What does it tell us about the debate? • How well can we automate it? Explicit Stance Debate Stance Michael Wojatzki & Torsten Zesch

12. Data • SemEval task 6 2016 (Mohammad et al., 2016) • subset on Atheism (Train + Test-Set) • 733 tweets • classes: ⨁ vs. vs. NONE • Annotation (three annotators) • Debate Stance (Reannotation) • Explicit Stances ⊖ Michael Wojatzki & Torsten Zesch

13. Reannotation of Debate Stance • original annotation by turkers from USA • different cultural background • same questionnaire • remove Tweets if stance not inferable (18 tweets) • @bdutt @lalitkmodi @mohdasim1 ohh .so i think why r u seculr ... nice friend • you're doing the work of ending domination." (Bell Hooks) (2/2) #feminism #civilrights Michael Wojatzki & Torsten Zesch

14. Annotation of Explicit Targets • Our use-case: overview on argumentation • degree of abstraction from surface forms needed • define (less) explicit targets • choose granularity with respect to data Christianity⨁ I believe in Jesus my Savior Christ is my Lord Explicit Stance Debate Stance Michael Wojatzki & Torsten Zesch

15. Data-driven Selection • Candidates • 50 most frequent concepts (Nouns + Named Entities) • 25 concepts most strongly associated (Dice (Smadja et al., 1996)) with Atheism⨁ and Atheism • manually grouped and filtered • at this stage of our work, we can not evaluate whether the set is best possible choice ⊖ Michael Wojatzki & Torsten Zesch

16. Explicit Targets Supernatural Power Freethinking Secularism Christianity Conservatism Islam Religious Freedom USA No Evidence Same-Sex Marriage Life after Death Michael Wojatzki & Torsten Zesch

17. Explicit Targets – Annotation • annotate targets only if there is some evidence in the text • do not infer a target if there is no textual hint on it <Then they will go away to eternal punishment, but the righteous to eternal life.> Matthew 25:46 • Christianity ⨁ ⊖ • Islam Michael Wojatzki & Torsten Zesch

18. Inter-Annotator-Agreement complete model (joint decision): κ =0.63 Michael Wojatzki & Torsten Zesch

19. Stance Pattern Analysis ⊖ Atheism • What explicit stances appear together? • What is the relation between explicit and debate stances? Christianity⨁ Existence of Hell ⨁ Michael Wojatzki & Torsten Zesch

20. Patterns 1st Order Atheism ⨁ ⊖ Atheism Michael Wojatzki & Torsten Zesch

21. Patterns 2nd Order ⊖ Atheism Atheism ⨁ Michael Wojatzki & Torsten Zesch

22. Supervised Classification • Can we detect the scheme automatically? • explicit stances? • debate stances? • Experiments with state-of-the-art stance-detection system (Mohammad et al., 2016) • three-way classification (⨁, ,NONE) • SVM (linear Kernel)* ⊖ Michael Wojatzki & Torsten Zesch

23. Features – Explicit Stance Detection ⊖ • for each of target classify: ⨁ vs. vs. NONE • word n-grams (1,2,3) • character n-grams (2,3,4,5) ⊖ ⨁ vs. vs. NONE word n-grams character n-grams Bible: infidels are going to hell! Michael Wojatzki & Torsten Zesch

24. Explicit Stances • only top 2 explicit targets significant gains over baseline • other targets in <5% of the instances • not enough data • specialized features possible Michael Wojatzki & Torsten Zesch

25. Features ⊖ • for the debate stance classify: ⨁ vs. vs. NONE • n-grams • explicit stances ⊖ ⨁ vs. vs. NONE word n-grams character n-grams explicit stance Bible: infidels are going to hell! Michael Wojatzki & Torsten Zesch

26. Debate Stance • models for explicit stance similar to state-of-the-art • significant gain for oracle condition Michael Wojatzki & Torsten Zesch

27. Future Work • fully automated creation of explicit targets • apply more sophisticated machine learning • unlabeled data (distant supervision, word embeddings) (Mohammad et al. 2016) • sequential classification, deep learning • transfer explicit classifiers between domains Michael Wojatzki & Torsten Zesch

28. Stance-based Argument Mining • applicable to noisy and implicit argumentation • reliable to annotate • interesting insights into nature of debates • potential to boost automated stance detection Michael Wojatzki & Torsten Zesch

29. References Stab, C., & Gurevych, I. (2014). Annotating Argument Components and Relations in Persuasive Essays. In COLING (pp. 1501-1510). Eckle-Kohler, J., Kluge, R., & Gurevych, I. (2015). On the Role of Discourse Markers for Discriminating Claims and Premises in Argumentative Discourse. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Lisbon, Portugal, to appear. Peldszus, A., & Stede, M. (2013). From argument diagrams to argumentation mining in texts: A survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 7(1), 1-31. SaifM. Mohammad, Svetlana Kiritchenko, ParinazSobhani, Xiaodan Zhu, and Colin Cherry. (2016). Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’16, San Diego, California, June. Green, N., Ashley, K., Litman, D., Reed C., & Walker V. (2014) Proceedings of the First Workshop on Argumentation Mining, ACL Mohammad, S. M., Sobhani, P., & Kiritchenko, S. (2016). Stance and sentiment in tweets. arXiv preprint arXiv:1605.01655. Michael Wojatzki & Torsten Zesch

30. Backup Slides Michael Wojatzki & Torsten Zesch

31. SemEval Stance Questionnaire

32. Explicit Targets Michael Wojatzki & Torsten Zesch

33. Granularity of Targets • frequent vs. as explicit as possible ATHEISM every utterance • data-driven approach tires to find optimum Michael Wojatzki & Torsten Zesch

34. Annotating Explicit Stance • as we need textual evidence, look for key words • Matthew 25:46, Christ, American Values, … • incorporate semantics • Do you have the feeling that the author is explicitly addressing the issue vs. continuing his logic Life after Death ⨁ Then they will go away to eternal punishment ⊖ No Evidence for Religion Michael Wojatzki & Torsten Zesch