1 / 42

Annotating Topics of Opinions

Annotating Topics of Opinions. Veselin Stoyanov Claire Cardie. Talk Overview. Fine-grained sentiment analysis Definitions Examples Opinion topic annotation Definitions Issues Approach and Corpus IA agreement. Background. Sentiment Analysis:

Thomas
Download Presentation

Annotating Topics of Opinions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Annotating Topics of Opinions Veselin Stoyanov Claire Cardie

  2. Talk Overview • Fine-grained sentiment analysis • Definitions • Examples • Opinion topic annotation • Definitions • Issues • Approach and Corpus • IA agreement LREC 2008, Marakech, Morocco

  3. Background • Sentiment Analysis: Extraction and representation of attitudes, evaluations, opinions, and sentiment in text. • Fine-grained Sentiment Analysis: At the level of individual expressions of opinions. LREC 2008, Marakech, Morocco

  4. Coarse-grained Sentiment classification Useful in the product review domain Fine-grained Individual expressions of opinions Multiple opinions per document (even sentence) The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty. Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given. Review 1 Positive [SThe Australian press] has launched a bitter attack on [TItaly] after seeing their beloved[TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty]. Lippi is preparing his side for the upcoming clash with Ukraine. [SHe] hailed 10-man [TItaly]'s determination to beat Australia and reiterated that the [Tpenalty] was rightly given. Review 2 Negative Fine-grained vs. Coarse-grained Sentiment Analysis LREC 2008, Marakech, Morocco

  5. Fine-grained opinions: Example The Australian press has launched a bitter attack on Italy. LREC 2008, Marakech, Morocco

  6. Fine-grained opinions: Example The Australian press has launched a bitter attack on Italy. • Opinion trigger (opinion words) • Source (opinion holder) • Polarity – positive/negative • Strength • Topic (target) • Definitions differ, but five main components: LREC 2008, Marakech, Morocco

  7. Fine-grained opinions: Example The Australian press has launched a bitter attack on Italy. • Opinion trigger (opinion words) • Source (opinion holder) • Polarity – positive/negative • Strength • Topic (target) • Definitions differ, but five main components: launched a bitter attack LREC 2008, Marakech, Morocco

  8. Fine-grained opinions: Example [SThe Australian press] has launched a bitter attack on Italy. • Opinion trigger (opinion words) • Source (opinion holder) • Polarity – positive/negative • Strength • Topic (target) • Definitions differ, but five main components: launched a bitter attack The Australian press LREC 2008, Marakech, Morocco

  9. Fine-grained opinions: Example [SThe Australian press] has launched a bitter attack on Italy. • Opinion trigger (opinion words) • Source (opinion holder) • Polarity – positive/negative • Strength • Topic (target) • Definitions differ, but five main components: launched a bitter attack The Australian press negative LREC 2008, Marakech, Morocco

  10. Fine-grained opinions: Example [SThe Australian press] has launched a bitter attack on Italy. • Opinion trigger (opinion words) • Source (opinion holder) • Polarity – positive/negative • Strength • Topic (target) • Definitions differ, but five main components: launched a bitter attack The Australian press negative high LREC 2008, Marakech, Morocco

  11. Fine-grained opinions: Example [SThe Australian press] has launched a bitter attack on [TItaly] • Opinion trigger (opinion words) • Source (opinion holder) • Polarity – positive/negative • Strength • Topic (target) • Definitions differ, but five main components: launched a bitter attack The Australian press negative high Italy LREC 2008, Marakech, Morocco

  12. Fine-grained opinions • Five components • Source (opinion holder) • e.g. [Bethard et al., 2004] [Choi et al., 2005] [Kim and Hovy, 2006] • Opinion trigger (opinion words) • e.g. [Yu and Hatzivassiloglou, 2003] [Riloff and Wiebe, 2003] • Polarity – positive/negative • As above • Strength • e.g. [Wilson et al. 2004] • Topic (target) • ???? LREC 2008, Marakech, Morocco

  13. Annotating Topics of Fine-grained Opinions • Definitions • Issues • Approach and Corpus • IA agreement LREC 2008, Marakech, Morocco

  14. Examples (1)[OH John] likes Marseille for its weather and cultural diversity. (2)[OH Al] thinks that the government should tax gas more in order to curb CO2 emissions. LREC 2008, Marakech, Morocco

  15. Definitions (1)[OH John] likes Marseille for its weather and cultural diversity. LREC 2008, Marakech, Morocco

  16. Definitions (1)[OH John] likes Marseille for its weather and cultural diversity. Topic: city of Marseille • Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder LREC 2008, Marakech, Morocco

  17. Definitions (1)[OH John] likes [TOPIC SPAN Marseille] for its weather and cultural diversity. Topic: city of Marseille • Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder • Topic span - the closest, minimal span of text that mentions the topic LREC 2008, Marakech, Morocco

  18. Definitions (1)[OH John] likes [TARGET+TOPIC SPAN Marseille] for its weather and cultural diversity. Topic: city of Marseille • Topic - the real-world object, event or abstract entity that is the subject of the opinion as intended by the opinion holder • Topic span - the closest, minimal span of text that mentions the topic • Target span - the span of text that covers the syntactic surface form comprising the contents of the opinion LREC 2008, Marakech, Morocco

  19. Definitions (2)[OH Al] thinks that the government should tax gas more in order to curb CO2 emissions. LREC 2008, Marakech, Morocco

  20. Definitions (2)[OH Al] thinks that [TARGET SPAN the government should tax gas more in order to curb CO2 emissions]. LREC 2008, Marakech, Morocco

  21. Definitions (2)[OH Al] thinks that [TARGET SPAN[TOPIC SPAN? the government] should [TOPIC SPAN? tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]]. LREC 2008, Marakech, Morocco

  22. Definitions (2)[OH Al] thinks that [TARGET SPAN the government should tax gas more in order to curb CO2 emissions]. Context: (3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution. LREC 2008, Marakech, Morocco

  23. Definitions (2)[OH Al] thinks that [TARGET SPAN the government should [TOPIC SPANtax gas] more in order to curb CO2 emissions]. Context: (3) Although he doesn’t like government imposed taxes, he thinks that a fuel tax is the only effective solution. LREC 2008, Marakech, Morocco

  24. Related Work • Product reviews • E.g. Kobayashi et al. (2004), Yi et al. (2003), Popescu and Etzioni (2005), Hu and Liu (2004 • Limit “topics” to mentions of product names, components, and their attributes • Lexicon look-up • Focused on methods for lexicon acquisition • MPQA corpus (Wiebe, Wilson, Cardie, 2004) • Fine-grained opinions • Topic annotation deemed too difficult • Target span annotation is underway • Kim & Hovy (2006) • Target span extraction using semantic frames • Limited evaluation LREC 2008, Marakech, Morocco

  25. Issues in Opinion Topic Identification • Multiple potential topics mentioned within a single target span (2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the government] should [TOPIC SPAN? tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2 emissions]]]. • Requires context Topic of an opinion is the entity that comprises the main information goal of the opinion based on the discourse context. LREC 2008, Marakech, Morocco

  26. Issues in Opinion Topic Identification • Opinion topics are not always explicitly mentioned (4) [OH John] believes the violation of Palestinian human rights is one of the main factors. Topic: ISRAELI-PALESTINIAN CONFLICT (5) [OH I] disagree entirely! LREC 2008, Marakech, Morocco

  27. A Coreference Approach • Hypothesize that the notion of topic coreference will facilitate identification of opinion topics • Easier than specifying the topic of each opinion in isolation Two opinions are topic-coreferent if they share the same opinion topic. LREC 2008, Marakech, Morocco

  28. Opinion Topic Corpus (www.cs.pitt.edu/mpqa) Build on the MPQA corpus: • 535 Documents manually annotated for fine-grained opinions • No opinion topic annotation • Our goal: Add the opinion topic information on top of the existing opinion annotations • Created and used a GUI LREC 2008, Marakech, Morocco

  29. List of opinions to be processed Set of current clusters Document text Annotation Process LREC 2008, Marakech, Morocco

  30. Annotation Process LREC 2008, Marakech, Morocco

  31. fuel tax Annotation Process LREC 2008, Marakech, Morocco

  32. Interannotator Agreement • Annotator 1 • 150 of the 535 MPQA documents • Annotator 2 • 20 of these 150 • IAG measures from noun phrase coreference resolution LREC 2008, Marakech, Morocco

  33. Interannotator Agreement • Annotator 1 • 150 of the 535 MPQA documents • Annotator 2 • 20 of these 150 • IAG measures from noun phrase coreference resolution LREC 2008, Marakech, Morocco

  34. Baselines • all-in-one • assigns all opinions to the same cluster • 1 opinion per cluster • assigns each opinion to its own cluster • same paragraph • opinions in the same paragraph are assigned to the same cluster LREC 2008, Marakech, Morocco

  35. Results • Baselines • vs. Interannotator agreement LREC 2008, Marakech, Morocco

  36. Questions? Thank you Annotation instructions + more information available at: www.cs.cornell.edu/~ves LREC 2008, Marakech, Morocco

  37. Example The Australian press has launched a bitter attack on Italy after seeing their beloved Socceroos eliminated on a controversial late penalty. Italian coach Lippi has been blasted for his favorable comments toward the penalty. Lippi is preparing his side for the upcoming clash with Ukraine. He hailed 10-man Italy's determination to beat Australia and reiterated that the penalty was rightly given. LREC 2008, Marakech, Morocco

  38. Example – fine-grained opinions [SThe Australian press] has launched a bitter attack on [TItaly] after seeing [Stheir]beloved[TSocceroos] eliminated on a controversial late [Tpenalty]. [S+TItalian coach Lippi] has also been blasted for his favorable comments toward [Tthe penalty]. Lippi is preparing his side for the upcoming clash with Ukraine. [SHe]hailed 10-man [TItaly]'s determination to beat Australia and reiterated that [Tthe penalty] was rightly given. LREC 2008, Marakech, Morocco

  39. Motivation • Sentiment analysis: Useful as stand-alone application • Product reviews • Tracking opinions in the press • Flame detection, etc. • Opinion information can benefit many NLP applications • Multi-Perspective Question Answering [Stoyanov, Cardie, Litman and Wiebe. AAAI WS 2004] and [Stoyanov, Cardie and Wiebe. HLT-EMNLP 2005] • Opinion-Oriented Information Retrieval • Clustering, etc. LREC 2008, Marakech, Morocco

  40. Annotation Process LREC 2008, Marakech, Morocco

  41. Annotation Process LREC 2008, Marakech, Morocco

  42. ` LREC 2008, Marakech, Morocco

More Related