1 / 14

A cognitive study of subjectivity extraction in sentiment annotation

A cognitive study of subjectivity extraction in sentiment annotation. Abhijit Mishra 1 , Aditya Joshi 1,2,3 , Pushpak Bhattacharyya 1 1 IIT Bombay, India 2 Monash University, Australia 3 IITB-Monash Research Academy.

keren
Download Presentation

A cognitive study of subjectivity extraction in sentiment annotation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A cognitive study of subjectivity extraction in sentiment annotation Abhijit Mishra1, Aditya Joshi1,2,3, Pushpak Bhattacharyya1 1 IIT Bombay, India 2 Monash University, Australia 3IITB-Monash Research Academy At 5th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, ACL 2014, Baltimore

  2. Subjectivity Extraction • Goal: To identify subjective portions of text

  3. Motivation • Strong AI suggests that a machine must be perform sentiment analysis in a manner and accuracy similar to human beings • Do humans perform subjective extraction as well? A “cognitive study” of subjectivity extraction in sentiment annotation

  4. Outline • Sentiment Oscillations & Subjectivity Extraction • Experiment Setup • Anticipation & Homing • Conclusion & Future Work

  5. Sentiment Oscillations & subjectivity extraction • Subjective documents may be: • Humans perform subjectivity extraction either as a result of “anticipation” or as “homing”. • Which of the two methods are adopted depends on the linear/oscillating nature of the subjective document. Linear: The story was captivating. The actors did a great job. I absolutely loved the movie! Oscillating: The story was captivating. Only if they had better actors. But then I enjoyed the movie, on the whole.

  6. Experiment Setup (1/2) • A human annotator reads a document and predicts its sentiment • A Tobii T120 eye-tracker records eye movements while he/she reads the document * No time restriction, no user input required: to minimize errors.

  7. Experiment Setup (2/2) • Dataset • 3 Movie reviews in English from imdb • One linear, one oscillating, one between the two extremes (D0, D1, D2 respectively) • Three documents? Really?! • To eliminate predictability • To reduce errors due to fatigue • 12 human annotators (P0, .. P11 respectively)

  8. Observations: Anticipation (1/2) • In case of linear subjective documents, an annotator reads some sentences and begins to skip sentences.

  9. Observations: Anticipation (2/2)

  10. Observations: Homing (1/3) • In case of oscillating subjective documents, an annotator (a) first reads all sentences, (b) revisits some sentences again

  11. Observations: Homing (2/3) • Considerable overlap between sentences that are read in the second pass • All of them are subjective. Reading statistics for D1 TFD: Total fixation duration for subjective extract; PTFD: Proportion of total fixation duration = (TFD)/(Total duration); TFC-SE: Total fixation count for subjective extract

  12. Observations: Homing (3/3) • Homing at a sub-sentence level • Sarcasm • Multiple regressions around the sarcasm portion for participant P1, document D1 • Participant P1 does not correctly detect the sentiment of the document • Thwarting

  13. Conclusion & Future Work • Based on how sentiment changes through a document, humans may perform subjectivity extraction as a result of anticipation or homing • Applications: • Pricing models for crowd-sourced annotation • Sentiment classifiers that incorporate “sentiment runlengths”

  14. References • WikiSent : Weakly Supervised Sentiment Analysis Through Extractive Summarization With Wikipedia, SubhabrataMukherjee and Pushpak Bhattacharyya, ECML PKDD 2012 • A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts, Bo Pang, Lillian Lee, ACL 2004

More Related