Annotating Expressions of Opinions and Emotions in Language

1. Annotating Expressions of Opinions and Emotions in Language • Wiebe, Wilson, Cardie

2. Overview • This paper focused on modeling the problem of sentiment analysis and on describing an annotation strategy for capturing sentiment expressions. • The authors discuss the increasing need for annotated corpora as NLP researchers adopt machine learning methods, and so the authors propose a strategy which they feel is widely applicable to the problem space. • They state their goal as follows: "The goals of the annotation scheme are to represent internal mental and emotional states, and to distinguish subjective information from material presented as fact."

3. Modeling The Problem • Their annotation strategy is organized around the two central ideas of private states and nested sources. • Private States: They define the idea of private states as "internal states that cannot be directly observed by others."  • Nested Sources: The authors also note that writers frequently express opinions about other people's opinions, and so they define nested sources to capture this recursive phenomenon.

4. Private State Frames • They define private states as "experiencers holding attitudes, optionally toward targets". In the example sentence "John hates Mary", the experiencer is John the attitude is hate and the target is Mary. • The authors define several types of private state frames which allow them to capture certain properties of each private state expression like: • Polarity: Is it a positive or negative sentiment? • Intensity: Is the sentiment intensity low, medium, high, or extreme?

5. Types of Frames • Direct subjective frame - This is used to represent explicitly mentioned private states (like John hates Mary), as well as subjective speech events (like "The report is full of absurdities," Xirao-Nima said.) • Expressive subjective frame - This is used to represent private states that are suggested by the choice of language (like "full of absurdities") • Objective speech event frame - This frame is used "to distinguish opinion-oriented material from material presented as factual."

6. Nested Sources • Nested sources can be seen fairly clearly with an example. Consider this sentence, "China criticized the U.S. report's criticism of China's human rights record." • With regard to the private state anchored on "criticism": • The first source is the actual author of the sentence • The second source is China • The third source is the U.S. report

7. Applying the Strategy & Observations • After the authors describe their annotation strategy they then have a team of people manually annotate a corpus of world news articles. • The authors then highlight some of the complexity they observed after reviewing the annotated corpus: • Many different parts of speech were tagged as anchors for private states • Many words used both subjectively and objectively • Many sentences mixed objectivity and subjectivity • Many sentences didn’t fall into “positive” or “negative” polarity categories • Sentiments with stronger intensity were easier for all annotators to identify

8. Inter-Annotator Analysis

9. Strengths & Weaknesses • Strengths • The idea of moving towards generally applicable annotation strategies seems like a good idea and could be useful. • They do a good job breaking down the different types of sentiment phenomena that occur. • Weaknesses • There is still a lot of complex phenomena with regard to sentiment that their strategy doesn't capture. • Using polarity and intensity to describe the nature of private states seems inadequate.