1 / 35

Annotating Opinions in the World Press

Annotating Opinions in the World Press. Theresa Wilson and Janyce Wiebe University of Pittsburgh Intelligent Systems Program and Department of Computer Science. Introduction. Private State : mental or emotional state, which cannot be directly observed or verified (Quirk et al., 1985).

maxwell-jordan
Download Presentation

Annotating Opinions in the World Press

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Annotating Opinions in the World Press Theresa Wilson and Janyce Wiebe University of Pittsburgh Intelligent Systems Program and Department of Computer Science

  2. Introduction Private State: mental or emotional state, which cannot be directly observed or verified (Quirk et al., 1985) Speculation Emotions Beliefs Opinions Evaluations SIGdial 2003

  3. Private States in Text Subjectivity: aspects of language used to express private states, including opinions, emotions, evaluations, speculations, sentiments (Banfield 1982) “Complained” “You Idiot!” “Terrible product” “Enthused” “Wonderful!” “Great idea” “Speculated” “Maybe” “I wonder” (See Fodor 1979, Cohn 1978, Chafe 1986, Fludernik 1993, Wiebe 1994, Stein & Wright 1995 for additional references.) SIGdial 2003

  4. Why do we care? • News articles mix of factual/subjective language • Help existing NLP applications: IE, IR, summarization systems, etc. • Support new tasks: multi-perspective question answering SIGdial 2003

  5. Two Main Ways of Expressing Private States • Direct mentions of private states The United States fears a spill-over from the anti-terrorist campaign. • Expressive subjective elements The part of the US human rights report about China is full of absurdities and fabrications. SIGdial 2003

  6. Private States and Speech Events • Speech Event  speaking or writing “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said. Medical Department head Dr. Hameed Saeed said the patient’s blood has been sent to the Institute for Virology in Johannesburg for analysis. SIGdial 2003

  7. (writer) (writer, Xirao-Nima, US) (writer, Xirao-Nima) “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer) (writer, Xirao-Nima) (writer, Xirao-Nima) “The report is full of absurdities,’’ he continued. Nested Sources SIGdial 2003

  8. Annotation Scheme • Implemented in GATE (http://gate.ac.uk) Shorthand for direct private state or speech event expression SIGdial 2003

  9. On (Direct Private State or Speech Event) nested-source onlyfactive overall-strength on-strength attitude-type attitude-toward is-implicit minor Expressive Subjective Element nested-source strength attitude-type Attributes for Annotation Types SIGdial 2003

  10. On (Direct Private State or Speech Event) nested-source onlyfactive overall-strength on-strength attitude-type attitude-toward is-implicit minor Expressive Subjective Element nested-source strength attitude-type Attributes for Annotation Types SIGdial 2003

  11. On (Direct Private State or Speech Event) nested-source onlyfactive overall-strength on-strength attitude-type attitude-toward is-implicit minor Expressive Subjective Element nested-source strength attitude-type Attributes for Annotation Types SIGdial 2003

  12. On (Direct Private State or Speech Event) nested-source onlyfactive overall-strength on-strength attitude-type attitude-toward is-implicit minor Expressive Subjective Element nested-source strength attitude-type Attributes for Annotation Types SIGdial 2003

  13. (writer, Xirao-Nima, US) onlyfactive=no (writer, Xirao-Nima) onlyfactive=yes (writer) onlyfactive=yes “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. onlyfactive Attribute SIGdial 2003

  14. Important! onlyfactive=no may still contain facts “John criticized Mary for smoking.” (writer, John) onlyfactive=no SIGdial 2003

  15. on-strength=neutral The Foreign Ministry said . . . on-strength=medium The Foreign Ministry objected . . . The Foreign Ministry vehemently denied . . . . . . inaccurate section on Chechnya. strength=low . . . odious section on Chechnya. strength=high Strength of Private State on-strength=high SIGdial 2003

  16. overall-strength=high on-strength=medium strength=high The Foreign Ministry objected in particular to the “odious” section on Chechnya. strength=medium strength=medium “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said. on-strength=neutral overall-strength=medium Overall Strength of Private State SIGdial 2003

  17. Data • English-language versions of articles from world press (FBIS + few other sources) • 8500 sentences have been annotated • Agreement Study • 3 annotators (A, M, S) • 210 sentences (13 documents) • 1/3 news, 1/3 news on “hot-topic”, 1/3 editorials SIGdial 2003

  18. Types of Agreement • Agreement for spans • Ons • Expressive Subjective Elements • Agreement for ON  onlyfactive attribute • Agreement for sentences • Derive sentence-level annotations from lower-level annotations SIGdial 2003

  19. A: His comments were immediately contested . . . M: His comments were immediately contested . . . Agreement Issues: Spans • Overlapping spans ok  it’s a match SIGdial 2003

  20. (writer, Russian Miliatary) A: Russian military policy has been to keep a distance from . . . M: Russian military policy has been to keep a distance from . . . (writer, Russian Miliatary) Agreement Issues: Spans • Different set of annotations for each annotator  Kappa inappropriate SIGdial 2003

  21. |A matching B| |A| Agreement Evaluation: spans • Agreement measured by percentage of A’s constituents matching B’s, and vice versa agr (a||b) = SIGdial 2003

  22. Span Agreement: Ons SIGdial 2003

  23. A:Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world." But main stream Russian military policy has been to keep a distance from America. M:Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“ But main stream Russian military policy has been to keep a distance from America.  writer on Measuring Agreement: ON  onlyfactive SIGdial 2003

  24. A:Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world." But main stream Russian military policy has been to keep a distance from America. M:Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“ But main stream Russian military policy has been to keep a distance from America.  writer on Measuring Agreement: ON  onlyfactive SIGdial 2003

  25. A: Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world." But main stream Russian military policy has been to keep a distance from America. M: Reformers told Putin all along that he should take advantage of "the rare opportunity for (Russia) to become integrated with the civilized world.“ But main stream Russian military policy has been to keep a distance from America.  writer on Measuring Agreement: ON  onlyfactive onlyfactive=yes onlyfactive=no onlyfactive=yes onlyfactive=no onlyfactive=no onlyfactive=yes SIGdial 2003

  26. Agreement results: onlyfactive Ave Kappa Borderlineonlyfactive: • At least one annotator marked expression onlyfactive=no • Neither annotator marked expression overall-strength > low SIGdial 2003

  27. (writer) onlyfactive=yes (writer, Sue) onlyfactive=no Subjective Sue said, “The election was fair.” (writer) onlyfactive=yes (writer, Sue) onlyfactive=yes Objective Sue said, “The election was Tuesday.” Deriving Sentence-level Annotations SIGdial 2003

  28. Agreement results: sentence-level subjective annotations Ave Kappa Borderline subjective sentence: • At least one annotator marked at least one expression onlyfactive=no • Neither annotator marked an overallstrength > low SIGdial 2003

  29. Conclusions • Detailed scheme for annotation of opinions and other private states • Good agreement between annotators • Strong agreement about clear cases of subjectivity • Lower-level annotations  higher-level subjective sentence annotations • Extendable SIGdial 2003

  30. Using GATE to perform private state annotations • GATE http://gate.ac.uk • General annotation instructions and GATE instructions http://www.cs.pitt.edu/~wiebe/pubs/ardasummer02 SIGdial 2003

  31. Thank you This work was performed in support of the Northeast Regional Research Center (NRRC) which is sponsored by the Advanced Research and Development Activity in Information Technology (ARDA), a U.S. Government entity which sponsors and promotes research of import to the Intelligence Community which includes but is not limited to the CIA, DIA, NSA, NIMA and NRO. SIGdial 2003

  32. Contingency Table: onlyfactive judgements Agreement for onlyfactive judgments, borderline cases removed Agreement for onlyfactive judgments SIGdial 2003

  33. Onlyfactive Agreement SIGdial 2003

  34. Subjective Sentence Agreement SIGdial 2003

  35. Why we exclude minor on annotations This is the first time that the Cuban government has expressed its position over the U.S. holding of the Taliban and al-Qaeda prisoners in its territory. (writer, Cuban gov) onlyfactive=no Objective Sentence! SIGdial 2003

More Related