Multi-Perspective Question Answering

1. Multi-Perspective Question Answering ARDA NRRC Summer 2002 Workshop

2. Janyce Wiebe Eric Breck Chris Buckley Claire Cardie Paul Davis Bruce Fraser Diane Litman David Pierce Ellen Riloff Theresa Wilson Participants

3. Part I: Overview

4. Finding and organizing opinions in the world press and other text Problem

5. Our Work will Support • Finding a range of opinions expressed on a particular topic, event, issue • Clustering opinions and their sources • Attitude (positive, negative, uncertain) • Basis for opinion (supporting beliefs, experiences) • Expressive style (sarcastic, vehement, neutral) • Building perspective profiles of individuals and groups over many documents and topics

6. Task: Conceptualization • Perspective in language • Implications for higher-level tasks

7. Manual annotation scheme for linguistic expressions of opinions “It is heresy,” said Cao. “The `Shouters’ claim they are biggerthan Jesus.” Task: Annotation (writer,Cao) (writer,Cao,Shouters) (writer,Cao) (writer,Cao)

8. ATTITUDE value: neg strength: medium <writer,many countries>: <writer>: <report> subjectivity index: 1.0 (1/1) subjectivity index: 0.4 (4/10) ATTITUDE value: neg strength: low ATTITUDE value: neg strength: high <US> Task: Summary Representation of Opinions • Built upon lower-level annotations

9. Task: Automate Manual Annotations • Machine learning • Identification of opinionated phrases, sources of opinions, …

10. Task: Organizing Perspective Segments • Unsupervised clustering • Text features + features from the annotation scheme + higher-level features

11. Evaluation • Exploratory manual clustering • Evaluation of annotation agreement • Evaluation of automatic annotations • Evaluation of automatic clustering

12. Part II: Conceptualization

13. Introduction • Perspective – Framework/filter through which we see an event • Language give insight into perspective of writer/ speaker and the people they describe • Our focus: subjectivity (reflections of private states) as conveyed through language • Goal: Draw from and elaborate on the Annotation Guidelines to expand our knowledge of how subjectivity is conveyed via linguistic expressions.

14. Influences on Perspective • Identity (President, Assistant Secretary of State) • Attitudes held by a person: • Micro (towards specific individuals, events) • Macro (belief systems embraced, culture values) • Genre of the product (editorial, report, white paper) • Context of the writing: • Temporal (writing now about 1960) • Spatial (writing located in war zone) • Events (writing now rather than before 9-11)

15. Lexical Clues to Subjectivity Attitude Expression <Source, Attitude, Object> • Explicit Attitudes – Direct realization of private states • Implicit Attitudes – Expressive subjective elements • Other linguistic clues

16. Attitude Expression • The Source (writer, speaker, another person an institution, a document • The Object (entities, actions, propositions, topic, context) • Attitudes – Different typologies

17. Attitude Typologies • Negative (reject); Positive (enjoy); Other (surprise) • Opinion (criticize); Emotions (fear); Speculation (probably) • Other (must, poor, better)

18. Attitude Types • Content Evidential (allegedly, think) Deontic (require, should) Assessment (like, excellent) • Process Discourse Markers (so, but) Operators (very, not)

19. Types of AE Combinations • Reinforced Attitude (A+) – Two AEs of the same type (I like her and I admire her as well) • Weakened Attitude (A-) – Two AEs, A and ~A (I like her. I can’t stand her politics) • Entailed Attitude (=>A)- AE A entails AE B (He terrorizes people => people fear) • Modified Attitude (A(B)) – AE A modifies AE B (He cruelly criticized the people)

20. Subjectivity: Relation to Ideology • Ideology – Beliefs and opinions held • Approximation of ideology by grouping AEs for a given Source • Grouping Sources with similar ideologies • Inferring ideology from limited AEs • Change of an ideology over time • Prediction of position based on ideology

21. Solution Architecture Annotation Architecture AnnotationTool Learning Architecture LearningAlgorithms Trained Taggers Application Architecture DocumentRetrieval PerspectiveTagging DocumentClustering Question Other Taggers

22. Part III: Annotation

23. Annotation • Find opinions, evaluations, emotions, speculations (private states)expressed in language Private state: state that is not open to objective observation or verification. Quirk, Greenbaum, Leech, Svartvik (1985). A Comprehensive Grammar of the English Language.

24. Two Ways of Expressing Private States • Explicit mentions of private states and speech events • The United States fears a spill-over from the anti-terrorist campaign • Expressive subjective elements • The part of the US human rights report about China is full of absurdities and fabrications.

25. Two Ways of Expressing Private States • Explicit mentions of private states and speech events: “Ons” • The United States fears a spill-over from the anti-terrorist campaign • Expressive subjective elements • The part of the US human rights report about China is full of absurdities and fabrications.

26. (writer, Xirao-Nima, US) (writer, Xirao-Nima) (writer, Xirao-Nima) (writer, Xirao-Nima) “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer) (writer) “The report is full of absurdities,’’ he continued. Nested Sources

27. OnlyFactive=no (writer, Xirao-Nima, US) (writer, Xirao-Nima) (writer) OnlyFactive=yes OnlyFactive OnlyFactive=yes “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities.

28. OnlyFactive • An example of all OnlyFactive=yes The government, it added, has amended Pakistan Citizenship Act 10 of 1951 to enable women of Pakistani descent to claim Pakistani nationality for their children born to foreign husbands.

29. “It is heresy,” said Cao. “The `Shouters’ claim they are bigger than Jesus.” (writer,Cao) (writer,Cao,Shouters) (writer,Cao) (writer,Cao) Example

30. (writer,FM) (writer,FM,FM) (writer,FM) (writer,FM,FM,SD) (writer,FM) (writer,FM) Example The Foreign Ministry said Thursday that it was “surprised, to put it mildly” by the U.S. State Department’s criticism of Russia’s human rights record and objected in particular to the “odious” section on Chechnya.

31. Other attributes • Strengths of various components • Types of private states • Objects

32. Documents and Annotators • Over 90 annotated documents • 100%: spans of sources, subjective elements, ons, with type and OnlyFactive attributes. • 39%: + sources, strengths, agent objects • Participants + part-time annotators

34. Pilot Agreement Evaluation:ons and subjective elements • Agreement measured by percentage of A’s constituents matching B’s, and vice versa

35. Overlapping Spans: Ons Bush has adopted the most pro-Taiwan posture of any president... Bush has adopted the most pro-Taiwan posture of any president...

36. Overlapping Spans: Expressive Subjective Elements …some of Mr. Chavez’s more alarming faults… …some of Mr. Chavez’s more alarming faults…

37. Untrained Annotators • 2 groups of 3 untrained annotators (the participants) each annotated 4 documents without any communication

38. Spans, Untrained Annotators

39. Overlapping Spans: Expressive Subjective Elements …some of Mr. Chavez’s more alarming faults… …some of Mr. Chavez’s more alarming faults…

40. C: 74 P: 196 D: 75 Expressive Subjectivity Agreement Di: 76 T: 153 E: 29; 97%

41. Spans, All Annotators

42. Pilot Agreement Evaluation:OnlyFactive • For the onsidentified by both annotators: Measured agreement for Onlyfactive • Cohen’s Kappa • Krippendorf: 0.67+ Tentative conclusions 0.80+ Definite conclusions

43. OnlyFactive Kappa Values

44. Future work • Continue annotations by trained annotators • Continue to explore annotations • Train new annotators, starting with settled scheme • Even more detail is possible

45. Part IV: Summary Representations of Opinions

46. Example The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

47. Sentence 1 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

48. <writer>:onlyfactive<writer>:expr-subj (medium) Sentence 1: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

49. Sentence 2 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

50. Sentence 2: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>: neg-attitude (medium) <report> <writer>: neg-attitude (medium) <writer>: neg-attitude (medium)