1 / 103

Participants

Janyce Wiebe Eric Breck Chris Buckley Claire Cardie Paul Davis. Bruce Fraser Diane Litman David Pierce Ellen Riloff Theresa Wilson. Participants. Part I: Overview. Finding and organizing opinions in the world press and other text. Problem. Our Work will Support.

joanne
Download Presentation

Participants

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Janyce Wiebe Eric Breck Chris Buckley Claire Cardie Paul Davis Bruce Fraser Diane Litman David Pierce Ellen Riloff Theresa Wilson Participants

  2. Part I: Overview

  3. Finding and organizing opinions in the world press and other text Problem

  4. Our Work will Support • Finding a range of opinions expressed on a particular topic, event, issue • Clustering opinions and their sources • Attitude (positive, negative, uncertain) • Basis for opinion (supporting beliefs, experiences) • Expressive style (sarcastic, vehement, neutral) • Building perspective profiles of individuals and groups over many documents and topics

  5. Task: Conceptualization • Perspective in language • Implications for higher-level tasks

  6. Manual annotation scheme for linguistic expressions of opinions “It is heresy,” said Cao. “The `Shouters’ claim they are biggerthan Jesus.” Task: Annotation (writer,Cao) (writer,Cao,Shouters) (writer,Cao) (writer,Cao)

  7. ATTITUDE value: neg strength: medium <writer,many countries>: <writer>: <report> subjectivity index: 1.0 (1/1) subjectivity index: 0.4 (4/10) ATTITUDE value: neg strength: low ATTITUDE value: neg strength: high <US> Task: Summary Representation of Opinions • Built upon lower-level annotations

  8. Task: Automate Manual Annotations • Machine learning • Identification of opinionated phrases, sources of opinions, …

  9. Task: Organizing Perspective Segments • Unsupervised clustering • Text features + features from the annotation scheme + higher-level features

  10. Evaluation • Exploratory manual clustering • Evaluation of annotation agreement • Evaluation of automatic annotations • Evaluation of automatic clustering

  11. Solution Architecture Annotation Architecture AnnotationTool Learning Architecture LearningAlgorithms Trained Taggers Application Architecture DocumentRetrieval PerspectiveTagging DocumentClustering Question Other Taggers

  12. Part II: Annotation

  13. Annotation • Find opinions, evaluations, emotions, speculations (private states)expressed in language Private state: state that is not open to objective observation or verification. Quirk, Greenbaum, Leech, Svartvik (1985). A Comprehensive Grammar of the English Language.

  14. Two Ways of Expressing Private States • Explicit mentions of private states and speech events • The United States fears a spill-over from the anti-terrorist campaign • Expressive subjective elements • The part of the US human rights report about China is full of absurdities and fabrications.

  15. Two Ways of Expressing Private States • Explicit mentions of private states and speech events: “Ons” • The United States fears a spill-over from the anti-terrorist campaign • Expressive subjective elements • The part of the US human rights report about China is full of absurdities and fabrications.

  16. (writer, Xirao-Nima, US) (writer, Xirao-Nima) (writer, Xirao-Nima) (writer, Xirao-Nima) “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities. (writer) (writer) “The report is full of absurdities,’’ he continued. Nested Sources

  17. OnlyFactive=no (writer, Xirao-Nima, US) (writer, Xirao-Nima) (writer) OnlyFactive=yes OnlyFactive OnlyFactive=yes “The US fears a spill-over’’, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities.

  18. OnlyFactive • An example of all OnlyFactive=yes The government, it added, has amended Pakistan Citizenship Act 10 of 1951 to enable women of Pakistani descent to claim Pakistani nationality for their children born to foreign husbands.

  19. “It is heresy,” said Cao. “The `Shouters’ claim they are bigger than Jesus.” (writer,Cao) (writer,Cao,Shouters) (writer,Cao) (writer,Cao) Example

  20. (writer,FM) (writer,FM,FM) (writer,FM) (writer,FM,FM,SD) (writer,FM) (writer,FM) Example The Foreign Ministry said Thursday that it was “surprised, to put it mildly” by the U.S. State Department’s criticism of Russia’s human rights record and objected in particular to the “odious” section on Chechnya.

  21. Other attributes • Strengths of various components • Types of private states • Objects

  22. Documents and Annotators • Over 90 annotated documents • 100%: spans of sources, subjective elements, ons, with type and OnlyFactive attributes. • 39%: + sources, strengths, agent objects • Participants + part-time annotators

  23. Pilot Agreement Evaluation:ons and subjective elements • Agreement measured by percentage of A’s constituents matching B’s, and vice versa

  24. Overlapping Spans: Ons Bush has adopted the most pro-Taiwan posture of any president... Bush has adopted the most pro-Taiwan posture of any president...

  25. Overlapping Spans: Expressive Subjective Elements …some of Mr. Chavez’s more alarming faults… …some of Mr. Chavez’s more alarming faults…

  26. Untrained Annotators • 2 groups of 3 untrained annotators (the participants) each annotated 4 documents without any communication

  27. Spans, Untrained Annotators

  28. Overlapping Spans: Expressive Subjective Elements …some of Mr. Chavez’s more alarming faults… …some of Mr. Chavez’s more alarming faults…

  29. C: 74 P: 196 D: 75 Expressive Subjectivity Agreement Di: 76 T: 153 E: 29; 97%

  30. Spans, All Annotators

  31. Pilot Agreement Evaluation:OnlyFactive • For the onsidentified by both annotators: Measured agreement for Onlyfactive • Cohen’s Kappa • Krippendorf: 0.67+ Tentative conclusions 0.80+ Definite conclusions

  32. OnlyFactive Kappa Values

  33. Future work • Continue annotations by trained annotators • Continue to explore annotations • Train new annotators, starting with settled scheme • Even more detail is possible

  34. Part III: Summary Representations of Opinions

  35. Example The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  36. Sentence 1 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  37. <writer>:onlyfactive<writer>:expr-subj (medium) Sentence 1: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  38. Sentence 2 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  39. Sentence 2: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>: neg-attitude (medium) <report> <writer>: neg-attitude (medium) <writer>: neg-attitude (medium)

  40. Sentence 3 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  41. Sentence 3: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>:onlyfactive <writer, many-countries>:neg-attitude (medium) <report> <writer, many-countries>:extreme <writer, many-countries>:neg-attitude(high, high, medium)

  42. Sentence 4 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  43. Sentence 4: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>:onlyfactive<writer, info-office>:neg-attitude (medium)<US>

  44. Sentence 5 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  45. Sentence 5: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>:onlyfactive <writer, chinarep>:onlyfactive

  46. Sentence 6 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  47. Sentence 6: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>: ?neg-attitude (medium)<US><writer>:expr-subj (low)

  48. Sentence 7 The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years.

  49. Sentence 7: Low-level The Annual Human Rights Report of the US State Department has been strongly criticized and condemned by many countries. Though the report has been made public for 10 days, its contents, which are inaccurate and lacking good will, continue to be commented on by the world media. Many countries in Asia, Europe, Africa, and Latin America have rejected the content of the US Human Rights Report, calling it a brazen distortion of the situation, a wrongful and illegitimate move, and an interference in the internal affairs of other countries. Recently, the Information Office of the Chinese People's Congress released a report on human rights in the United States in 2001, criticizing violations of human rights there. The report quoting data from the Christian Science Monitor, points out that the murder rate in the United States is 5.5 per 100,000 people. In the United States, torture and pressure to confess crime is common. Many people have been sentenced to death for crime they did not commit as a result of an unjust legal system. More than 12 million children are living below the poverty line. According to the report, one American woman is beaten every 15 seconds. Evidence show that human rights violations in the United States have been ignored for many years. <writer>: neg-attitude (low)  <US><writer>:expr-subj (low) <writer>: neg-attitude (medium)

More Related