Opinions in Question Answering

1. Opinions in Question Answering Jan Wiebe University of Pittsburgh Claire Cardie Cornell University Ellen Riloff University of Utah

2. Overview • Techniques and tools to support multi-perspective question answering (MPQA) • Goals: • produce high-level summaries of opinions • incorporate rich information about opinions extracted from text

3. Opinion Summary Template Overview • Opinion-oriented information extraction • Extract opinion frames for individual expressions • Combine to create opinion-oriented “scenario” templates

4. MPQA Corpus • Grew out of the 2002 ARDA NRRC Workshop on Multi-Perspective Question Answering • Detailed annotations of opinions • Freely available (thanks to David Day): nrrc.mitre.org/NRRC/publications.htm

5. Collaborations • Interactions with end-to-end system teams • Integrated corpus annotation • Pilot opinion evaluation

6. Outline • Recent activities • Subjective sentence identifier • Clause intensity identifier • Extended annotation scheme Version 1 • Q&A corpus • Nested opinions • Opinion summaries • What’s next

7. Subjective Sentence Identifier • Input is unlabeled data • Evaluated on manual annotations of the MPQA corpus • Accuracy as good as supervised systems which classify all sentences

8. Subjective Sentence Identifier • Bootstraps from a known subjective vocabulary, labeling the sentences it can with confidence • Extraction pattern learner finds clues of subjectivity in that corpus • Incorporated into a statistical model trained on the automatically labeled data • Multiple classification strategies • 76% accuracy with 54% baseline • 80% subj. precision and 66% subj. recall • 80% obj. precision: and 51% obj. recall

9. Clause-level intensity (strength) identification • Maximum intensity of the opinions in a clause • Neutral, low, medium, high • Evaluated on manual annotations of the MPQA corpus

10. Example I am furious that my landlord refused to return my security deposit until I sued them. am High Strength I furious that refused Opinionated Sentence landlord return until my to deposit sued Medium Strength my security I them Neutral

11. Clause-level intensity (strength) identification • Classification and regression learners • Accuracy: how many clauses are assigned exactly the correct class? • Mean Squared Error: how close are the answers to the right ones? • Accuracy: classification > regression • 23-79% over baseline • MSE: regression > classification • 57-64% over baseline

12. Opinion Frames The report has been strongly criticized and condemned by many countries. direct subjective annotation Span: “strongly criticized and condemned” Source: <writer,many-countries> Strength (intensity): high Attitudes: negative toward the report Target: report

13. Major Attitude Types • Positive • Negative • Arguing for ones world view • Intention

14. Negative and Positive Example • People are happy because Chavez has fallen, she said. direct subjective annotation span: are happy source: <writer, she, People> attitude: attitude annotation span: are happy because Chavez has fallen type: positive and negative positive target: negative target: target annotation span: Chavez has fallen target annotation span: Chavez

15. Arguing for World View Example • Putin remarked that events in Chechnia “could be interpreted only in the context of the struggle against international terrorism.” direct subjective annotation span: remarked source: <writer, Putin> attitude: attitude annotation span: could be interpreted only in the context of the struggle against international terrorism type: argue for world view target: target annotation span: events in Chechnia

16. Characteristics • Sarcastic "Great, keep on buying dollars so there'll be more and more poor people in the country," shouted one. • Speculative Leaders probably held their breath… • Characteristics of the linguistic realization

17. Q&A Corpus • Includes 98 documents from the NRRC corpus, split into four topics: • Kyoto Protocol • 2002 elections in Zimbabwe • U.S. annual human rights report • 2002 coup in Venezuela

18. Q&A Corpus • Includes 30 questions • 15 questions classified as fact • What is the Kyoto Protocol about? • What is the Kiko Network? • Where did Mugabe vote in the 2002 presidential election? • 15 questions classified as opinion • How do European Union countries feel about the US opposition the Kyoto protocol? • Are the Japanese unanimous in their opinion of Bush’s position on the Kyoto Protocol? • What was the American and British reaction to the reelection of Mugabe?

19. Q&A Corpus • Answers annotations added by two annotators • Minimal spans that constituted or contributedto an answer • Confidence • Partial?

20. Difficulties in Corpus Creation • Annotating answers • Difficult to decide what constitutes an answer: • Q: “Did most Venezuelans support the 2002 coup?” • A: “Protesters…failed to gain the support of the army.” ??? • Not clear what sources to attribute to collective entities • European Union: The EU Parliament? Great Britain? GB government? Tony Blair? • The Japanese: The Japanese government? Emperor Akihito? Empress Michiko? The Kiko Network?

21. Q&A Corpus • Interannotator agreement • 85% on average • using Wiebe et. al’s agr(a||b) measure • 78% and 93%, respectively for each annotator

22. Evaluating MPQA Opinion Annotations • Answer probability: estimate P(opinion answer | opinion question) P(fact answer | fact question) • Low-level opinion information reliable predictor • facts: 78% • opinions: 93% • Answer rank • Sentence-based retrieval • Filter based on opinion annotations • Examine rank of first sentence w/answer • Filtering improves answer rank

23. Direct subjective annotation Source: Attitude: Opinion Summary Template Summary Representations of Opinions

24. Reporting in text • Clappsums upthe environmental movement’s reaction: “The polluters are unreasonable’’ • Charlie was angry at Alice’s claim that Bob was unhappy

25. implicit speech event implicit speech event sums up angry claim unhappy reaction Charlie was angry at Alice’s claim that Bob was unhappy Clappsums upthe environmental movement’s reaction: “The polluters are unreasonable’’ Hierarchy of Perspective & Speech Expressions

26. implicit 66% correct angry unhappy unhappy claim unhappy Baseline 1: Only filter through writer

27. implicit angry claim claim claim unhappy unhappy unhappy 72% correct Baseline 2: Dependency Tree

28. 78% correct ML Approach • Features • Parse-based • Positional • Lexical • Genre-specific • IND decision trees (mml criterion)

29. Direct subjective annotation Source: Attitude: Opinion Summary Template Summary Representations of Opinions

30. Opinion Summaries • Summaries based on manual annotations • Single-document summaries • Opinion annotations grouped by source and target • Sources characterized by degree of subjectivity/objectivity • Simple graph-based graphical interface • Overview of entire graph • Focus on portion of the graph • Drill-down to opinion annotations (highlighted) • Grouping/deleting of sources/targets • JGRAPH package

31. The next 6 months • Identify individual expressions of subjectivity • Perform manual annotations • Extract Sources • Opinion summaries with automatic annotations