1 / 47

Question Answering & Reading Comprehension

Question Answering & Reading Comprehension. Edward Loper University of Pennsylvania. Why Reading Comprehension?. Each answer occurs exactly once: Only one chance to find the answer More incentive to solve the difficult cases Answer is guaranteed to exist Limited text to process:

ajon
Download Presentation

Question Answering & Reading Comprehension

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Question Answering & Reading Comprehension Edward Loper University of Pennsylvania

  2. Why Reading Comprehension? • Each answer occurs exactly once: • Only one chance to find the answer • More incentive to solve the difficult cases • Answer is guaranteed to exist • Limited text to process: • Allows us to experiment w/ deep processing • Simplifies evaluation

  3. Reading Comprehension Corpora • Two evaluation corpora: • A collection of existing reading comprehension tests • New reading comprehension questions based on existing WSJ articles (to leverage existing annotation) • Annotate both passages & questions with: • Syntax (TreeBank) • Verbal predicate information (PropBank) • Noun predicate information (NomBank) • Course grained sense tagging (PropBank II) • Nominal coreference (PropBank II) • Discourse connectives (PropBank II) • Event variables (PropBank II)

  4. Goals • Evaluate system performance • Unbiased questions • Drive research forward • See how our current techniques need to be improved • Include examples just beyond the current state of the art • Include examples that are well beyond current capabilities

  5. Which Examples to Annotate? • Passage: • Random or Selected Topic? (person, episode, location,..) • 2 degrees of Difficulty (writing style, vocabulary, etc.) • Question: • Topic (person, duration, cause, etc.) • Reasoning needed (spatial, event, domain specific, etc.) • Knowledge needed (coref, discourse analysis, etc.) • Difficulty

  6. An Informal Survey of Reading Comprehension Passages • What knowledge resources do we need? • Examine Q/A pairs from reading comprehension tests • What resources do we need to answer each question? • To get the right sentence? • To get the right phrase?

  7. “One of those days” When did James’ alarm clock ring? Q: At six o’clock yesterday morning, James’ alarm clock rang. A: • ? Need:

  8. “One of those days” When did James’ alarm clock ring? Q: At six o’clock yesterday morning, James’ alarm clock rang. A: • Simple normalization • Temporal phrase detection Need:

  9. “Dale Earnhardt” When was his first win? Q: He won his first championship in 1980. A: • ? Need:

  10. “Dale Earnhardt” When was his first win? Q: He won his first championship in 1980. A: • Mapping arguments between a verb and its nominalization • Coreference (in both question & passage) Need:

  11. “Abraham Lincoln” How did Lincoln die? Q: He was shot by John Wilkes Booth in 1865. A: • ? Need:

  12. “Abraham Lincoln” How did Lincoln die? Q: He was shot by John Wilkes Booth in 1865. A: • Match “shoot” w/ “kill” (via VerbNet or WordNet) • WN: hypernym • VerbNet: share a superclass • Match “kill” w/ “cause to die” (WN def.) • Use PropBank to line up arguments (Lincoln=arg1) Need:

  13. “One of those days” What did he have for breakfast? Q: Then he went downstairs and had his breakfast. This consisted of a bowl of cereal with a banana and some milk. A: • ? Need:

  14. “One of those days” What did he have for breakfast? Q: Then he went downstairs and had his breakfast. This consisted of a bowl of cereal with a banana and some milk. A: • Sense tagging (“consume” sense of have = WN#6) • Specify answer type as food (from predicate/argument information about the “consume” sense of have). • Treat consists as equality (“consists” is a hyponym of “to be”) • Coreference (to match this=[his] breakfast) Need:

  15. “Teddy Bears” Why did President Roosevelt refuse to shoot the bear? Q: President Roosevelt was on a hunting trip in Mississippi when members of the hunting party caught a black bear and tied him to a tree. President Roosevelt was called to the area to shoot the bear, which he refused to do and said it was unsportsmanlike and showed poor manners. A: • ? Need:

  16. “Teddy Bears” Why did President Roosevelt refuse to shoot the bear? Q: President Roosevelt was on a hunting trip in Mississippi when members of the hunting party caught a black bear and tied him to a tree. President Roosevelt was called to the area to shoot the bear, which he refused to do and said it was unsportsmanlike and showed poor manners. A: • Coreference (“do” = “shoot the bear”) • Coreference (“he” = “President Roosevelt”) • Discourse connective: “and” = “because” in the last sentence. • For complete answer: discover that “it” = “to shoot a bear tied to a tree,” not just “to shoot a bear.” (general reasoning?) Need:

  17. “Electing our President” The person elected on November 7th will be President for how many years? Q: On Tuesday, November 7, 2000, the people of America will be heard…. The presidential election is very important to the American people because the person elected will be the leader of our country for the next four years. A: • ? Need:

  18. “Electing our President” The person elected on November 7th will be President for how many years? Q: On Tuesday, November 7, 2000, the people of America will be heard…. The presidential election is very important to the American people because the person elected will be the leader of our country for the next four years. A: • Temporal reasoning: we’re still talking about Nov 7. • Match “President” w/ “the leader of our country” (domain knowledge?) • Temporal/event reasoning: go from “the next four years” to “four years” Need:

  19. Knowledge Resources Needed • Vocabulary variation: Lexical matching • Syntactic variation: Phrasal matching • Alternations (incl. lexical-specific alternations) • Coreference • Discourse structure analysis & discourse connectives • Reasoning about locations • Reasoning about times and events • Reasoning about opinions • World knowledge (scenario reasoning/matching)

  20. Existing Resources • WordNet - lexical variation • PropBank - lexico-syntactic variation • VerbNet - lexico-syntactic & semantic variation • FrameNet • Phrase Detectors • Named entities • Time & place phrases • Etc. • Coreference Detectors

  21. Extending Resources • Extending VerbNet • Extending coverage of PropBank • PropBank II • Event variables (“continue growing rice”) • Sense tagging w/ grouped senses • Nominal coreference • Discourse connectives

  22. Penn Discourse Treebankhttp://www.cis.upenn.edu/~pdtb/ Joshi, Miltisaki, Prasad [He failed the exam] although [he had studied hard.] When [the stock market dropped nearly 7% Oct 13, for instance], [the Mexico Fund plunged about 18%, and the Spain Fund fell 16%.]

  23. VerbNethttp://www.cis.upenn.edu/group/verbnet/

  24. PropBankhttp://www.cis.upenn.edu/~ace

  25. Discussion • How much processing do you need to answer questions? • Are all of these resources useful? • What other resources might we use?

  26. “One of those days” Where did he go after he got up? Q: He got up from bed and went to the bathroom. A: • Discourse connectives (to match and=after) • Locative phrase detection Need:

  27. “One of those days” What did he have for breakfast? Q: Then he went downstairs and had his breakfast. This consisted of a bowl of cereal with a banana and some milk. A: • Sense tagging (“consume” sense of have = WN#6) • Specify answer type as food (from predicate/argument information about the “consume” sense of have). • Treat consists as equality (“consists” is a hypernym of “to be”) • Coreference (to match this=[his] breakfast) Need:

  28. “One of those days” What did he do after breakfast? Q: After breakfast, he went to his bedroom and got dressed for work. A: • Simple match (w/ do pro-form) Need:

  29. “One of those days” How did he get to the metro? Q: He arrived downstairs and walked to the metro. A: • Match go=walk (both motion verbs, walk is a specialization of go) Need:

  30. “One of those days” What happened in the metro? Q: He arrived downstairs and walked to the metro. He was too late, he missed his regular metro car and he waited 10 minutes for the next one. He arrived late to work. A: • Locative reasoning: “he walked to the metro” implies he is at the metro; “he arrived late to work” implies he is no longer at the metro. • Discourse structure: what block of discourse is relevant to the question? Need:

  31. “One of those days” Why was his boss angry with him? Q: He arrived late for work. His boss was angry with him. He went outside to his truck and saw that he had a flat tire. A: • Discourse connectives (connection between “he arrived late” and “his boss was angry” is explanatory). • General reasoning (what are possible reasons why his boss might be angry). Need:

  32. “Labor Day” Who do we honor by celebrating Labor Day? Q: Labor day is a legal holiday. It is a day that honors everyone who works. Because almost everyone in America works, this day honors most of the people in our country. A: • Coreference • Match “a day honors X” = “celebrating a day honors X” Need:

  33. “Labor Day” When is Labor day? Q: Labor day is celebrated on the first Monday in September. Most people think of Labor Day as the last day of summer. A: • First answer: Match “X is celebrated on” = “X is on” • Second answer: reasoning about opinions Need:

  34. “Labor Day” What are some ways Labor Day is celebrated? Q: Labor day is celebrated in many ways. Some people go to parades. Others go to hear speakers talk about ways for workers to have better lives. Some people spend the day visiting friends. Many people go on picnics. And some people just rest. A: • Discourse structure reasoning Need:

  35. “Teddy Bears” Why did President Roosevelt refuse to shoot the bear? Q: President Roosevelt was on a hunting trip in Mississippi when members of the hunting party caught a black bear and tied him to a tree. President Roosevelt was called to the area to shoot the bear, which he refused to do and said it was unsportsmanlike and showed poor manners. A: • Coreference (“he” = “President Roosevelt”) • Coreference (“do” = “shoot the bear”) • Discourse connective: “and” = “because” in the last sentence. • For complete answer: discover that “it” = “to shoot a bear tied to a tree,” not just “to shoot a bear.” (general reasoning?) Need:

  36. “Teddy Bears” Who sold the first Teddy Bears? Q: Morris Michtom, a shopkeeper in Brooklyn, New York placed two toy bears in the window of his shop. Mr. Michtom requested permission from the President to call them “ Teddy Bears”. A: • Heuristic domain-specific reasoning: “placed X in shop window” may imply “sold X” • Coreference (“them” = “two bears”) Need:

  37. “Teddy Bears” Where was President Roosevelt hunting when he refused to shoot the bear? Q: President Roosevelt was on a hunting trip in Mississippi when members of the hunting party caught a black bear and tied him to a tree. President Roosevelt was called to the area to shoot the bear, which he refused to do and said it was unsportsmanlike and showed poor manners. A: • Discourse structure/locative reasoning: Roosevelt is still in Mississippi in the second sentence. • Informativeness measures? (“the area” is not an informative answer) • Coreference • Answer merging: we want “near a tree in Mississippi” Need:

  38. “Teddy Bears” In what year was the first Teddy Bear sold? Q: Have you ever wondered where these cute little teddy bears came from? They were named for President Theodore Roosevelt in 1902. […] A: • Similar requirements to “who sold the first Teddy Bear?” -- the article never explicitly says that Teddy Bears were sold. • Temporal reasoning -- the date established by this sentence applies to the next two paragraphs. Need:

  39. “Teddy Bears” In what state was the first Teddy Bear sold? Q: Morris Michtom, a shopkeeper in Brooklyn, New York placed two toy bears in the window of his shop. Mr. Michtom requested permission from the President to call them “ Teddy Bears”. A: • Similar requirements to “who sold the first Teddy Bear?” • Knowledge that “New York” is a state. Need:

  40. “Teddy Bears” Which is not true: a) people collect Teddy Bears; b) Teddy Bears can be found in museums; c) the first Teddy Bear was made by Mr. Mitchtom’s wife; d) President Roosevelt shot the black bear in 1902. Q: …he refused to do it and said it was unsportsmanlike… The Teddy Bears were made by Mr. Mitchtom’s wife…. They are also collected by people and many are displayed in museums… A: • Coreference • Match “The Teddy Bears” with “the first Teddy Bear” Need:

  41. “Teddy Bears” How many Teddy Bears were made by Mrs. Mitchtom and placed in the window of their shop? Q: Morris Michtom, a shopkeeper in Brooklyn, New York placed two toy bears in the window of his shop. Mr. Michtom requested permission from the President to call them “ Teddy Bears”… The Teddy Bears were made by Mr. Michtom’s wife. A: • Coreference (match the “two bears” with “The Teddy Bears”) • Name updates (“call them ‘Teddy Bears’” connects “them”=“two bears” to “Teddy Bears”.) Need:

  42. “Teddy Bears” What did Mr. Michtom do after he sold the Teddy Bears in 1902? Q: Morris Michtom, a shopkeeper in Brooklyn, New York placed two toy bears in the window of his shop. Mr. Michtom requested permission from the President to call them “ Teddy Bears”… Mr. Michtom formed a new business called the Ideal Novelty and Toy Corporation. A: • Temporal reasoning • General reasoning? • Discourse structure? Need:

  43. “Electing our President” When will the American people vote for President? Q: On Tuesday, November 7, 2000, the people of America will be heard. The presidential election will be held and voters across America will go to the polls or voting sites to cast their ballot. People vote for the candidate, or person running for President, who believes most like they do. A: • Temporal reasoning (“on Tuesday…” applies to the following sentences.) • Match “candidate, or person running for President” with “President” • Match “People” w/ “American people” (coref w/ “voters across America”?) Need:

  44. “Electing our President” The document used to cast your vote is a ____. Q: The document used to cast your vote is called a ballot. A: • Match “is” w/ “is called” Need:

  45. “Dale Earnhardt” When was his first win? Q: He won his first championship in 1980. A: • Mapping arguments between a verb and its nominalization • Coreference (in both question & passage) Need:

  46. “Abraham Lincoln” How did Lincoln die? Q: He was shot by John Wilkes Booth in 1865. A: • Match “shoot” w/ “kill” (via VerbNet or WordNet) • WN: hypernym • VerbNet: share a superclass • Match “kill” w/ “cause to die” (WN def.) • Use PropBank to line up arguments (Lincoln=arg1) Need:

  47. Passage Sources: • abcteach reading comprehension: http://www.abcteach.com/directory/reading_comprehension/ • The Quiz Zone http://www.geocities.com/RainForest/Vines/8255/begin101.htm • ETS http://www.ets.org/toefl/learners/pbt/reading.html • ARCO GMAT Test Prep http://www.west.net/~stewart/gmat/qmread.htm • Palma Sola Elementary School http://www.manatee.k12.fl.us/sites/elementary/palmasola/

More Related