1 / 21

Question Answering

Question Answering. Off-line answer extraction / Jori Mur. Overview. Using patterns to extract answers off-line High precision, low recall Simple patterns not robust enough Answer not in one sentence No binary relation in question Hard to come up with new patterns. Joost.

cruz-bridges
Download Presentation

Question Answering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Question Answering • Off-line answer extraction / Jori Mur

  2. Overview Using patterns to extract answers off-line High precision, low recall Simple patterns not robust enough Answer not in one sentence No binary relation in question Hard to come up with new patterns

  3. Joost

  4. Off-line answer extraction using patterns /[Noun] [Name]/... premier Balkenende ... /[Name], [Noun]/... Balkenende, premier ... /[Name] [Verb] as [Noun]/... Balkende heeft als premier ... /[Name] [be] [Det] [Noun]/... Balkenende is de premier ... ...

  5. High precision, low recall Tjong Kim Sang, Bouma en de Rijke. Developing Offline Strategies for Answering Medical Questions, 2005 Bernardi, Jijkoun, Mishne en de Rijke.Selectively Using Linguistic Resources Throughout the Question Answering Pipeline,2003

  6. Investigating experiment Joost 99 function questions from CLEF2003 and CLEF2004 Who was the author of Die Gotterdämmerung? Of which company is Christian Blanc president? 15 patterns /[ADJ][NOUN][NAME]/Franse president Sarkozy /[NAME], [NOUN] van [COUNTRY]/Sarkozy, president van Frankrijk

  7. Results Extraction QA

  8. Error analysis: 46 No answer found In which company is Christian Blanc president? Christian Blanc, president of the rather destitute French aviation company Air France • Who is the head of state of Australia? • The prime-minister of Australia explained his plan to transfer the role of the head of state from the British queen Elizabeth to (...) • Who is the daughter of Deng Xiaopeng? • Will Deng Xiaoping make it until the Chinese New Year? (...) when Deng's youngest daughter, Deng Rong, recently said (...)

  9. Error analysis: 18 Incorrect Who is the German minister of Economy? Who was the president of the US during the second World War?

  10. 4 Challenges Simple patterns are not robust enough Not easy to think of new patterns Answer not in one sentence Not always binary relations

  11. 1. Simple patterns are not robust enough ... George Bush, yesterday he became 61, said today as 43rd president of the US ... Subj Predm Obj Mod Obj ... George Bush said as president of the US ... Obj Mod Obj Predm Subj • <said, subj, George Bush> • <said, predm, as> • <as, obj, president> • <president, mod, of> • <of, obj, US>

  12. 2. Answer not in one sentence Balkenende was happy with the result, allthough he did not win the elections.

  13. 2. Evaluation and Results Coreference MUC score : |S|-|p(S)| |S| -1 Recall: 45.6% Precision: 67.9% Extraction Without coreference With coreference 127.711 facts 146.210 facts QA Without coreference With coreference 153/233 (65.7%) 159/233 (68.2%) Question 20050107: Wie was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde? ‘Who was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope?’. The answer in the AD of September 19th, 1995: Bowersox was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde. ‘Bowersox was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope’.

  14. 3. Not always binary relations When was Jimi Hendrix born?Jimi Hendrix – 1942 Who is the president of Amerika?Bush – president – Amerika Who was the president of Amerika in 1944?Roosevelt – president – Amerika – 1944 Who succeeded president Roosevelt as president of Amerika in the second world war?Truman – president – Amerika – WO II Truman – successor – Roosevelt

  15. 3. Dependency relation overlap Q: Who succeeded president Rooseveltas president of Amerika in the second world war?A1: George Bush, president of Amerika said that ...A2: At the end of the second world war Harry S. Truman succeeded president Roosevelt as president of Amerika. A1: <president, mod, of> <of,obj,Amerika> A2: <succeed, subj, X> <of, obj, Amerika> <succeed, obj, president Roosevelt> <world war, det, the> <succeed, predm, as> <world war, mod, second> <as, obj, president> <president, mod, of>

  16. 4. Hard to transfer to new question type Learn patterns in bootstrapping way1. Start with 10 seed-pairs2. Find dependency path between two terms3. Replace terms by variables to create pattern4. Use patterns to find new facts5. Use new facts to find new patternsFind balance precision/recall3a. Filter patterns

  17. 4. Calculate precision patterns Replace only the answer term with a variable. P = Ca/Co Co : the total number of times the pattern matched a phrase in the corpus Ca : the total number of times the pattern matched a phrase in the corpus containing the correct answer term

  18. 4. Experiment and Results learning Stop at precision of table < 0.5 or more than 100 patterns 24 capital question from Clef, 3 turned out NIL

  19. 4. Nice examples of found patterns ... is na H de grootste stad van L...Petersburg is na Moskou de grootste stad van Rusland ... de L-adj tenniskampioenschappen in H...de open Franse tenniskampioenschappen in Parijs ... de L-adj president opent in H...De Tjechische president Havel opent op 14 juni in Praag ... de L-adj ministers komen bijeen in H...Volgende maand komen de Europese minister van industrie opnieuw bijeen in Brussel

  20. Summary Qatar – off-line answer extraction module in JoostHigh precision, low recall Dependency patterns Coreference resolution Score for dependency overlap Learning patterns

  21. Questions?

More Related