Automatic Answer Validation in Open -Domain Question Answering

1. Automatic Answer Validation in Open-Domain Question Answering Hristo Tanev TCC,ITC - IRST

2. Open Domain Question Answering Which is the capital of Italy? • Automatic extracting of the answer of a natural language question • Related fields: • Information Extraction • Information Retrieval • Deeper text analysis ROME

3. How it works

4. Question type:Which-LOCATION Keywords: capital Italy Which is the capital of Italy? Question processing

5. Question type:Which-LOCATION Keywords: capital Italy Document collection IR engine Selected documents/paragraphs

6. Candidate answers Selected documents/paragraphs Question type:Which-LOCATION Keywords: capital Italy Paris, Milan, Rome, Texas Answer extraction

7. Candidate answers Paris - 1, Milan - 1, Rome - 2 Answer evaluation and validation Knowledge bases, Abduction

8. ROME!

9. The complexity of the QA task • The variety of question classes • The infinite number of answer formulations • Anaphora, ellipsis, synonymy • Sometimes syntactic and semantic analysis are necessary, also world knowledge

10. Answer inference • The problem: How to infer if a candidate answer is relevant with respect to the question? • Filtering out the irrelevant answer candidates • Score the candidate answers according to their relevance

11. Contemporary approaches for answer inference • Deducing the question logical form (QLF) from the text logical form Sanda M. Harabagiu and Marius Pasca and Steven Maiorano “Experiments with Open-Domain Textual Question Answering”, COLING 2000,292-298 Example: Q: Why did David Koresh ask FBI for a word processor? A: Mr. Koresh sent a request for word processor to FBI to enable him to write his revelations QLF: ask(Koresh, FBI, word processor,reason=?) ALF: sent request(Koresh, FBI, word processor,reason: to write his revelations) Heuristic: send request => ask,ALF => QLF

12. Contemporary approaches for answer inference (continued) • Abduction, using pragmatic axioms, and semantic representation Sanda Harabagiu , Steven Maiorano “Finding Answers in Large Collections of Texts: Paragraph Indexing + Abductive Inference” • action1 (e1, Person1) & action1 (e2, Person2) & related_events(e1, e2) => related(Person1, Person2) • Q:Who was Lincoln’s Secretary of State • A:Booth schemed to kill Lincoln, while his compatriots would murder Vice President Andrew Johnson and Secretary of State William Seward. • kill(e1, Lincoln) & murder(e2, Secretary of State William Seward) & related(e1 , e2) =>related(Lincoln, Secretary of State William Seward)

13. Contemporary approaches for answer inference (continued) • Lexico – syntactic patterns Q: What forms of international crime exist? A:…international forms of crime, includingterrorism, blackmailanddrug-related problems. • These kinds of patterns are appropriate for certain type of questions, asking for taxonomic information

14. Contemporary approaches for answer inference - disadvantages • A very large open domain knowledge base is requisite • The creation of knowledge bases is very expensive in time and resources • The present world knowledge bases (such as WordNet or ThoughtTreasure) are far away from being comprehensive • The question and its answer can be very different lexically, this poses the necessity from deep semantic analysis to infer the relation between the discourse entities

15. Data Driven Answer Inference

16. The simple approach – ask the oracle • The database should be large enough to encode a great part of the human knowledge • It should provide the necessary redundancy to contain different reformulationsfor the facts • It should be changed dynamicallyto reflect recent state of thehuman knowledge about the world • It should be easily accessible Rome is the capital of Italy

17. World Wide Web as a source of knowledge • Comprehensive • Open domain nature • Constantly updated and expanded • Search indices and engines • Implicit knowledge • My journey in Italy began in the capital Rome… • Disadvantages: • Knowledge is in unstructured text form • Access to the search engines may be slow

18. Web as a gigantic corpus • Parameters: • 100.000.000 hosts • AltaVista indexes over 1.000.000.000 Web pages • Google 2.000.000.000 Web pages • 86% English language pages, 5.8% German, 2.36 French, 1.6% Italian • Accessibility Different public accessible search engines: AltaVista, Fast, Google, Excite, Lycos, Yahoo!, Northern Light

19. Validation Statements Question Candidate Answer • Who is Galileo? astronomer • Galileo is an astronomer • Which is the capital of Italy? Rome • Rome is the capital of Italy • Why the moon turns orange? because it enters the Earth shadow • The moon turns orange because it enters the Earth shadow

20. Validation Statements (continued) The core of the data-driven answer validation is searching on-line texts, similar to the validation statement for a question-answer pair

21. The Answer Validation Algorithm • Question + Answer = Validation Pattern • Q: How far is it from Denver to Aspen • A: 200 miles • QAP : [Denver … Aspen … 200 miles] • Submit validation pattern to search engine • Infer the power of the relation betweenQuestion & Answeron the basis of the search engine result

22. An Example • QA pair: Who is Galileo? astronomer • Submit to AltaVista the query “Galileo” • AltaVista returns 2000 hits • Submit to AltaVista the query “astronomer” • AltaVista returns 10000 hits • Submit to AltaVista the query Galileo NEAR astronomer • AltaVista returns 1000 hits • PMI(Galileo, astronomer) = 14 > threshold

23. Validation Patterns The validation pattern is the base of the query which is submitted to the search engine to check if the question and the answer tend to appear together

24. Word Level Validation Patterns • Qk1, Qk2,…. The question keywords • A The Answer • The query to the search engine is formed by linking the question keywords and answer with operators like AND or NEAR • Qk1 NEAR Qk2 NEAR …NEAR A • Qk1 AND Qk2 AND …AND A • (Qk1 AND Qk2 …) NEAR A • This way co-occurrence between the question and answer keywords is searched in Intenet

25. Phrase Level Validation Patterns • Validation pattern is composed by syntactic phrases instead of separate keywords • Example: • Q: What city had a world fair in 1900? • A: Paris • Query: ( city NEAR “world fair” NEAR “in 1900” )NEAR Paris • Pages found by these type of patterns are more likely to contain texts confirming the answer corectness • Disadvantage: less probable, often obtain 0 hits even for the right answer

26. Phrase Level Validation Patterns (continued) • The phrases may be extracted by parser from the question • More probable and coherent phrases should be preferred over the rare and non coherent phrases • The phrase frequency may be measured using Web as a corpus

27. Sentence Level Patterns • If the question and the answer are short, the whole validation statement can be submitted to the search engine • “When did Hawaii become a state?” – 1959 • “Hawaii became a state in 1959” • Linguistic transformations are necessary to transform the QA pair in a validation statement

28. Morphological Variations and Symonymy in Patterns • The question and answer keywords may occur in different morphological forms • Synonyms can also appear instead of the original keywords • Most search engines (Google, AltaVista, Yahoo) allow the use of keyword variants by OR operator • Q: What date did John Lenondie? • Question pattern: John NEAR Lenon NEAR (die OR died)

29. Types of data driven answer inference • Pure quantitative approach: only the number of hits, returned by the search engine are considered. Statistical techniques form the core of this class of approaches • Qualitative approaches: the document content is processed

30. Statistical answer validation • By search engine queries are obtained the frequencies of the question pattern, the answer and the question-answer validation pattern • Example Question: How far is it from Denver to Aspen? Question Pattern : far NEAR Denver NEAR Aspen Answer : 200 miles QAP: far NEAR Denver NEAR Aspen NEAR 200 miles • Search engine: • Frequency(Question Pattern) • Frequency(Answer) • Frequency(QAP)

31. Statistical answer validation • Using the frequencies and the number approximating the pages indexed by the search engine are calculated the following probabilities for occurrence in Web: P(Question Pattern), P(Answer), P(Question-Answer co-occurrence)

32. Statistical answer validation • Thus calculated probabilities are combined in formulae, which are derived from classical co-occurrence formulae. • The difference from the classical co-occurrence task is that we search how the appearance of the question pattern implies the appearance of the answer. Thus non symmetrical formulae are necessary. • These formulae return a value, which is an indication for the answer corectness with respect to the question.

33. Statistical answer validation Answer validation formulae

34. Qualitative Approach • The qualitative answer validation considers the content of the obtained documents as a result of the validation pattern submition to the search engine • The distance between the question and answer keywords is considered

35. Qualitative Approach • The use of document snippets can speed up this approach • Certain search engines, like Yahoo! and Google return text snippets from the documents, where the keywords appear

36. Qualitative Approach. Extraction of data from the snippets. Q: Who is the first man to fly across the Pacific Ocean? A: Pangborn Query, submitted to Google:first AND man AND fly OR flew AND Pacific AND Ocean AND Pangborn Text snippets returned: “Pangbornbecame the firstpilot to crossPacific” “Pangbornwith co-pilot Hew Herndon flewacross Pacific”

37. Qualitative Approach. Extraction of data from the snippets (continued). Obtained co-occurrence relations: (Pangborn, first, Pacific) (Pangborn, fly, Pacific) Numerical values, obtained from the relations: Proportion of question keywords, related to answer (0.6 in the example , 3 question keywords (first, Pacific, fly) related to answer from total of 5 question keywords) Number of different relations and their length

38. Qualitative Approach. Calculating answer relevance • Only the different co-occurrence relations are considered, co-occurrences, which are included in others are excluded PQK percent of question keywords, related to the answer r relations, obtained for the answer from the snippets length(r) the number of words in the co-occurrence relation r

39. Qualitative Approach. Calculating answer relevance (continued) • Keyword density in the co-occurrence relations may also be considered • The formula may be the sum of the keyword densities for all the relations

40. Combining approaches • The qualitative approach can be used to extract co-occurrences • Statistical techniques can be used to evaluate these co-ocurrences

41. Experiments and results

42. Experiment • The statistical approach was tested • The TREC10 question-answer list has been used, provided by NIST • for total of 492 questions maximum three right and three wrong answers are taken • Two experiments were carried out Performance of the system on the full set of questions Named entities questions • A baseline model was introduced

43. Experiment (continued) • For every 50 byte answer the algorithm extracts only the entities that correspond to the question type • The pairs question – answer were evaluated using AltaVista • Phrase level patterns and two types of word-level patterns has been used

44. Experiment. The Patterns. • Three types of patterns: • Phrase,Word level with NEAR,Word level with AND • Example: • Q: “What city had a world fair in 1900?” • A: Paris • Phrase pattern: • (cityNEAR “worldfair” NEAR 1900)NEAR Paris • Word level with NEAR: • (cityNEARworldNEAR fair NEAR 1900)NEAR Paris • World level with AND: • (cityANDworldAND fair AND 1900)NEAR Paris

45. Results

46. Future Directions

47. Much more to do… • Improvement of the statistical formulae • Research on the search engine use • Combining the qualitative and statistical approach • Creation of reliable validation patterns • Introducing new techniques for answer validation • Integration in QA system

48. END