Question Answering From Zero to Hero

1. Question AnsweringFrom Zero to Hero Elena Eneva 11 Oct 2001 Advanced IR Seminar

2. Sources • TREC-9. 2001. • http://la.lti.cs.cmu.edu/Javelin • E. Voorhees. "The Overview of the TREC-9 Question Answering track." • J. Prager, E. Brown, A. Coden and D. Radev. "Question answering by predictive annotation." SIGIR '00. • C.L.A. Clarke, G.V. Cormack and T.R. Lynam. "Exploiting redundancy in question answering." In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 2001. V P C

3. Question Answering • IR • Successful in large scale text search problems • Retrieve full documents • IE • Successful in extracting very precise answers from text • Work on pre-specified domains • Combining the strengths

4. QA track in TREC • Collection of unstructured documents (table 1 in V) • Short factual questions in English (Why can't ostriches fly ? Where did Bill Gates go to college ?) also figure 1 in V • Return answer as a ranked list of 5 fragments of documents (2 categories: 50 and 250 bytes)

5. Evaluation • By people • Reciprocal rank of first correct answer or 0 • % answers which were found • Strict and Lenient scores (supported and unsupported judgment) • Short and Long version

6. 2 QA TREK systems • Question Answering by Predictive Annotation - Prager, Brown, Coden (IBM) and Radev (U of Michigan) • Exploiting Redundancy in Question Answering - Clarke, Cormack, Lynam (U of Waterloo) • Ranking - Table 2 in V

7. Exploiting Redundancy in Question Answering Figure 1 in C Question -> a query for submission to a passage retrieval component -> a set of selection rules what guides the process of extracting answers from the passages (answer category) Get a list of k passages Identify possible answers Rank the possible answers Question analysis – IR – IE

8. 3 features with greatest contribution • Flexibility of the parser • Passage retrieval technique (high quality passages) • Redundancy in the answer selection component – contribution of evidence from multiple passages to identify the most likely answer

9. Passage Retrieval techniques • Each document D is an ordered sequence of terms D= d1 d2 d3 … dm • Extent (u, v) (minimal) • Query Q generated from the question Q={q1, q2, q3, …} • Compute the score for an extent(u, v) for which TQ is a cover • Higher scores to passages whose P of occurrence is lower

10. Redundancy • Each candidate term t is is assigned a weight that takes into account the number of distinct passages in which the term appears, as well as the relative frequency of the term in the database • Wt = Ct log (N/ft) • Ct is the number of distinct passages in which t appears • Summing the weights of a all terms in a candidate answer • Determine the first one, reduce weights to 0, do all over until have 5 • Figure 2 in C

11. Exploiting redundancy • “Who” questions • 100 GB corpus • K depth, W width • Figure 2 in C

12. Who wants to be a Millionaire? • Real life example • 70% correct overall • Figure 5 in C

13. Question answering by predictive annotation • IBM system • Shallow NLP • System structure Figure 1 in P • Annotation • Indexing