Is Question Answering an Acquired Skill?

2. Is Question Answeringan Acquired Skill? Soumen Chakrabarti G. RamakrishnanD. ParanjpeP. Bhattacharyya IIT Bombay

3. Web search and QA • Information need – words relating “things” + “thing” aliases = telegraphic Web queries • Cheapest laptop with wireless  best price laptop 802.11 • Why is the sky blue?  sky blue reason • When was the Space Needle built? “Space Needle” history • Entity and relation extraction technology better than ever (SemTag, KnowItAll) • Ontology extension (e.g., is a kind of) • List extraction (e.g., is an instance of) • Slot-filling (author X wrote book Y) Chakrabarti

4. Factoid QA • Specialize given domain to a token related to ground constants in the query • What animal is Winnie the Pooh? • hyponym(“animal”) NEAR “Winnie the Pooh” • When was television invented? • instance-of(“time”) NEAR “television” NEAR synonym(“invented”) • Three kinds of useful question tokens • Appear unchanged in passage (selector) • Specialize to answer tokens (atype) • Improve belief in answer via synonymy etc. Chakrabarti

5. A new relational view of QA Question Atypeclues Attributeor columnname Selectors Locate whichcolumn to read Directsyntacticmatch Entity class IS-A Limit searchto certain rows “Landingzone” Questionwords Answerpassage “Landing zone” • Entity class or atype may be expressed by • A finite IS-A hierarchy (e.g. WordNet, TAP) • A surface pattern matching infinitely many strings (e.g. “digit+”, “Xx+”, “preceded by a preposition”) • Match selectors, specialize atype to answer tokens Chakrabarti

6. Benefits of the relational view • “Scaling up by dumbing down” • Next stop after vector-space • Far short of real knowledge representation and inference • Barely getting practical at (near) Web scale • Can set up as a learning problem: train with questions and answers embedded in passage context • Transparent, self-tuning, easy to deploy • Feature extractors used in entity taggers • Relational/graphical learning on features Chakrabarti

7. Subproblems • Identify atype clues • Easy: who, when, where, how many, how tall… • Harder: What…, which…, name… • Map atype clues to likely entity classes • Data- and task-driven question classification • Train quickly on new corpus and QA samples • Identify selectors for keyword query • Based on question context and global stats • Get candidate passages from IR system • Re-rank candidate passages Chakrabarti

8. Mapping “self-evident” atypes • Whoperson, whentime, whereplace • Not always trivial: how_many vs. when • Question classification + handcrafted map • Needs task knowledge and skilled effort • Laborious to move to new corpus, language… • Task-driven information extraction • Enough info in training QA pairs to learn map • Map clue to a generalization of the answer • Surface patterns: hasDigit, [in] DDDD, NNP, CD • WordNet-based: region#n#3, quantity#n#1 Chakrabarti

9. Mapping examples how who abstraction#n#6NNS NNP, person fast far many rich wrote first rate#n#2 explorer mile#n#3linear_unit#n#1 paper_money#n#1 currency#n#1 WordNet writer, composer,artist, musician measure#n#3definite_quantity#n#1 rate#n#2magnitude_relation#n#1 A cheetah can chase its preyat up to 90 km/h Nothing moves faster than186,000 miles per hour, thespeed of light How fast can a cheetah run? How fast does light travel? Chakrabarti

10. What…, which…, name… atype clues • Assumption: Question sentence has a wh-word and a main/auxiliary verb • Observation: Atype clues are embedded in a noun phrase (NP) adjoining the main or auxiliary verb • Heuristic: Atype clue = head of this NP • Use a shallow parser and apply rule • Head can have attributes • Which (American(general)) is buried in Salzburg? • Name (Saturn’s (largest (moon))) Chakrabarti

11. Atype clue extraction stats • Simple heuristic surprisingly effective • If successful, extracted atype is mapped to WordNet synset (mooncelestial body etc.) • If no atype of this form available, try the “self-evident” atypes (who, when, where, how_X etc.) Chakrabarti

12. Learning selectors • Which question words are likely to appear (almost) unchanged in an answer passage? • Constants in select-clauses of SQL queries • Guides backoff policy for keyword query • Local and global features • POS of word, POS of adjacent words, case info, proximity to wh-word • Suppose word is associated with synset set S • NumSense: size of S (how polysemous is the word?) • NumLemma: average #lemmas describing sS POS@-1 POS@0 POS@1 Chakrabarti

13. Selector results • Decision trees better than logistic regression • F1=81% as against LR F1=75% • Intuitive decision branches • But logistic regression gives scores for query backoff • Global features (IDF, NumSense, NumLemma) essential for accuracy • Best F1 accuracy with local features alone: 71—73% • With local and global features: 81% Chakrabarti

14. Putting together a QA system Learning tools TrainingCorpus Shallow parser Wordnet QASystem POSTagger N-E Tagger Chakrabarti

15. Noun andverb markers Taggedquestion Tokenizer POS Tagger ShallowParser AtypeExtractor Atype clues SelectorLearner • Learning to rerank passages • Sample features: • Do selectors match? How many? • Is some non-selector passage token a specialization of the question’s atype clue? • Min, avg linear token distance between candidate token and matched selectors Is QA pair? Taggedpassage Tokenizer POS TaggerEntity Extractor LogisticRegression Rerankedpassages Putting together a QA system Question Keyword querygenerator Keyword query PassageIndex Candidatepassage Sentence splitterPassage indexer Corpus Chakrabarti

16. Surface pattern hasDigits selector match WordNet match 5 tokens apart 1 Learning to re-rank passages • Remove passage tokens matching selectors • User already knows these are in passage • Find passage token/s specializing atype • For each candidate token collect • Atype of question, original rank of passage • Min, avg linear distances to matched selectors • POS and entity tag of token if available How many inhabitants live in the town of Ushuaia Ushuaia, a port of about 30,000 dwellers set between the Beagle Channel and … Chakrabarti

17. Effect of re-ranking results • Categorical andnumeric attributes • Logistic regression • Good precision,poor recall • Use logit score tore-rank passages • Rank of first correctpassage shifts substantially Log scale Chakrabarti

18. Mean reciprocal rank studies • nq = smallest rank among answer passages • Re-ranking reduces nqdrastically • MRR = (1/|Q |) qQ(1/nq) • Substantial gain in MRR • TREC 2000 top MRRs:0.76 0.71 0.46 0.46 0.31 Chakrabarti

19. Generalization across corpora • Across-year numbers close to train/test split on a single year • Features and model seem to capture corpus-independent linguistic Q+A artifacts Chakrabarti

20. Re-ranking benefits by question type • All question types benefit from re-ranking • Benefits differ by question type • Large benefits for “what” and “which” questions, thanks to WordNet • Without WordNet customization Chakrabarti

21. Conclusion • A clean-room view of QA as feature extraction plus learning • Recover structure info from question • Learn correlations between question structure and passage features • Competitive accuracy with negligible domain expertise or manual intervention • Ongoing work • Use redundancy available from the Web • Model how selector and atype are related • Treat all question types uniformly Chakrabarti