Question Answering Passage Retrieval Using Dependency Parsing

1. Question Answering Passage Retrieval Using Dependency Parsing Hang CuiRenxu SunKeya LiMin-Yen KanTat-Seng ChuaDepartment of Computer ScienceNational University of Singapore Question Answering Passage Retrieval Using Dependency Parsing

2. Passage Retrieval in Question Answering QA System Document Retrieval • To narrow down the search scope • Can answer questions with more context Passage Retrieval • Lexical density based • Distance between question words Answer Extraction Question Answering Passage Retrieval Using Dependency Parsing

3. Density Based Passage Retrieval Method • However, density based can err when … <Question> What percent of the nation's cheese does Wisconsin produce? Incorrect: … the number of consumers who mention California when asked about cheese has risen by 14 percent, while the number specifying Wisconsin has dropped 16 percent. Incorrect: The wry “It's the Cheese” ads, which attribute California's allure to its cheese _ and indulge in an occasional dig at the Wisconsin stuff'' … sales of cheese in California grew three times as fast as sales in the nation as a whole 3.7 percent compared to 1.2 percent, … Incorrect: Awareness of the Real California Cheese logo, which appears on about 95 percent of California cheeses, has also made strides. Correct: In Wisconsin, where farmers produce roughly 28 percent of the nation'scheese, the outrage is palpable. Relationships between matched words differ … Question Answering Passage Retrieval Using Dependency Parsing

4. Our Solution • Examine the relationship between words • Dependency relations • Exact match of relations for answer extraction • Has low recall because same relations are often phrased differently • Fuzzy match of dependency relationship • Statistical similarity of relations Question Answering Passage Retrieval Using Dependency Parsing

5. Measuring Sentence Similarity Sim (Sent1, Sent2) = ? Sentence1 Sentence 2 Matched words Lexical matching Similarity of relations between matched words + Similarity of individual relations Question Answering Passage Retrieval Using Dependency Parsing

6. Outline • Extracting and Paring Relation Paths • Measuring Path Match Scores • Learning Relation Mapping Scores • Evaluations • Conclusions Question Answering Passage Retrieval Using Dependency Parsing

7. Outline • Extracting and Paring Relation Paths • Measuring Path Match Scores • Learning Relation Mapping Scores • Evaluations • Conclusions Question Answering Passage Retrieval Using Dependency Parsing

8. What Dependency Parsing is Like • Minipar (Lin, 1998) for dependency parsing • Dependency tree • Nodes: words/chunks in the sentence • Edges (ignoring the direction): labeled by relation types What percent of the nation's cheese does Wisconsin produce? Question Answering Passage Retrieval Using Dependency Parsing

9. Extracting Relation Paths • Relation path • Vector of relations between two nodes in the tree produce< P1: subj > Wisconsin percent< P2: prep pcomp-n > cheese • Two constraints for relation paths: • Path length (less than 7 relations) • Ignore those between two words that • are within a chunk, e.g.New York. Question Answering Passage Retrieval Using Dependency Parsing

10. Paired Paths from Question and Answer In Wisconsin, where farmers produce roughly 28 percent of the nation's cheese, the outrage is palpable. What percent of the nation's cheese does Wisconsin produce? < P1(Q) :subj > < P1(Sent) : pcomp-n mod i > Paired Relation Paths SimRel(Q, Sent) =∑i,j Sim (Pi(Q), Pj(Sent)) Question Answering Passage Retrieval Using Dependency Parsing

11. Outline • Extracting and Paring Relation Paths • Measuring Path Match Scores • Learning Relation Mapping Scores • Evaluations • Conclusions Question Answering Passage Retrieval Using Dependency Parsing

12. Measuring Path Match Degree • Employ a variation of IBM Translation Model 1 • Path match degree (similarity) as translation probability • MatchScore (PQ, PS) →Prob (PS | PQ ) • Relations as words • Why IBM Model 1? • No “word order” – bag of undirected relations • No need to estimate “target sentence length” • Relation paths are determined by the parsing tree Question Answering Passage Retrieval Using Dependency Parsing

13. Calculating Translation Probability (Similarity) of Paths Given two relation paths from the question and a candidate sentence Considering the most probable alignment (finding the most probable mapped relations) Take logarithm and ignore the constants (for all sentences, question path length is a constant) MatchScores of paths are combined to give the sentence’s relevance to the question. ? Question Answering Passage Retrieval Using Dependency Parsing

14. Outline • Extracting and Paring Relation Paths • Measuring Path Match Scores • Learning Relation Mapping Scores • Evaluations • Conclusions Question Answering Passage Retrieval Using Dependency Parsing

15. Training and Testing Testing Training • Mutual information (MI) based • Expectation Maximization (EM) based Sim ( Q, Sent ) = ? Q - A pairs Similarity between relation vectors Prob ( PSent | PQ ) = ? Paired Relation Paths Similarity between individual relations P ( Rel (Sent) | Rel (Q)) = ? Relation Mapping Scores Relation Mapping Model Question Answering Passage Retrieval Using Dependency Parsing

16. Approach 1: MI Based • Measures bipartite co-occurrences in training path pairs • Accounts for path length (penalize those long paths) • Uses frequencies to approximate mutual information Question Answering Passage Retrieval Using Dependency Parsing

17. Approach – 2: EM Based • Employ the training method from IBM Model 1 • Relation mapping scores = word translation probability • Utilize GIZA to accomplish training • Iteratively boosting the precision of relation translation probability • Initialization – assign 1 to identical relations and a small constant otherwise Question Answering Passage Retrieval Using Dependency Parsing

18. Outline • Extracting and Paring Relation Paths • Measuring Path Match Scores • Learning Relation Mapping Scores • Evaluations • Can relation matching help? • Can fuzzy match perform better than exact match? • Can long questions benefit more? • Can relation matching work on top of query expansion? • Conclusions Question Answering Passage Retrieval Using Dependency Parsing

19. Evaluation Setup • Training data • 3k corresponding path pairs from 10k QA pairs (TREC-8, 9) • Test data • 324 factoid questions from TREC-12 QA task • Passage retrieval on top 200 relevant documents by TREC Question Answering Passage Retrieval Using Dependency Parsing

20. Comparison Systems • MITRE –baseline • Stemmed word overlapping • Baseline in previous work on passage retrieval evaluation • SiteQ – top performing density based method • using 3 sentence window • NUS • Similar to SiteQ, but using sentences as passages • Strict Matching of Relations • Simulate strict matching in previous work for answer selection • Counting the number of exactly matched paths • Relation matching are applied on top of MITRE and NUS Question Answering Passage Retrieval Using Dependency Parsing

21. Evaluation Metrics • Mean reciprocal rank (MRR) • On the top 20 returned passages • Measure the mean rank position of the correct answer in the returned rank list • Percentage of questions with incorrect answers • Precision at the top one passage Question Answering Passage Retrieval Using Dependency Parsing

22. Performance Evaluation • All improvements are statistically significant (p<0.001) • MI and EM do not make much difference given our training data • EM needs more training data • MI is more susceptible to noise, so may not scale well Fuzzy matching outperforms strict matching significantly. Question Answering Passage Retrieval Using Dependency Parsing

23. Performance Variation to Question Length • Long questions, with more paired paths, tend to improve more • Using the number of non-trivial question terms to approximate question length Question Answering Passage Retrieval Using Dependency Parsing

24. Error Analysis • Mismatch of question terms • e.g.In which city is the River Seine • Introduce question analysis • Paraphrasing between the question and the answer sentence • e.g. write the book → be the author of the book • Most of current techniques fail to handle it • Finding paraphrasing via dependency parsing (Lin and Pantel) Question Answering Passage Retrieval Using Dependency Parsing

25. Performance on Top of Query Expansion • On top of query expansion, fuzzy relation matching brings a further 50% improvement • However • query expansion doesn’t help much on a fuzzy relation matching system • Expansion terms do not help in paring relation paths Rel_EM (NUS) 0.4761 Question Answering Passage Retrieval Using Dependency Parsing

26. Outline • Extracting and Paring Relation Paths • Measuring Path Match Scores • Learning Relation Mapping Scores • Evaluations • Conclusions Question Answering Passage Retrieval Using Dependency Parsing

27. Conclusions • Proposed a novel fuzzy relation matching method for factoid QA passage retrieval • Brings dramatic 70%+ improvement over the state-of-the-art systems • Brings further 50% improvement over query expansion • Future QA systems should bring in relations between words for better performance • Query expansion should be integrated to relation matching seamlessly Question Answering Passage Retrieval Using Dependency Parsing

28. Q & A Thanks! Question Answering Passage Retrieval Using Dependency Parsing