1 / 20

Toward Dependency Path based Entailment

Toward Dependency Path based Entailment. Rodney Nielsen, Wayne Ward, and James Martin. Why Entailment. Intelligent Tutoring Systems Student Interaction Analysis Are all aspects of the student’s answer entailed by the text and the gold standard answer

watson
Download Presentation

Toward Dependency Path based Entailment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Toward Dependency Path based Entailment Rodney Nielsen, Wayne Ward, and James Martin

  2. Why Entailment • Intelligent Tutoring Systems • Student Interaction Analysis • Are all aspects of the student’s answer entailed by the text and the gold standard answer • Are all aspects of the desired answer entailed by the student’s response

  3. Dependency Path-based Entailment • DIRT (Lin and Pantel, 2001) • Unsupervised method to discover inference rules • “X is author of Y ≈ X wrote Y” • “X solved Y ≈ X found a solution to Y” • Based on Harris’ Distributional Hypothesis • words occurring in the same contexts tend to be similar • If two dependency paths tend to link the same sets of words, they hypothesize that their meanings are similar

  4. ML Classification Approach Dependency Path Based Entailment • Features derived from corpus statistics • Unigram co-occurrence • Surface form bigram co-occurrence • Dependency-derived bigram co-occurrence • Mixture of experts: • About 18 ML classifiers from Weka toolkit • Classify by majority vote or average probability Bag of Words Graph Matching

  5. Corpora • 7.4M articles, 2.5B words, 347 words/doc • Gigaword (Graff, 2003) – 77% of documents • Reuters Corpus (Lewis et al., 2004) • TIPSTER • Lucene IR engine • Two indices • Word surface form • Porter stem filter • Stop words = {a, an, the}

  6. Word Alignment Features • Unigram word alignment

  7. Core Features

  8. Word Alignment Features • Bigram word alignment • Example: • <t>Newspapers choke on rising paper costs and falling revenue.</t><h>The cost of paper is rising.</h> • MLE(cost, t) = ncost of, costs of /ncosts of = 6086/35800 = 0.17

  9. Hypothesis h Text t rising choke cost is Newspapers on The of costs paper rising paper and revenues falling Dependency Features • Dependency bigram features

  10. Hypothesis h Text t rising choke cost is Newspapers on The of costs paper rising paper and revenues falling Dependency Features • Descendent relation statistics

  11. Hypothesis h Text t rising choke cost is Newspapers on of The costs paper rising paper and revenues falling Dependency Features • Descendent relation statistics

  12. Hypothesis h Text t rising choke cost is Newspapers on The of costs paper rising paper and revenues falling Dependency Features • Descendent relation statistics

  13. Hypothesis h Text t rising choke cost is Newspapers on The of costs paper rising paper and revenues falling Dependency Features • Descendent relation statistics

  14. Hypothesis h Text t rising choke cost is Newspapers on The of costs paper rising paper and revenues falling Verb Dependency Features • Combined verb descendent relation features • Worst verb descendent relation features

  15. Hypothesis h Text t rising choke cost is Newspapers on The of costs paper rising paper and revenues falling SubjectDependencyFeatures • Combined and worst subject descendent relations • Combined and worst subject-to-verb paths

  16. Other Dependency Features • Repeat these same features for: • Object • pcomp-n • Other descendent relations

  17. Results

  18. Feature Analysis • All feature sets are contributing according to cross validation on the training set • Most significant feature set: • Unigram stem based word alignment • Most significant core repeated feature: • Average Probability

  19. Conclusions • While our current dependency path features are only a step in the direction of our proposed inference system, they provided a significant improvement over the best results from the first PASCAL Recognizing Textual Entailment challenge (RTE1) • Our system (after fixing a couple of bugs) ranked 6th in accuracy and 4th in average precision out of 23 entrants at this year’s RTE2 challenge • We believe our proposed system will provide an effective foundation for the detailed assessment of students’ responses to an intelligent tutor

  20. choke Newspapers on rising costs cost is rising paper and revenues The of falling paper Questions Dependency Path Based Entailment • Mixture of experts classifier using corpus co-occurrence statistics • Moving in the direction of DIRT • Domain of Interest: Student response analysis in intelligent tutoring systems Bag of Words Graph Matching Hypothesis h Text t

More Related