1 / 40

Vectorial Representations of Meaning for a Computational Model of Language Comprehension

Vectorial Representations of Meaning for a Computational Model of Language Comprehension. Stephen Wu University of Minnesota Thesis Defense June 23, 2010. Outline. Natural Language Understanding Structured Vectorial Semantics (SVS) Background: Semantics, Syntax

karlyn
Download Presentation

Vectorial Representations of Meaning for a Computational Model of Language Comprehension

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Vectorial Representations of Meaning for a Computational Model of Language Comprehension Stephen Wu University of Minnesota Thesis Defense June 23, 2010

  2. Outline • Natural Language Understanding • Structured Vectorial Semantics (SVS) • Background: Semantics, Syntax • Contribution: The SVS framework • 3 Instantiations • Incremental SVS • Background: Right-corner trees & parsing • Contribution: Incremental SVS • Implications of incremental SVS

  3. Outline • Natural Language Understanding • Structured Vectorial Semantics (SVS) • Background: Semantics, Syntax • Contribution: The SVS framework • 3 Instantiations • Incremental SVS • Background: Right-corner trees & parsing • Contribution: Incremental SVS • Implications of incremental SVS

  4. How do you understand language? • Incremental interpretation (Tanenhaus et al., 1995) • Interactive interpretation (Ford, Bresnan, & Kaplan, 1985) • Coherent mental representations (Grosz & Sidner, 1986) • Dynamic context (Groenendijk & Stokhof, 1991) Example (A Short Story) But when Little Red Riding Hood noticed some lovely flowers in the woods, she forgot her promise to her mother. She was enjoying the warm summer day so much that she didn’t notice... Example (A Short Story) But when Little Red Riding Hood noticed some lovely flowers in the woods, she forgot her promise to her mother. She was enjoying the warm summer day so much that she didn’t notice... a dark shadow approaching out of the forest behind her, Example (A Short Story) But when Little Red Riding Hood noticed some lovely flowers in the woods, she forgot her promise to her mother. She was enjoying the warm summer day so much that she didn’t notice... a dark shadow approaching out of the forest behind her, so they broke up rocks and dug up mounds of earth which were transported to the edge of the Bo Sea in baskets.

  5. Natural Language Processing • Many systems: Pipelined • SVS: Interactive… but factored sentence detector spelling corrector tokenizer normalizer shallow parser dictionary NE recog. POS tagger WSD Morphology Syntax Semantics Pragmatics Discourse 001110101010111011101010010100111010100011010101101010111110100010101010101010

  6. Language understanding: What for? • Cognitive/Linguistic research • Information Extraction • Temporality • Context-dependency • Relation extraction • Document/Sentence Classification • Search • Speech/HCI

  7. Outline • Natural Language Understanding • Structured Vectorial Semantics (SVS) • Background: Semantics, Syntax • Contribution: The SVS framework • 3 Instantiations • Incremental SVS • Background: Right-corner trees & parsing • Contribution: Incremental SVS • Implications of incremental SVS

  8. Semantics in Vectors: Co-occurrences • Columns = relationship • Rows = words • Dimensionality Reduction • LSA, pLSA, LDA (Hoffman, 2001) • Sparse → dense • Documents →Topics • Rows? • Distributed, quantitative representation (Kintsch, 2001) • Transpose • • • Medvedev Euro Russian President central Federation Federal regulatory overhaul independence power Immigration fight vote illegal divided reform Washington Lakers Celtics final NBA Angeles Kyrgyz .12 .14 .15 .08 .25 .17 .04 .02 .03 .01 .03 .02 .04 .06 .09 .03 .06 .03 .02 .02 .02 .07 .04 .06 1 2 1 1 3 2 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 .04 .02 .03 .11 .23 .32 .14 .03 .07 .04 .15 .02 .09 .01.06 .29 .13 .56 .02 .04 .05 .08 .05 .07 0 0 0 0 1 0 1 4 2 1 1 3 1 1 1 2 1 4 0 0 0 0 0 0 .02 .14 .06 .29 .13 .56 .03 .07 .08 .05 .07 .04 .02 .03 .04 .05 .23 .32 .11 .04 .05 .02 .09 .01 0 0 0 0 4 4 0 1 0 0 0 0 0 1 0 0 0 0 3 2 1 0 1 0 .02 .04 .03 .07 .04 .05 .02 .09.05 .08 .05 .56 .03 .11 .23 .32 .14 .06 .29 .13 .01 .07 .04 .02 0 2 2 1 2 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 3 .05 .02 .09 .01 .02 .04 .15 .08 .05 .07 .14 .02 .03 .11 .23 .32 .14 .06 .29 .13 .56 .03 .07 .04 0 0 1 0 0 0 1 3 2 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 • • • • • • • • • • • •

  9. Vector Semantic Composition Addition (e.g., Kintsch, 2001) Pointwise mult. General (Mitchell & Lapata, 2008) With syntax: • Pado & Lapata, 2007 • Erk & Pado, 2008 • Mitchell & Lapata, 2009 syntax knowledge base anything!

  10. Finding (syntactic) Structure: Parsing Grammar (partial) P( ) = 0.8 P( ) = 0.6 P( ) = 0.7 P( ) = 0.3 P( ) = 0.1 P( ) = 0.4 P( ) = 0.3 P( ) = 0.01 P( ) = 0.2 P( ) = 0.1 P( ) = 0.4 P( ) = 0.02 P( ) = 0.08 Grammar S  NP VP NP  DT NN VP  VBD NP VBD  VBD PRT NN  NN NN DT  the DT  an NN  engineers VBD  pulled PRT  off P  off NN  engineering NN trick Tree Annotations: • headwords (Charniak, 1997) • latent annot. (Matsuzaki et al., 2005) • formal semantics (Ge and Mooney, 2005)

  11. StructuredVectorialSemantics:Overview • Vectors in syntactic context • Composition & Parsing • Semantic spaces: • Headword-lexicalization SVS • Relational-clustering SVS • Logical-interpretation SVS • Instantiate with: Interactive

  12. Extending Parsing to Semantics Factoring Syn + Sem Parsing Semantic referents, e Semantic relations, l

  13. Vectors in Syntactic Context Factoring Syn + Sem Parsing S Semantic referents, e Semantic relations, l NP VP NP DT NN VBD the engineers pulled NN DT trick a

  14. Vector Composition Vector Composition… and Parsing! diagonal-listing matrix in, vector out syntax left-child syn+sem right-child syn+sem Interactive syntax and semantics!

  15. Instantiate: {Headwords, Clusters} Headword lexicalization Relational clustering EM Clustering

  16. Inside-Outside Algorithm (EM) E-step: • Estimates  annot. rule • Weight against real data M-step: • Estimate latent grammar rules • Imagine annot. • Frequency count Parent , Sibling , , or Child , ,

  17. Relational Clustering SVS • 5 SVS implementation equations estimated in EM estimated in EM backed off from EM byproduct of EM byproduct of EM

  18. EM-learned Relational Clusters • 1000 headwords → 10 referent concepts • in syntactic context (plural nouns)

  19. EM-learned Relational Clusters • 1000 headwords → 10 referent concepts • in context (transitive past-tense verbs)

  20. Eval: Parsing Accuracy vs. Clusters • Do semantics help parsing? • Are more clusters better?

  21. Eval: Speed with Vectors • Rich dependencies • With & w/o vectorization • O(n3) runtime • Coefficients? • 0.66505 un-vectorized • 0.00267 vectorized • Efficient operations 10e md-rlnclust

  22. Logical Interpretation (sets) World model Semantic Vectors/Matrices Some special matrices for relations e.g., ALL, NEG, HALF

  23. Logical Interpretation SVS Cannot assume n = 1 “Find the square containing all non-squares”

  24. Summary of SVS • Cognitive model • Interactive syntax and semantics • Vectorial representations • Practical model • Headword-lexicalization SVS • Relational-clustering SVS • Logical-interpretation SVS

  25. Outline • Natural Language Understanding • Structured Vectorial Semantics (SVS) • Background: Semantics, Syntax • Contribution: The SVS framework • 3 Instantiations • Incremental SVS • Background: Right-corner trees & parsing • Contribution: Incremental SVS • Implications of incremental SVS

  26. Manner of Interpretation • Interactive SVS syn+sem • Incremental (not bottom-up!) • Bounded short-term memory • center-embedding (Chomsky & Miller, 1965) • dispreferred (Grice, 1975) “manner” • Bound/Incrementalize SVS: • Right-corner transform • Hierarchic HMM (HHMM) parsing The drug the intern the nurse supervised administered the intern cured the patient. the drug [ [ ] ]

  27. Right-corner transform • Incremental by nature • Flatter structure – unary rules • Trunks and memory

  28. HHMM Parsing of right-corner trees • 1 word per time • 1 depth per trunk (+ bounded) • Many hypotheses

  29. HHMM Parsing of right-corner trees CEE cross-element expansion CER cross-element reduction AWT awaited transition IER in-element reduction ACT active transition (IEE)

  30. HHMM Parsing of right-corner trees CEE cross-element expansion CER cross-element reduction AWT awaited transition IER in-element reduction ACT active transition (IEE)

  31. HHMM Parsing Equations • Same output trees • Approximations cancel

  32. Incremental SVS Parsing Equations • Semantics • Factorization • Vectorization

  33. Characteristics of Incremental SVS Incremental SVS Distributed semantics Interactive interpretation Incremental interpretation Bounded memory 5 prob. models →5 parser ops. SVS Distributed semantics Interactive interpretation Input: 5 prob. models RC-HHMM Incremental interpretation Bounded memory 5 parser operations

  34. Implications (of Incr. SVS) • Bottom-up • Incremental (AWT case) • n = 1 ok • General n→more memory

  35. Implications (of Incr. SVS)

  36. Quantifier distribution in WSJ RC transform center-embedding memory cost Incremental SVS non-exist. quant. memory cost Gricean manner less frequency more memory cost

  37. Summary of Thesis • Structured Vectorial Semantics • Full parsing (syntax) • Vector composition (semantics) • Instantiations • Incremental SVS • Interactive (SVS) • Incremental (HHMM) • Memory-bounded (Right-corner transform) • Implications

  38. Future Work • Domains • Medical • Other languages • Language modeling • Semantic role labeling • Episodic memory • Coreference resolution • Text statistics and visualization

  39. Acknowledgements • Prof. William Schuler • NLP lab: • Tim Miller • Lane Schwartz • Andy Exley • Dingcheng Li • Luan Nguyen • Committee • Prof. Gini • Prof. Boley • Prof. Fletcher • Dr. Savova • You!

  40. The end.

More Related