1 / 114

Part III Hierarchical Bayesian Models

Part III Hierarchical Bayesian Models. Universal Grammar. Hierarchical phrase structure grammars (e.g., CFG, HPSG, TAG). Grammar. Phrase structure. Utterance. Speech signal. Vision. (Han and Zhu, 2006). Word learning. Whole-object principle Shape bias Taxonomic principle

lev
Download Presentation

Part III Hierarchical Bayesian Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Part IIIHierarchical Bayesian Models

  2. Universal Grammar Hierarchical phrase structure grammars (e.g., CFG, HPSG, TAG) Grammar Phrase structure Utterance Speech signal

  3. Vision (Han and Zhu, 2006)

  4. Word learning Whole-object principle Shape bias Taxonomic principle Contrast principle Basic-level bias Principles Structure Data

  5. Hierarchical Bayesian models • Can represent and reason about knowledge at multiple levels of abstraction. • Have been used by statisticians for many years.

  6. Hierarchical Bayesian models • Can represent and reason about knowledge at multiple levels of abstraction. • Have been used by statisticians for many years. • Have been applied to many cognitive problems: • causal reasoning (Mansinghka et al, 06) • language (Chater and Manning, 06) • vision (Fei-Fei, Fergus, Perona, 03) • word learning (Kemp, Perfors, Tenenbaum,06) • decision making (Lee, 06)

  7. Outline • A high-level view of HBMs • A case study • Semantic knowledge

  8. Universal Grammar Hierarchical phrase structure grammars (e.g., CFG, HPSG, TAG) P(grammar | UG) Grammar P(phrase structure | grammar) Phrase structure P(utterance | phrase structure) Utterance P(speech | utterance) Speech signal

  9. Hierarchical Bayesian model U Universal Grammar P(G|U) G Grammar P(s|G) s1 s2 s3 s4 s5 s6 Phrase structure P(u|s) u1 u2 u3 u4 u5 u6 Utterance

  10. Hierarchical Bayesian model U Universal Grammar P(G|U) G Grammar P(s|G) s1 s2 s3 s4 s5 s6 Phrase structure P(u|s) u1 u2 u3 u4 u5 u6 Utterance A hierarchical Bayesian model specifies a joint distribution over all variables in the hierarchy:P({ui}, {si}, G | U) = P ({ui} | {si}) P({si} | G) P(G|U)

  11. Knowledge at multiple levels • Top-down inferences: • How does abstract knowledge guide inferences at lower levels? • Bottom-up inferences: • How can abstract knowledge be acquired? • Simultaneous learning at multiple levels of abstraction

  12. Top-down inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Given grammar G and a collection of utterances, construct a phrase structure for each utterance.

  13. Top-down inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Infer {si} given {ui}, G: P( {si} | {ui}, G) α P( {ui} | {si} ) P( {si} |G)

  14. Bottom-up inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Given a collection of phrase structures, learn a grammar G.

  15. Bottom-up inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Infer G given {si} and U: P(G| {si}, U) α P( {si} | G) P(G|U)

  16. Simultaneous learning at multiple levels U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Given a set of utterances {ui} and innate knowledge U, construct a grammar G and a phrase structure for each utterance.

  17. Simultaneous learning at multiple levels U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance • A chicken-or-egg problem: • Given a grammar, phrase structures can be constructed • Given a set of phrase structures, a grammar can be learned

  18. Simultaneous learning at multiple levels U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Infer G and {si} given {ui} and U: P(G, {si} | {ui}, U) α P( {ui} | {si} )P({si} |G)P(G|U)

  19. Hierarchical Bayesian model U Universal Grammar P(G|U) G Grammar P(s|G) s1 s2 s3 s4 s5 s6 Phrase structure P(u|s) u1 u2 u3 u4 u5 u6 Utterance

  20. Knowledge at multiple levels • Top-down inferences: • How does abstract knowledge guide inferences at lower levels? • Bottom-up inferences: • How can abstract knowledge be acquired? • Simultaneous learning at multiple levels of abstraction

  21. Outline • A high-level view of HBMs • A case study: Semantic knowledge

  22. Folk Biology The relationships between living kinds are well described by tree-structured representations R: principles mouse S: structure squirrel chimp gorilla D: data “Gorillas have hands”

  23. Folk Biology R: principles Structural form: tree mouse squirrel S: structure chimp gorilla D: data

  24. Outline • A high-level view of HBMs • A case study: Semantic knowledge • Property induction • Learning structured representations • Learning the abstract organizing principles of a domain

  25. Property induction R: principles Structural form: tree mouse squirrel S: structure chimp gorilla D: data

  26. Property Induction Structural form: tree Stochastic process: diffusion R: Principles mouse squirrel S: structure chimp gorilla D: data Approach: work with the distribution P(D|S,R)

  27. Horses have T4 cells. Elephants have T4 cells. All mammals have T4 cells. Horses have T4 cells. Seals have T4 cells. All mammals have T4 cells. Property Induction Previous approaches: Rips (75), Osherson et al (90), Sloman (93), Heit (98)

  28. Bayesian Property Induction Hypotheses

  29. Bayesian Property Induction Hypotheses

  30. Horses have T4 cells. Elephants have T4 cells. Cows have T4 cells. } D C

  31. Chimps have T4 cells. Gorillas have T4 cells. Taxonomic similarity Poodles can bite through wire. Dobermans can bite through wire. Jaw strength Salmon carry E. Spirus bacteria. Grizzly bears carry E. Spirus bacteria. Salmon carry E. Spirus bacteria. Grizzly bears carry E. Spirus bacteria. Food web relations Choosing a prior

  32. Bayesian Property Induction • A challenge: • We have to specify the prior, which typically includes many numbers • An opportunity: • The prior can capture knowledge about the problem.

  33. Property Induction Structural form: tree Stochastic process: diffusion R: Principles mouse squirrel S: structure chimp gorilla D: data

  34. Biological properties • Structure: • Living kinds are organized into a tree • Stochastic process: • Nearby species in the tree tend to share properties

  35. Structure:

  36. Structure:

  37. Stochastic Process • Nearby species in the tree tend to share properties. • In other words, properties tend to be smooth over the tree. Smooth Not smooth

  38. Stochastic process Hypotheses

  39. Generating a property y h where y tends to be smooth over the tree: threshold

  40. S

  41. The diffusion process where Ө(yi) is 1 if yi≥ 0 and 0 otherwise the covariance K encourages y to be smooth over the graph S

  42. p(y|S,R): Generating a property Let yi be the feature value at node i i } j (Zhu, Lafferty, Ghahramani 03)

  43. Biological properties Structural form: tree Stochastic process: diffusion R: Principles mouse squirrel S: structure chimp gorilla D: data Approach: work with the distribution P(D|S,R)

  44. Horses have T4 cells. Elephants have T4 cells. Cows have T4 cells. } D C

  45. Cows have property P. Elephants have property P. Horses have property P. Dolphins have property P. Seals have property P. Horses have property P. Results Human Model (Osherson et al)

  46. Gorillas have property P. Mice have property P. Seals have property P. All mammals have property P. Results Human Model Cows have property P. Elephants have property P. Horses have property P. All mammals have property P.

  47. Spatial model Structural form: 2D space Stochastic process: diffusion R: principles squirrel mouse S: structure gorilla chimp D: data

  48. Structure:

  49. Structure:

  50. Tree vs 2D Tree + diffusion 2D + diffusion “horse” “all mammals”

More Related