Hierarchical Bayesian Models for Universal Grammar

1. Part IIIHierarchical Bayesian Models

2. Universal Grammar Hierarchical phrase structure grammars (e.g., CFG, HPSG, TAG) Grammar Phrase structure Utterance Speech signal

3. Vision (Han and Zhu, 2006)

4. Word learning Whole-object principle Shape bias Taxonomic principle Contrast principle Basic-level bias Principles Structure Data

5. Hierarchical Bayesian models • Can represent and reason about knowledge at multiple levels of abstraction. • Have been used by statisticians for many years.

6. Hierarchical Bayesian models • Can represent and reason about knowledge at multiple levels of abstraction. • Have been used by statisticians for many years. • Have been applied to many cognitive problems: • causal reasoning (Mansinghka et al, 06) • language (Chater and Manning, 06) • vision (Fei-Fei, Fergus, Perona, 03) • word learning (Kemp, Perfors, Tenenbaum,06) • decision making (Lee, 06)

7. Outline • A high-level view of HBMs • A case study • Semantic knowledge

8. Universal Grammar Hierarchical phrase structure grammars (e.g., CFG, HPSG, TAG) P(grammar | UG) Grammar P(phrase structure | grammar) Phrase structure P(utterance | phrase structure) Utterance P(speech | utterance) Speech signal

9. Hierarchical Bayesian model U Universal Grammar P(G|U) G Grammar P(s|G) s1 s2 s3 s4 s5 s6 Phrase structure P(u|s) u1 u2 u3 u4 u5 u6 Utterance

10. Hierarchical Bayesian model U Universal Grammar P(G|U) G Grammar P(s|G) s1 s2 s3 s4 s5 s6 Phrase structure P(u|s) u1 u2 u3 u4 u5 u6 Utterance A hierarchical Bayesian model specifies a joint distribution over all variables in the hierarchy:P({ui}, {si}, G | U) = P ({ui} | {si}) P({si} | G) P(G|U)

11. Knowledge at multiple levels • Top-down inferences: • How does abstract knowledge guide inferences at lower levels? • Bottom-up inferences: • How can abstract knowledge be acquired? • Simultaneous learning at multiple levels of abstraction

12. Top-down inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Given grammar G and a collection of utterances, construct a phrase structure for each utterance.

13. Top-down inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Infer {si} given {ui}, G: P( {si} | {ui}, G) α P( {ui} | {si} ) P( {si} |G)

14. Bottom-up inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Given a collection of phrase structures, learn a grammar G.

15. Bottom-up inferences U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Infer G given {si} and U: P(G| {si}, U) α P( {si} | G) P(G|U)

16. Simultaneous learning at multiple levels U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Given a set of utterances {ui} and innate knowledge U, construct a grammar G and a phrase structure for each utterance.

17. Simultaneous learning at multiple levels U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance • A chicken-or-egg problem: • Given a grammar, phrase structures can be constructed • Given a set of phrase structures, a grammar can be learned

18. Simultaneous learning at multiple levels U Universal Grammar G Grammar s1 s2 s3 s4 s5 s6 Phrase structure u1 u2 u3 u4 u5 u6 Utterance Infer G and {si} given {ui} and U: P(G, {si} | {ui}, U) α P( {ui} | {si} )P({si} |G)P(G|U)

19. Hierarchical Bayesian model U Universal Grammar P(G|U) G Grammar P(s|G) s1 s2 s3 s4 s5 s6 Phrase structure P(u|s) u1 u2 u3 u4 u5 u6 Utterance

20. Knowledge at multiple levels • Top-down inferences: • How does abstract knowledge guide inferences at lower levels? • Bottom-up inferences: • How can abstract knowledge be acquired? • Simultaneous learning at multiple levels of abstraction

21. Outline • A high-level view of HBMs • A case study: Semantic knowledge

22. Folk Biology The relationships between living kinds are well described by tree-structured representations R: principles mouse S: structure squirrel chimp gorilla D: data “Gorillas have hands”

23. Folk Biology R: principles Structural form: tree mouse squirrel S: structure chimp gorilla D: data

24. Outline • A high-level view of HBMs • A case study: Semantic knowledge • Property induction • Learning structured representations • Learning the abstract organizing principles of a domain

25. Property induction R: principles Structural form: tree mouse squirrel S: structure chimp gorilla D: data

26. Property Induction Structural form: tree Stochastic process: diffusion R: Principles mouse squirrel S: structure chimp gorilla D: data Approach: work with the distribution P(D|S,R)

27. Horses have T4 cells. Elephants have T4 cells. All mammals have T4 cells. Horses have T4 cells. Seals have T4 cells. All mammals have T4 cells. Property Induction Previous approaches: Rips (75), Osherson et al (90), Sloman (93), Heit (98)

28. Bayesian Property Induction Hypotheses

29. Bayesian Property Induction Hypotheses

30. Horses have T4 cells. Elephants have T4 cells. Cows have T4 cells. } D C

31. Chimps have T4 cells. Gorillas have T4 cells. Taxonomic similarity Poodles can bite through wire. Dobermans can bite through wire. Jaw strength Salmon carry E. Spirus bacteria. Grizzly bears carry E. Spirus bacteria. Salmon carry E. Spirus bacteria. Grizzly bears carry E. Spirus bacteria. Food web relations Choosing a prior

32. Bayesian Property Induction • A challenge: • We have to specify the prior, which typically includes many numbers • An opportunity: • The prior can capture knowledge about the problem.

33. Property Induction Structural form: tree Stochastic process: diffusion R: Principles mouse squirrel S: structure chimp gorilla D: data

34. Biological properties • Structure: • Living kinds are organized into a tree • Stochastic process: • Nearby species in the tree tend to share properties

35. Structure:

36. Structure:

37. Stochastic Process • Nearby species in the tree tend to share properties. • In other words, properties tend to be smooth over the tree. Smooth Not smooth

38. Stochastic process Hypotheses

39. Generating a property y h where y tends to be smooth over the tree: threshold

40. S

41. The diffusion process where Ө(yi) is 1 if yi≥ 0 and 0 otherwise the covariance K encourages y to be smooth over the graph S

42. p(y|S,R): Generating a property Let yi be the feature value at node i i } j (Zhu, Lafferty, Ghahramani 03)

43. Biological properties Structural form: tree Stochastic process: diffusion R: Principles mouse squirrel S: structure chimp gorilla D: data Approach: work with the distribution P(D|S,R)

44. Horses have T4 cells. Elephants have T4 cells. Cows have T4 cells. } D C

45. Cows have property P. Elephants have property P. Horses have property P. Dolphins have property P. Seals have property P. Horses have property P. Results Human Model (Osherson et al)

46. Gorillas have property P. Mice have property P. Seals have property P. All mammals have property P. Results Human Model Cows have property P. Elephants have property P. Horses have property P. All mammals have property P.

47. Spatial model Structural form: 2D space Stochastic process: diffusion R: principles squirrel mouse S: structure gorilla chimp D: data

48. Structure:

49. Structure:

50. Tree vs 2D Tree + diffusion 2D + diffusion “horse” “all mammals”