1 / 83

9.012 Brain and Cognitive Sciences II

9.012 Brain and Cognitive Sciences II. Part VIII: Intro to Language & Psycholinguistics - Dr. Ted Gibson. Presented by Liu Lab. Fighting for Freedom with Cultured Neurons. Distributed Representations, Simple Recurrent Networks, And Grammatical Structure Jeffrey L. Elman (1991)

ewebster
Download Presentation

9.012 Brain and Cognitive Sciences II

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 9.012Brain andCognitive Sciences II Part VIII: Intro to Language & Psycholinguistics - Dr. Ted Gibson

  2. Presented by Liu Lab Fighting for Freedom with Cultured Neurons

  3. Distributed Representations, Simple Recurrent Networks, And Grammatical Structure Jeffrey L. Elman (1991) Machine Learning Nathan Wilson

  4. Distributed Representations/ Neural Networks • are meant to capture the essence of neural computation: many small, independent units calculating very simple functions in parallel.

  5. Distributed Representations/ Neural Networks: EXPLICIT RULES?

  6. Distributed Representations/ Neural Networks: EXPLICIT RULES?

  7. Distributed Representations/ Neural Networks: EXPLICIT RULES? EMERGENCE!

  8. Distributed Representations/ Neural Networks • are meant to capture the essence of neural computation: many small, independent units calculating very simple functions in parallel.

  9. FeedForward Neural Network (from Sebastian’s Teaching)

  10. Don’t forget the nonlinearity!

  11. FeedForward Neural Network (from Sebastian’s Teaching)

  12. Recurrent Network (also from Sebastian)

  13. Why Apply Network / Connectionist Modeling to Language Processing? • Connectionist Modeling is Good at What it Does • Language is a HARD problem

  14. What We Are Going to Do

  15. What We Are Going to Do • Build a network

  16. What We Are Going to Do • Build a network • Let it learn how to “read”

  17. What We Are Going to Do • Build a network • Let it learn how to “read” • Then test it!

  18. What We Are Going to Do • Build a network • Let it learn how to “read” • Then test it! • Give it some words in a reasonably grammatical sentence • Let it try to predict the next word, • Based on what it knows about grammar

  19. What We Are Going to Do • Build a network • Let it learn how to “read” • Then test it! • Give it some words in a reasonably grammatical sentence • Let it try to predict the next word, • Based on what it knows about grammar • BUT: We’re not going to tell it any of the rules

  20. What We Are Going to Do • Build a network

  21. FeedForward Neural Network (from Sebastian’s Teaching)

  22. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 INPUT

  23. What We Are Going to Do • Build a network • Let it learn how to “read”

  24. Methods > Network Implementation > Training Words We’re going to Teach it: • - Nouns: • boy | girl | cat | dog | • boys | girls | cats | dogs • - Proper Nouns: • John | Mary • “Who” • - Verbs: • chase | feed | see | hear | walk | live | • chases | feeds | sees | hears | walks | lives • “End Sentence”

  25. Methods > Network Implementation > Training 1. Encode Each Word with Unique Activation Pattern

  26. Methods > Network Implementation > Training 1. Encode Each Word with Unique Activation Pattern • - boy => 000000000000000000000001 • girl => 000000000000000000000010 • feed => 000000000000000000000100 • -sees => 000000000000000000001000 • . . . • who => 010000000000000000000000 • End sentence => • 100000000000000000000000

  27. Methods > Network Implementation > Training 1. Encode Each Word with Unique Activation Pattern • - boy => 000000000000000000000001 • girl => 000000000000000000000010 • feed => 000000000000000000000100 • -sees => 000000000000000000001000 • . . . • who => 010000000000000000000000 • End sentence => • 100000000000000000000000 2. Feed these words sequentially to the network (only feed words in sequences that make good grammatical sense!)

  28. Methods > Network Implementation > Structure INPUT

  29. Methods > Network Implementation > Structure 1000000000000 INPUT

  30. Methods > Network Implementation > Structure HIDDEN 1000000000000 INPUT

  31. Methods > Network Implementation > Structure 100100100100100100100100 HIDDEN 1000000000000 INPUT

  32. Methods > Network Implementation > Structure OUTPUT 100100100100100100100100 HIDDEN 1000000000000 INPUT

  33. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 INPUT

  34. Methods > Network Implementation > Training 1. Encode Each Word with Unique Activation Pattern • - boy => 000000000000000000000001 • girl => 000000000000000000000010 • feed => 000000000000000000000100 • -sees => 000000000000000000001000 • . . . • who => 010000000000000000000000 • End sentence => • 100000000000000000000000 2. Feed these words sequentially to the network (only feed words in sequences that make good grammatical sense!)

  35. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 INPUT

  36. What We Are Going to Do • Build a network • Let it learn how to “read”

  37. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 INPUT

  38. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN If learning word relations, need some sort of memory from word to word! 1000000000000 INPUT

  39. FeedForward Neural Network (from Sebastian’s Teaching)

  40. Recurrent Network (also from Sebastian)

  41. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 INPUT

  42. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 100100100100100100100100 INPUT CONTEXT

  43. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 100100100100100100100100 INPUT CONTEXT

  44. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 100100100100100100100100 INPUT CONTEXT

  45. Methods > Network Implementation > Structure 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 100100100100100100100100 INPUT CONTEXT

  46. Methods > Network Implementation > Structure BACKPROP! 0000000000001 OUTPUT 100100100100100100100100 HIDDEN 1000000000000 100100100100100100100100 INPUT CONTEXT

  47. What We Are Going to Do • Build a network • Let it learn how to “read” • Then test it! • Give it some words in a reasonably grammatical sentence • Let it try to predict the next word, • Based on what it knows about grammar • BUT: We’re not going to tell it any of the rules

  48. Results > Emergent Properties of Network > Subject-Verb Agreement • After Hearing: • “boy….” • Network SHOULD predict next word is: • “chases” • NOT: • “chase” • Subject and verb should agree!

  49. Results > Emergent Properties of Network > Noun-Verb Agreement • After Hearing: • “boy….” • Network SHOULD predict next word is: • “chases” • NOT: • “chase” • Subject and verb should agree!

  50. Results > Emergent Properties of Network > Noun-Verb Agreement boy….. End of Sentence “Who” Plural Verb, DO Impossible Plural Verb, DO Required Plural Verb, DO Optional What Word Network Predicts is Next Single Verb, DO Impossible Single Verb, DO Required Single Verb, DO Optional Plural Noun Single Noun 0.0 0.2 0.4 0.6 0.8 1.0 Activation

More Related