Left over from units..

1. Left over fromunits.. • Bias weight • General cell excitability is plastic, regardless of particular synaptic inputs (weights). See review by Mozzachido & Byrne, 2010 TINS on non-synapticplasticity.

2. Networks Layers and layers ofdetectors...

3. Networks • Biology of networks: thecortex • Excitation: • Unidirectional(transformations) • Bidirectional (pattern completion,amplification) • Inhibition: Controlling bidirectionalexcitation. • Constraint Satisfaction: Putting it alltogether.

4. Cortex:Neurons • Two separatepopulations: • Excitatory (glutamate): Pyramidal, Spinystellate. • Inhibitory (GABA): Chandelier,Basket.

5. More recentimages.. Animation:file:///Users/frankmj/teach/cogsim/ctxhippo.mpg

6. Excitatory vs InhibitoryNeurons • Excitatory neurons both project locally and make long-range projections between different corticalareas • Inhibitory neurons primarily project within small, localized regions of cortex • Excitatory neurons carry the information flow (long rangeprojections) • Inhibitory neurons are responsible for (locally) regulating the activation of excitatoryneurons

7. Laminar Structure ofCortex

8. Layers • Layer = a bunch of neurons with similarconnectivity • Localized to a particular region (physicallycontiguous) • All cells within a layer receive input from approximately the same places (i.e,. from a common collection oflayers) • All cells within a layer send their outputs to approximately the same places (i.e., to a common collection oflayers)

9. Laminar Structure of Cortex Three FunctionalLayers

10. Laminar Structure of Cortex Three FunctionalLayers Hidden layer transforms input-outputmappings

11. Laminar Structure of Cortex Three FunctionalLayers Hidden layer transforms input-outputmappings More hidden layers → richertransformations

12. Laminar Structure of Cortex Three FunctionalLayers Hidden layer transforms input-outputmappings More hidden layers → richertransformations → less reflex-like... smarter? more“free”?

16. Networks • Biology: Cortical layers andneurons • Excitation: • Unidirectional(transformations) • Similarity and Distributed Representations • Bidirectional (pattern completion,amplification) • Inhibition: Controlling bidirectionalexcitation. • Constraint Satisfaction: Putting it alltogether.

17. Networks • Biology: Cortical layers andneurons • Excitation: • Unidirectional(transformations) • Similarity and Distributed Representations • Bidirectional (pattern completion,amplification) • Inhibition: Controlling bidirectionalexcitation. • Constraint Satisfaction: Putting it alltogether.

18. Excitation (Unidirectional):Transformations

19. Excitation (Unidirectional):Transformations • Detectors work in parallel to transform input activity pattern to hidden activity pattern.

20. Excitation (Unidirectional):Transformations • Emphasizes some distinctions, collapses acrossothers.

21. Excitation (Unidirectional):Transformations • Emphasizes some distinctions, collapses acrossothers. • Function of what detectors detect (and what theyignore).

22. Emphasizing/CollapsingDistinctions 0 1 2 3 4 5 6 7 8 9 Hidden 2 7 8 3 1 6 5 0 9 4 Input

23. Emphasizing/CollapsingDistinctions 0 1 2 3 4 5 6 7 8 9 Hidden 2 7 8 3 1 6 5 0 9 4 Input Other (more interesting) examples?...

24. [transform.proj] • digitdetectors: • tested with noisydigits • tested withletters

25. Networks • Biology: Cortical layers andneurons • Excitation: • Unidirectional(transformations) • Similarity and Distributed Representations • Bidirectional (pattern completion,amplification) • Inhibition: Controlling bidirectionalexcitation. • Constraint Satisfaction: Putting it alltogether.

26. Neural Pattern Similarity Motivating points Networks deal with observations (data: cats, tigers, sounds, events,..) as: patterns of spikes/rates over many neurons • If the data are continuous, there is zero probability of seeing exactly any given instance again. Why? (Relatedly, we would need infinitely many rules.) • Even if the data are not actually continuous, they are often practically continuous. Newton’s laws vs QM, etc. • How many binary patterns are there for this 3x3 grid? • 29 = 512, which is basically infinity. For a 5x5 grid? • 225 = 33,554,432, which I cannot count to.

27. Neural Pattern Similarity Euclidean distance: motivating points • So we usually can’t have rules dealing with individual things. • To class: But then how could we ever tell a network how we want it to act on infinitely many things? • We need to lump things together. Things which are lumped together based on their properties are ______. • How do we do that? How similar are two things? • trivial similarity (identity): If not my toothbrush, not similar enough. • overlap and near overlap (X-Y example) • p-adic metric (google it) if you are wily, + infinitely many more

28. Euclidean distance: motivating points Overlap and near overlap Add “Y” Four squares “missing”, two squares “added”. List the squares, determine differences, (xi - yi) add them. If we want to these not to cancel, square them: Σ(xi - yi)2

29. ClusterPlots • Cluster plots provide a means of visualizing similarity relationships between patterns of activity in a network • Cluster plots are constructed based on the distances between patterns of activity • Euclidean distance = sum (across all units) of the squared difference inactivation 2 d= (x −y ) i i i • Example...

34. [transform.proj] cluster plots (digits, noisy digits,hidden).

35. Emphasizing/Collapsing Distinctions:Categorization

36. Emphasizing/Collapsing Distinctions:Categorization Emphasize distinctions: digits separated, even though they have perceptualoverlap.

37. Emphasizing/Collapsing Distinctions:Categorization Emphasize distinctions: digits separated, even though they have perceptualoverlap. Collapse distinctions: Noisy digits categorized as same, even though they have perceptual differences.

38. Detectors are Dedicated,Content-Specific

39. Localist vs. DistributedRepresentations

40. Localist vs. DistributedRepresentations • Localist = 1 unit responds to 1 thing (e.g., digits, grandmother cell).

41. Localist vs. DistributedRepresentations • Localist = 1 unit responds to 1 thing (e.g., digits, grandmother cell). • Distributed = Many units respond to 1 thing, one unit responds to manythings.

42. Localist vs. DistributedRepresentations • Localist = 1 unit responds to 1 thing (e.g., digits, grandmothercell). • Distributed = Many units respond to 1 thing, one unit responds to manythings. • With distributed representations, units correspond to stimulusfeatures • as opposed to complete stimuli

43. Digits With DistributedRepresentations Hidden Input

44. Advantages of DistributedRepresentations Efficiency: Fewer UnitsRequired Same question as earlier: Q: How many binary patterns can you encode with these two units?

45. Advantages of DistributedRepresentations Efficiency: Fewer UnitsRequired The digits network can represent 10 digits using 5 “feature” units Each digit is a unique combination of the 5 features,e.g., “0” = feature3 “1” = features 1,4 “2” = features 1,2 “3” = features 1, 2,5 “4” = features 3,4 “5” = features 1, 2,3 There are 32 unique ways to combine 5features (2n) There are > 1 million unique ways to combine 20features

46. Advantages of DistributedRepresentations Similarity andGeneralization:

47. Advantages of DistributedRepresentations Similarity andGeneralization: If you represent stimuli in terms of their constituent features, stimuli with similar features get assigned similarrepresentations This allows you to generalize to novel stimuli based on their similarity to previously encounteredstimuli