What to make of: distributed representations summation of inputs Hebbian plasticity ?

1. What to make of: • distributed representations • summation of inputs • Hebbian plasticity ? Competitive nets Pattern associators Autoassociators

2. "Hebb-Marr" associative nets McNaughton & Morris, 1987

3. Between 1968 and 1971, David Marr published the first series of theoretical papers that attempted to interpret the anatomical connections and known physiology of specific brain structures (cerebellum, hippocampus, neocortex) in terms of associative memory systems using Hebb's rule. He made numerous specific predictions about the physiological properties of these nets that have since been verified.

8. Competitive Networks • “discover” structures in input • space • may remove redundancy • may orthogonalize • may categorize • may sparsify representations • can separate out even linear • combinations • can easily be generalized to self- • organizing topographic maps

10. Competitive Networks • “discover” structures in input • space • may remove redundancy • may orthogonalize • may categorize • may sparsify representations • can separate out even linear • combinations • can easily be generalized to self- • organizing topographic maps

11. Pattern associators • generalize to nearby input • vectors • gracefully degrade/fault tolerant • may extract prototypes • may reduce fluctuations • are fast (feed forward) • p (the # of associations) is • proportional to C (# inputs/unit) • and also grows with decreasing • a (sparser patterns) Y X r (X) = r (Y) after learning

12. 1 1 0 1 0 0

13. 1 1 0 1 0 0

14. 0 0 1 0 1 1

15. 1 0 0 1 1 0

16. Associative retrieval: The inhibitory interneuron divides down the excitation in proportion to the total input to the net, so that only the most strongly excited cells reach threshold (i.e., integer division). 3 3 2 2 3 3 2 3 1 0 0 1 1 0

17. 3 3 2 2 3 3 2 3 1 0 0 1 1 0

18. Pattern completion: The inhibitory interneuron divides down the excitation in proportion to the total input to the net, so that only the most strongly excited cells reach threshold (i.e., integer division). 2 1 1 2 1 2 2 2 0 0 1 0 1 1

19. 0 0 1 0 1 1

20. Pattern associators • generalize to nearby input • vectors • gracefully degrade/fault tolerant • may extract prototypes • may reduce fluctuations • are fast (feed forward) • p (the # of associations) is • proportional to C (# inputs/unit) • and also grows with decreasing • a (sparser patterns) Y X r (X) = r (Y) after learning

21. Storage is formally equivalent to forming the so-called outer product of two binary vectors. Multiple memories are stored by combining outer product matrices using the logical OR operation. Retrieval is formally equivalent to taking the inner (or dot) product of the input vector and the storage matrix (i.e., multiply through the rows and sum the columns) and subjecting it to a non-linear normalization (i.e., integer division).

22. Properties of interneurons predicted by the simple Hebb-Marr net model • Inhibitory cells must be driven by the same excitatory afferents that activate the principal cells . • The inhibitory mechanism must implement a division operation on the excitation that reaches the cell body from the dendrites. • Inhibitory cells can be much fewer in number than principal cells, but they must have extensive connectivity. • Inhibitory cells must respond to a synchronous input at lower threshold and at shorter latency than principal cells. Shorter latency is necessary to ensure that the appropriate division operation is already set up at the somata of the principal cells by the time the excitation from the same input arrives there via the principal cell dendrites. • Whereas principal cells will be quite selective in their response characteristics, inhibitory neurons will not be particular about which afferents are active at a given time, only about how many are active. Thus they will convey little information in the principal cells' response domain. • Excitatory synapses onto interneurons should not be plastic • In unfamiliar situations (i.e. when current input elicits reduced output) extrinsic modulation of inhibitory neurons might lower output threshold, successively probing for a complete pattern. This might also serve as a gate enabling the activation of the synaptic modification process.

23. Axons of basket inhbitory interneurons project widely. 1-2 mm in transverse plane, 2-4 mm in long axis Freund & Buzsaki, 1996

24. CA3 is dominated by recurrent collaterals

25. Autoassociation in a recurrent net

26. Autoassociation in a recurrent net The input pattern is kept on for 2 time steps so that the output at time t-1 is associated with the input at time t

27. Recurrent net after storage of 3 input vectors

28. Error correction (pattern completion)

29. "Reverberation": persistence of pattern after removal of input Persistence requires synaptic weights to be symmetrical: Wij = Wji , although for large nets symmetry does not have to be perfect.

30. Sequence Learning in Recurrent Nets If inputs change each cycle, the output at time t-1 is associated with the input at time t.

31. This leads to sequence storage.

32. The sequence may be of arbitrary length.

33. Presentation of any vector in the sequence leads to sequence completion if and only if all elements of the sequence are unique.

36. Autoassociators • complete a partial cue • can continue a sequence • generalize • gracefully degrade/fault tolerant • extract prot. / reduce fluct. • are also fast, but their feedback • may sustain short term memory • pc is again proportional to C • and grows with decreasing a • pc = k C / [a log (1/a)] Y r (Y’) = r (Y) after learning

37. Constraints on Storage Capacity 1) Connectivity. In real networks connectivity is much less than all-to-all. Capacity is proportional to the number of synapses/unit, C. 2) Sparsity of coding, a. Sparsity roughly speaking refers to the fraction of the units active at any one time. The continuum ranges from extremely sparse (a=1/N) to fully distributed (a=1). Sparse activity allows more patterns, but not more information. 3) "Orthogonality" of coding. By definition, orthogonal vectors have an overlap of zero. In practice, the maximum number of stored patterns is achieved when the patterns are uncorrelated, not mutually orthogonal. Correlated patterns cause local saturation.

38. Sparser inputs to the associative memory system can be generated by more efficient coding of the input (redundancy reduction). Much of the processing of visual and other sensory information in 'primary' and 'secondary' sensory cortex is devoted to this process of "feature extraction", with various simple feature detectors developmentally programmed. Association cortex learns to develop high order feature detectors based on past experience.