Chapter 12: Outputs of Visual Processing

1. Chapter 12:Outputs of Visual Processing IT: Inferior Temporal Visual Cortex LPF/PF: (Lateral) Prefrontal Cortex RC: Recurrent Collateral Connections Attractor Networks: Feedback like system

2. We will discuss: • Ventral visual processing stream • IT output provides distributed representation of…. “what” • (Parietal sends “where” info) • Both IT & P output to STM in what region… • PFC (Prefrontal Cortex) • why STM system must be separate from IT & P system • interaction based on “attractor networks”

3. There are both superficial and deep cortical neuron layers in each attractor network • Superficial layers (s) project onto self or higher cortical level (like s for superman; up up and away!) • Deep layers (d) project onto self or downwards (d for downward)

4. How does STM work??? • Network must maintain firing rate after stimulus • For how long? • In a monkey IT, usually between 0.1 – 10 seconds; in PF even longer • How is it maintained? (also see figure) • The collateral connections of (nearby) pyramid cells create feedback loops. These undergo associative modification. • These collateral feedback loops are called: recurrent collateral connections

5. Intramodular connections IT PF • This model also says connections between networks must be weak. • This parameter, g, indicates the relative strength of inter-modular to intramodular connections.

6. Intramodular connections IT PF • Weak g is needed for PF to maintain firing rate (ie. a percept) while several intervening stimuli pass through the posterior IT networks

7. What is the Two Network Model ofthe visual-STM system??? • Separate networks are needed for both perception and STM to work simultaneously. So where are they?? • Network in IT: perceptual functions • Network in PF: for maintaining STM during intervening stimuli • These networks are coupled; ie. they communicate with each other with forward and back projections • How do we know…..

8. Stimulus Match to Sample Task • Used to understand STM. • Response in RC of PF is activated by both visual input of matching stimulus & backprojected memory of sample stimulus • Two networks firing in synchrony augment response choice

9. Stimulus Match to Sample Task • This model says that for continuous perception to occur, there must be a module that maintains representation of the stimulus during intervening stimuli (ie. a STM must exist) • Two network model allows planning which is dependent on ability to maintain several short term memories simultaneously.

10. Visual Search Task

11. Yummy, equations :-p Iai is current for neuron i in module a a = {IT, PF} (ie. a is either IT or PF) Iai *decreases in time with constant t, *increases proportionally to the activity elsewhere in the network is the synaptic efficacy between two neurons. (ie. between neuron ai and presynaptic neuron j of module b.) vbj is the weight of neuron j of module b (ie. firing rate) is current from the stimulus that is external to network This models how current (ie. firing rate) changes in time

12. This model computes synaptic efficacy (firing rate correspondence) between neurons within a module (either IT or PF) P is number of binary patterns of active neurons that are possible both for IT and PF h is a neuron with value 1 or 0 with probabilities f and (1-f) Nt is the normalization constant

13. Synaptic Modification • This is process of setting up neurons to create an attractor which can hold an item as STM • Once set up it may be reused when triggered by appropriate cue, even without further synaptic modification

14. Part II: Visual Outputs to LTM IT projects to the Hippocampus to create LTMs In the hippocampus objects may be located as part of a spatial scenes. This is called… episodic memory • especially primates, not so much rats • Primate object-memory representation could be generated by a continuous attractor

15. Damaged Hippocampus Anterograde amnesia. What causes thisin humans?? bilateral damage to HC and nearby parts of IT (Squire) Which LTM memories located in HC • HC shown necessary to learn declarative memories, epsiodic • Not spatial processing; eg. An object-place memory task which requires memory of object and environment or context in form of snapshot LTM functions not affected by HC damage? • procedural memory, anterograde amnesia does not impair procedural memory

16. Rats and Monkeys • Rat HC pyramidal cells recognize a previously learned “scene” only from the same location and orientation: egocentric (body based) • Learning a new scene for rat HC pyramidal cells takes about 10 minutes • A group of cells can map 2 spaces simultaneously, and is able to process/represent only one them • Rats also have task related HC cells • cells respond to olfactory stimuli with particular behavior response

17. Rats and Monkeys • Primate HC has spatial view cells that respond to where monkey is looking, from any vantage point: allocentric (world based) • Idiothetic cues to trigger memory when scene’s details are obscured: • eye position • head direction • linear and axial whole body motion

18. Why Study Spatial View Cells • Primates have spatial view cells: • They enable object-place memory (a kind of episodic memory) • some HC neurons respond to combo of object info and spatial info • HC cells respond both to presented image and location for response: this is called memory convergence (2 stimuli  1 memory)

19. Show figure Rats and Monkeys compared: RatsMonkeys place cellsspatial view cells wide visual foveate vision field • Perhaps cells use same computational process and respond according to their different input apparatuses

20. Hippocampus Models • CA3 Recurrent Collateral Connections create auto-association networks enabling…. Every CA3 neuron to associate with any other CA3 neuron involved in the same memory • The number of different memories p in the CA3 system: RC = recurrent connections a = sparseness k≈ synaptic efficacy Eg. For a rat: CRC = 12000; this translates to between 12k – 36k memories Low sparseness (ie .02)  more memories Same idea going between networks: back projections down to CA1 

21. CA3 RC Network • Mossy fibres help learning only, not recall

22. Continuous Attractors • To represent spatial patterns  use continuous attractor networks • HC  represents spatial patterns  uses continuous attractor networks • Continuous networks can explain how in the dark, rats can still maintain place cell firing (or even update the response with idiothetic inputs) • Now known, attractors can store both continuous and discrete patterns • locations in space = continuous • objects = discrete • HC receives and combines both input types: • For an event with both discrete and continuous aspects: spatial info can be retrieved from object cues and vice versa

23. Cool? Thanks!

24. I am intrigued by Rolls et al.'s research about specialized hippocampal cells (p.416). For example, they find cells that respond to a combination of which picture is shown and where the response must be made. Cells like these, that integrate perception of stimuli with what response is needed, seem like they could be close to feeding straight into consciousness (i.e. sound homunculus-like). However, since these cells are not tuned to respond to any particular stimuli or to coordinate any particular response, I can't get a handle on quite how they could work. Is that known? -S.R.

25. Rolls and others have found neurons responding to many strange stimuli that I should never have thought of: for instance, hippocampal cells that register not only where the animal is (rat) but where it is looking (primate): cells that respond to the physical properties of a stimulus (sweet vs. salty), others that respond to its reward contingencies. We take as an article of faith that every mental event has an accompanying brain event.  Does every mental event have a corresponding single-neuron event?  Perhaps yes, if Barlow is right and grandmother cells exist.  No, if many 'events' have a distributed code that spreads them across many neurons. In the present state of the art we can record events in single neurons (with electrodes) and in rather coarsely defined brain areas (with fMRI).  Between the microscopic and macroscopic techniques lies a mesoscopic gap - we cannot record events of the size of cell assemblies.  Is this where most of the brain action is? -S.A.

26. Meanwhile, what would we find in a catalogue of every stimulus that is known to excite single neurons in the brain? It would include faces, localities, reward contingenciesŠ  Would it contain every imaginable stimulus, and if not, what would it leave out?  Of these left-out items, would cell assemblies respond to them?  I'm really wondering what we could conclude from such a catalogue.  If it contains everything, does it really tell us anything?  And do we know of any instructive omissions from such a catalogue? (dogs that did not bark during the night). I've heard many informal criticisms of Rolls' book  --   too concentrated on his own work, ignoring the work of others; in particular, hard to read and hard to follow.  But what do we think, now that we have (in theory) read the book?  Is most of it "going nowhere", or is Rolls really "on to something" that we ignore at our peril?  Maybe his insights will give us huge breakthroughs, that we lack the intelligence or perseverance to descry?  What does everybody here think? -S.A.

27. In section 12.4, the authors point out how translation invariance in IT is a problem for the issue of how information about an object's spatial coordinates is made available to the motor system. They propose that to solve this problem, the motor system simply directs action toward the object foveated, since in most cases that is the object of interest. Although this explanation does seem possible, I am still not sold. I think a much more parsimonious explanation would be that even after translation invariance takes place in IT, some sort of representation of the target's spatial location is preserved within the system. Why should the motor system start from scratch when information about the spatial coordinates has already been extracted, albeit at an earlier level of visual processing? – L.L.

28. 1. The authors assume that thresholds are exceeded. Is there any mention in the book about how this may occur. 2. The authors assume there is an "activation function." How might this be derived from fundamerntal principles of neuronal activity rather than just be a convenient assumption that makes the model fit some data? 3. The measure of "sparseness" is claimed to have a value that can only occur in a very extreme and limited citcumstance. Is "sparseness" necessary? If so, what would be a better way to compute it. 4. Multiplication occurs in equations throughout the book. How can multiplication occur, physically? 5. Signals passing through the nervous system are representable as independent stochastic random variables. But when a threshold is posited the sum of such signals is no longer a sum of independent RVs because the sum is limited (ie must at least equal the threshold). The conventional algrbra of independent random variables no longer applies. What effect can this have on the theory? On any theory?

29. 6. Although the signal transmission system of the nervous system is known to be a stochastic process, and therefore the algebra of probability theory applies to models that represent the nervous system, the book uses real numbers and real algrebra and essentially linear system for its modelling. Is there something very wrong here? –S.L.

30. Page 416:  "These primate 'spatial view' hippocampal cells encode information in allocentric (world-centred, as opposed to egocentric) coordinates." There are two ways in which (for instance) a ship codes its own position:  visually, by measuring nearby visual landmarks: or from dead reckoning, by recording and remembering all its own motions since it left its home port.  Dead reckoning is subject to cumulative drift, whilst visual sighting is probably not.  (I ignore Harrison's chronometric measurements of longitude, and GPS methods).  Which of these two methods, or what combination thereof, do (or could) these hippocampal cells use? – S.A.