Michael Arbib: CS564 - Brain Theory and Artificial Intelligence University of Southern California, Fall 2001

1. Michael Arbib: CS564 - Brain Theory and Artificial IntelligenceUniversity of Southern California, Fall 2001 • Lecture 18. • The FARS model of Control of Reaching and Grasping • Reading Assignments: • Motor Schemas and Cortical Regions: • TMB 2, Sections 2.2, 5.3, 6.3* • FARS Model: • Fagg, A. H., and Arbib, M. A., 1998, Modeling Parietal-Premotor Interactions in Primate Control of Grasping, Neural Networks, 11:1277-1303. • * Caution: Most of the neuroanatomy in 6.3 is still reliable, but much research has updated our understanding of cortical correlates of motor control since 1989.

2. Perceptual And Motor Schemas • A perceptual schemaembodies the process whereby the system determines whether a given domain of interaction is present in the environment. {Recall our discussion of VISIONS, TMB2 §5.2} • A schema assemblagecombines an estimate of environmental state with a representation of goals and needs • The internal state is also updated by knowledge of the state of execution of current plans made up of motor schemas • which are akin to control systems but distinguished by the fact that they can be combined to form coordinated control programs

3. Preshaping While Reaching to Grasp

4. Hypothetical coordinated control program for reaching and grasping Perceptual Schemas Motor Schemas Dashed lines — activation signals; solid lines — transfer of data. (Adapted from Arbib 1981)

5. reach programming Parietal Cortex How (dorsal) grasp programming Visual Cortex Inferotemporal What (ventral) Cortex "What" versus "How” in Human DF: Jeannerod et al. Lesion here: Inability to Preshape (except for objects with size “in the semantics” Monkey Data: Mishkin and Ungerleider on “What” versus “Where” AT: Goodale and Milner Lesion here: Inability to verbalize or pantomime size or orientation

6. Reaching for an object by a patient with a lesion of the parietal cortex: Jeannerod, M., Michel, F., Prablanc, C., 1984, The Control of Hand Movements in a Case of Hemianaesthesia Following a Parietal Lesion, Brain107:899-920. Consider the implications for Project 1.

7. Virtual Fingers Arbib, Iberall and Lyons

8. Opposition Spaces and Virtual Fingers The goal of a successful preshape, reach and grasp is to match the opposition axis defined by the virtual fingers of the hand with the opposition axis defined by an affordance of the object (Iberall and Arbib 1990)

9. Planning for the reach must take account of the planned grasp

10. Somatosensory areas SMA FEF (saccades) SMA = pre-SMA + SMA-proper

11. Somatosensory data: A key to motor control

12. Do data support the idea of virtual fingers? • Iberall and Arbib 1988 suggest that multiple, seemingly non-somatotopic, representations of the digits could be due to a virtual finger representation. • Using the tentative identifications: • VF1 involves palm and thumb areas • VF2 involves the index with or without other fingers • VF3 involves finger combinations excluding the thumb and the index • yields a possible mapping of virtual fingers onto the caudal kinesthetic map that Strick and Preston (1962) found in the squirrel monkey.

13. Iberall and Arbib’s 1988 view of cortical contributions to the coordinated control program for reaching and grasping. Use it as an evaluation point as we develop the FARS and MNS models. What do we gain, what have we lost? See TMB2 §6.3 for the details.

14. Introducing AIP and F5 (Grasping) in Monkey A key theme of visuomotor coordination: parietal affordances (AIP) drive frontal motor schemas (F5) AIP - grasp affordances in parietal cortex Hideo Sakata F5 - grasp commands in premotor cortex Giacomo Rizzolatti

15. Grasp Specificity in an F5 Neuron • Precision pinch (top) • Power grasp (bottom) • (Data from Rizzolatti et al.)

16. The Sakata Protocol

17. Grip Selectivity in a Single AIP Cell A cell that is selective for side opposition (Sakata)

18. Differential Timing of Activity Peaks in Different AIP Neurons • Note the need for a broad database of many cells within each region to see that cells are not just “pattern recognizers” but also have a relationship to the time course of the ongoing behavior.

19. Size Specificity in a Single AIP Cell • This cell is selective toward small objects, somewhat independent of object type ( Hideo Sakata) • Note: Some cells show size specificity; others do not.

20. FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Overview AIP Dorsal Stream: Affordances Ways to grab this “thing” Task Constraints (F6) Working Memory (46?) Instruction Stimuli (F2) Ventral Stream: Recognition “It’s a mug” IT PFC AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.

21. Secondary Somatosensory Cortex (SII) • In the grasp versus point comparisonin a PET study of humans, we found a marked increase of activity in the secondary somatosensory cortex (SII). • Ablation of SII in non-human primates results in decrements in tactile discriminationand impaired tactile learning. • Focal lesions of the parietal operculum in humans characteristically produce tactile agnosia without loss of simple tactile sensation, or motor control. This deficit can includethe inability to sort objects based on size or shape,although sorting on texture is preserved. • The model relates the augmented response to higher order tactile feedbackor tactile expectation.

22. Motor Commands, Expectations, and Feeedback F5 (grasp type) expectation A7 (sensory (internal model) hyperfeatures) SII MI (muscle assemblies) (elementary SI sensory motor commands features) sensory info hand

23. Interaction of AIP and F5 During the Sakata Task Activation Connection Inhibitory Connection Priming Connection AIP precision-related cell AIP power-related cell

24. The Problem of Serial Order in Behavior (Karl Lashley) • If we tried to learn a sequence like • A  B A C • by reflex chaining, what is to stop A triggering B every time, • to yield the performance A  B A B A  ….. • (or we might get A  B+C A B+C A  …..) • A solution: Store the “action codes” (motor schemas) A, B, C, … in one part of the brain (F5 in FARS) and have another area (pre-SMA in FARS) hold “abstract sequences” and learn to pair the right action with each element: • (pre-SMA):x1  x2 x3 x4 abstract sequence • (F5): A B C action codes/motor schemas • Hypothesis:The “Sakata-Protocol Sequencing” is not mediated within F5 --Sequences are stored in pre-SMA and administered by the Basal Ganglia (BG)

25. Basal Ganglia Anatomy in the Rat • From Prescott et al., HBTNN 2e, to appear

26. A 2-Function View of the Basal Ganglia Skeletomotor Pathway Cortex Indirect Pathway Direct Pathway

27. Bischoff-Grethe Sequencing Model Pre-SMA SMA-Proper SMA-Proper Motor Cortex Basal Ganglia

28. The “Visual Front End” of the FARS Model F4 VIP Parietal (arm goal position) (position) Cortex How (dorsal) (object/grasp transform) AIP PIP F5 (shape, size, orientation) (grasp type) Visual Cortex IT What (ventral)

29. Positioning F2, F6 and Areas 46 and SII in Monkey

30. Prefrontal Influences on F5 pre-SMA Inferior Premotor Cortex F6 F4 (arm goal position) Frontal Cortex 46 F5 (grasp type) Dorsal premotor cortex F2 (abstract stimuli)

31. The Complete FARS Model