Michael Arbib with Erhan Oztop (Modeling) Giacomo Rizzolatti (Neurophysiology)

1. Michael Arbib • with • Erhan Oztop (Modeling) • Giacomo Rizzolatti (Neurophysiology) • The Mirror Neuron System for Grasping

2. Part 1 • Classic Concepts of Grasp Control

3. Preshaping While Reaching to Grasp Jeannerod & Biguer 1979

4. A Classic Coordinated Control Program for Reaching and Grasping recognition visual criteria input visual visual input input Visual Location activation of Size Orientation target visual search Recognition Recognition location size orientation visual , activation visual and kinesthetic, and of reaching kinesthetic input tactile input Hand Fast Phase Hand Preshape Movement Rotation Slow Phase Actual Movement Grasp Grasping Hand Reaching Perceptual schemas Motor schemas (Arbib 1981)

5. Opposition Spaces and Virtual Fingers The goal of a successful preshape, reach and grasp is to match the opposition axis defined by the virtual fingers of the hand with the opposition axis defined by an affordance of the object (Iberall and Arbib 1990)

6. Hand State • Our current representation of hand state defines a 7-dimensional trajectory F(t) with the following components • F(t) = (d(t), v(t), a(t), o1(t),o2(t), o3(t), o4(t)): • d(t): distance to target at time t • v(t): tangential velocity of the wrist • a(t): Aperture of the virtual fingers involved in grasping at time t • o1(t): Angle between the object axis and the (index finger tip – thumb tip) vector [relevant for pad and palm oppositions] • o2(t): Angle between the object axis and the (index finger knuckle – thumb tip) vector [relevant for side oppositions] • o3(t), o4(t): The two angles defining how close the thumb is to the hand as measured relative to the side of the hand and to the inner surface of the palm.

7. Part 2 • Basic Neural Mechanisms of Grasp Control

8. Visual Control of Grasping in Macaque Monkey A key theme of visuomotor coordination: parietal affordances (AIP) drive frontal motor schemas (F5) AIP - grasp affordances in parietal cortex Hideo Sakata F5 - grasp commands in premotor cortex Giacomo Rizzolatti

9. Grasp Specificity in an F5 Neuron • Precision pinch (top) • Power grasp (bottom) • (Data from Rizzolatti et al.)

10. A Broad Perspective on F5-AIP Interactions PP (Posterior Parietal) affordances for PM precise motor coordinates for motor cortex tell PM the possible categories the details - action parameters Cerebellum “here’s my choice” PM (Premotor Cortex) codes action fairly abstractly Motor Cortex and MPGs overall action Tuning and coordinating MPGs

11. FARS (Fagg-Arbib-Rizzolatti-Sakata)Model Overview AIP Dorsal Stream: Affordances Ways to grab this “thing” Task Constraints (F6) Working Memory (46?) Instruction Stimuli (F2) F5 Ventral Stream: Recognition “It’s a mug” IT PFC AIP extracts affordances - features of the object relevant to physical interaction with it. Prefrontal cortex provides “context” so F5 may select an appropriate affordance

12. Part 3 • Neural Mechanisms of Grasp Control • that Support a “Mirror System”

13. Mirror Neurons Rizzolatti, Fadiga, Gallese, and Fogassi, 1995:Premotor cortex and the recognition of motor actions Mirror neurons form the subset of grasp-related premotor neurons of F5 which discharge when the monkey observes meaningful hand movements made by the experimenter or another monkey. F5 is endowed with anobservation/execution matching system [The non-mirror grasp neurons of F5 are called F5 canonical neurons.]

14. What is the mirror system (for grasping) for? Mirror neurons: The cells that selectively discharge when the monkey executes particular actions as well as when the monkey observes an other individual executing the same action. Mirror neuron system (MNS): The mirror neurons and the brain regions involved in eliciting mirror behavior. Interpretations: • Action recognition • Understanding (assigning meaning to other’s actions) • Associative memory for actions

15. Computing the Mirror System Response • The FARS Model: • Recognize object affordances and determine appropriate grasp. • The Mirror Neuron System (MNS) Model: • We must add recognition of • trajectoryand • hand preshape • to • recognition of object affordances • and ensure that all three are congruent. • There are parietal systems other than AIP adapted to this task.

16. cIPS: caudal intraparietal sulcus Axis orientation and surface orientation cIPS cIPS cIPS Spatial coding for objects, analysis of motion during interaction of objects and self-motion 7a (PG): caudal part of the posterior parietal lobule Detection of biologically meaningful stimuli (e.g.hand actions) Motion related activity (MT/MST part) STS: Superior Temporal Sulcus 7b (PF): Rostral part of the posterior parietal lobule Mainly somatosensory Mirror-like responses Further Brain Regions Involved

17. cIPS cIPS cIPS cell response Surface orientation selectivity of a cIPS cell Sakata et al. 1997

18. Key Criteria for Mirror Neuron Activation When Observing a Grasp • a) Does the preshape of the hand correspond to the grasp encoded by the mirror neuron? • b) Does this preshape match an affordance of the target object? • c) Do samples of the hand state indicate a trajectory that will bring the hand to grasp the object? • Modeling Challenges: • i) To have mirror neurons self-organize to learn to recognize grasps in the monkey’s motor repertoire • ii) To learn to activate mirror neurons from smaller and smaller samples of a trajectory.

19. Hypothesis on Mirror Neuron Development The development of the (grasp) mirror neuron system in a healthy infant isdriven by the visual stimuligenerated by the actions (grasps) performed by theinfant himself. The infant (with maturation of visual acuity) gains the ability tomap other individual’s actionsinto his internal motor representation. [In the MNS model, the hand state provides the key representation for this transfer.] Then the infant acquires the ability to create (internal) representations fornovel actionsobserved. Parallel to these achievements, the infant develops anaction predictioncapability (the recognition of an action given the prefix of the action and the target object)

20. The Mirror Neuron System (MNS) Model

21. Part 4 • Implementing the Basic Schemas of the Mirror Neuron System (MNS) Model • using Artificial Neural Networks • (The Work of Erhan Oztop)

22. Curve recognition Solution: Fit a cubic spline to the sampled values. Then normalize and re-sample from the spline curve. Result:Very good generalization. Better performance than using the Fourier coefficients to recognize curves. The general problem: associate N-dimensional space curves with object affordances A special case: The recognition of two (or three) dimensional trajectories in physical space Simplest solution: Map temporal information into spatial domain. Then apply known pattern recognition techniques. Problem with simplest solution: The speed of the moving point can be a problem! The spatial representation may change drastically with the speed Scaling can overcome the problem. However the scaling must be such that it preserves thegeneralization ability of the pattern recognition engine.

23. Curve recognition Curve recognition system demonstrated for hand drawn numeral recognition (successful recognition examples for 2, 8 and 3). Spatial resolution: 30 Network input size: 30 Hidden layer size: 15 Output size: 5 Training : Back-propagation with momentum.and adaptive learning rate Sampled points Point used for spline interpolation Fitted spline

24. STS hand shape recognition Model Matching Precision grasp Hand Configuration Classification Color Coded Hand Feature Extraction Step 1 of hand shape recognition: system processes the color-coded hand image and generates a set of features to be used by the second step Step 2: The feature vector generated by the first step is used to fit a 3D-kinematics model of the hand by the model matching module. The resulting hand configuration is sent to the classification module.

25. STS hand shape recognition 1:Color Segmentation and Feature Extraction Color Expert (Network weights) Preprocessing Training phase: A color expert is generated by training a feed-forward network to approximate human perception of color. Features NN augmented segmentation system Actual processing: The hand image is fed to the augmented segmentation system. The color decision during segmentation is done by consulting color expert.

26. STS hand shape recognition2:3D Hand Model Matching Feature Vector Error minimization Result of feature extraction Grasp Type Classification The model matching algorithm minimizes the error between the extracted features and the model hand. A realistic drawing of hand bones. The hand is modelled with 14 degrees of freedom as illustrated.

27. Virtual Hand/Arm and Reach/Grasp Simulator A precision pinch A power grasp and a side grasp

28. Power grasp time series data +: aperture; *: angle 1; x: angle 2; : 1-axisdisp1; :1-axisdisp2; : speed; : distance.

29. Object Affordances Motor program Object affordance -hand state association Integrate temporal association F5canonical Hand shape recognition & Hand motion detection Action recognition (Mirror Neurons) Motor program Mirror Feedback Motor execution Hand-Object spatial relation analysis F5mirror Core Mirror Circuit Object affordance Mirror Neurons (F5mirror) Association (7b) Neurons Mirror Neuron Output Hand state Motor Program (F5 canonical) Mirror Feedback

30. Connectivity pattern Object affordance (AIP) STS F5mirror 7b Motor Program (F5canonical) Mirror Feedback 7a

31. A single grasp trajectory viewed from three different angles How the network classifies the action as a power grasp. Empty squares: power grasp output; filled squares: precision grasp; crosses: side grasp output The wrist trajectory during the grasp is shown by square traces, with the distance between any two consecutive trace marks traveled in equal time intervals.

32. Power and precision grasp resolution

33. Part 5 • Quo Vadis?

34. Future Directions • 1) Technology: Increasing the robustness and learning rates of the schemas using improved learning algorithms for artificial neural nets. • 2) Neuroscience: Implementing the schemas with biologically plausible neural nets to model neurophysiological data. • 3) Learning to recognize variations on, and assemblages of, familiar actions. • 4) Imitation • 5) Language!

35. Michael Arbib: CS564 - Brain Theory and Artificial IntelligenceUniversity of Southern California, Fall 2001 • Lecture 10. • The Mirror Neuron System Model (MNS) 1 • Reading Assignment: • Schema Design and Implementation of • the Grasp-Related Mirror Neuron System • Erhan Oztop and Michael A. Arbib

36. Visual Control of Grasping in Macaque Monkey A key theme of visuomotor coordination: parietal affordances (AIP) drive frontal motor schemas (F5) AIP - grasp affordances in parietal cortex Hideo Sakata F5 - grasp commands in premotor cortex Giacomo Rizzolatti

37. Mirror Neurons Rizzolatti, Fadiga, Gallese, and Fogassi, 1995:Premotor cortex and the recognition of motor actions Mirror neurons form the subset of grasp-related premotor neurons of F5 which discharge when the monkey observes meaningful hand movements made by the experimenter or another monkey. F5 is endowed with anobservation/execution matching system

38. F5 Motor Neurons • F5 Motor Neurons include all F5 neurons whose firing is related to motor activity. • We focus on grasp-related behavior. Other F5 motor neurons are related to oro-facial movements. • F5 Mirror Neurons form the subset of grasp-related F5 motor neurons of F5 which discharge when the monkey observes meaningful hand movements. • F5 Canonical Neurons form the subset of grasp-related F5 motor neurons of F5 which fire when the monkey sees an object with related affordances.

39. What is the mirror system (for grasping) for? Mirror neurons: The cells that selectively discharge when the monkey executes particular actions as well as when the monkey observes an other individual executing the same action. Mirror neuron system (MNS): The mirror neurons and the brain regions involved in eliciting mirror behavior. Interpretations: • Action recognition • Understanding (assigning meaning to other’s actions) • Associative memory for actions

40. Computing the Mirror System Response • The FARS Model: • Recognize object affordances and determine appropriate grasp. • The Mirror Neuron System (MNS) Model: • We must add recognition of • trajectoryand • hand preshape • to • recognition of object affordances • and ensure that all three are congruent. • There are parietal systems other than AIP adapted to this task.

41. cIPS: caudal intraparietal sulcus cIPS cIPS cIPS 7a (PG): caudal part of the posterior parietal lobule STS: Superior Temporal Sulcus 7b (PF): Rostral part of the posterior parietal lobule Further Brain Regions Involved Axis and surface orientation Spatial coding for objects, analysis of motion during interaction of objects and self-motion Detection of biologically meaningful stimuli (e.g.hand actions) Motion related activity (MT/MST part) Mainly somatosensory Mirror-like responses

42. cIPS cIPS cIPS cell response Surface orientation selectivity of a cIPS cell Sakata et al. 1997

43. Key Criteria for Mirror Neuron Activation When Observing a Grasp • a) Does the preshape of the hand correspond to the grasp encoded by the mirror neuron? • b) Does this preshape match an affordance of the target object? • c) Do samples of the hand state indicate a trajectory that will bring the hand to grasp the object? • Modeling Challenges: • i) To have mirror neurons self-organize to learn to recognize grasps in the monkey’s motor repertoire • ii) To learn to activate mirror neurons from smaller and smaller samples of a trajectory.

44. Initial Hypothesis on Mirror Neuron Development The development of the (grasp) mirror neuron system in a healthy infant isdriven by the visual stimuligenerated by the actions (grasps) performed by theinfant himself. The infant (with maturation of visual acuity) gains the ability tomap other individual’s actionsinto his internal motor representation. [In the MNS model, the hand state provides the key representation for this transfer.] Then the infant acquires the ability to create (internal) representations fornovel actionsobserved. Parallel to these achievements, the infant develops anaction predictioncapability (the recognition of an action given the prefix of the action and the target object)

45. The Mirror Neuron System (MNS) Model

46. Implementing the Basic Schemas of the Mirror Neuron System (MNS) Model • using Artificial Neural Networks • (Work of Erhan Oztop)

47. Opposition Spaces and Virtual Fingers The goal of a successful preshape, reach and grasp is to match the opposition axis defined by the virtual fingers of the hand with the opposition axis defined by an affordance of the object (Iberall and Arbib 1990)

48. Hand State • Our current representation of hand state defines a 7-dimensional trajectory F(t) with the following components • F(t) = (d(t), v(t), a(t), o1(t),o2(t), o3(t), o4(t)): • d(t): distance to target at time t • v(t): tangential velocity of the wrist • a(t): Aperture of the virtual fingers involved in grasping at time t • o1(t): Angle between the object axis and the (index finger tip – thumb tip) vector [relevant for pad and palm oppositions] • o2(t): Angle between the object axis and the (index finger knuckle – thumb tip) vector [relevant for side oppositions] • o3(t), o4(t): The two angles defining how close the thumb is to the hand as measured relative to the side of the hand and to the inner surface of the palm.

49. Curve recognition Solution: Fit a cubic spline to the sampled values. Then normalize and re-sample from the spline curve. Result:Very good generalization. Better performance than using the Fourier coefficients to recognize curves. The general problem: associate N-dimensional space curves with object affordances A special case: The recognition of two (or three) dimensional trajectoriesin physical space Simplest solution: Map temporal information into spatial domain. Then apply known pattern recognition techniques. Problem with simplest solution: The speed of the moving point can be a problem! The spatial representation may change drastically with the speed Scaling can overcome the problem. However the scaling must be such that it preserves thegeneralization ability of the pattern recognition engine.

50. Curve recognition Curve recognition system demonstrated for hand drawn numeral recognition (successful recognition examples for 2, 8 and 3). Spatial resolution: 30 Network input size: 30 Hidden layer size: 15 Output size: 5 Training : Back-propagation with momentum.and adaptive learning rate Sampled points Point used for spline interpolation Fitted spline