Learning Early Constructions: Building Models of Verbs Based on Motor Control Primitives

1. Learning early constructions (Chang, Mok) The ICSI/Berkeley Neural Theory of Language Project ECG

2. Moving from Spatial Relations to Verbs • Open class vs. closed class • How do we represent verbs (say of hand motion) • Can we build models of verbs based on motor control primitives? • If so, how can models overcome central limitations of Regier’s system? • Inference • Abstract uses

3. Coordination of Pattern Generators

4. Coordination • PATTERN GENERATORS, separate neural networks that control each limb, can interact in different ways to produce various gaits. • In ambling (top) the animal must move the fore and hind leg of one flank in parallel. • Trotting (middle) requires movement of diagonal limbs (front right and back left, or front left and back right) in unison. • Galloping (bottom) involves the forelegs, and then the hind legs, acting together

5. Preshaping While Reaching to Grasp

7. Internal Model and Efference Copy

9. Many areas code for motion parameters

11. Multiple, chronically implanted, intracranial microelectrode arrays would be used to sample the activity of large populations of single cortical neurons simultaneously. The combined activity of these neural ensembles would then be transformed by a mathematical algorithm into continuous three-dimensional arm-trajectory signals that would be used to control the movements of a robotic prosthetic arm. A closed control loop would be established by providing the subject with both visual and tactile feedback signals generated by movement of the robotic arm.

12. A New Picture Rizzolatti et al. 1998

13. The fronto-parietal networks Rizzolatti et al. 1998

14. F5 Mirror Neurons Gallese and Goldman, TICS 1998

15. Category Loosening in Mirror Neurons (~60%) Observed: A is Precision Grip B is Whole Hand Prehension Action: C: precision grip D: Whole Hand Prehension (Gallese et al. Brain 1996)

16. A (Full vision) B (Hidden) C (Mimicking) D (HiddenMimicking) Umiltà et al. Neuron 2001

17. F5 Audio-Visual Mirror Neurons Kohler et al. Science (2002)

18. Summary of Fronto-Parietal Circuits Motor-Premotor/Parietal Circuits PMv (F5ab) – AIP Circuit “grasp” neurons – fire in relation to movements of hand prehension necessary to grasp object F4 (PMC) (behind arcuate) – VIP Circuit transforming peri-personal space coordinates so can move toward objects PMv (F5c) – PF Circuit F5c different mirror circuits for grasping, placing or manipulating object Together suggest cognitive representation of the grasp, active in action imitation and action recognition

19. Evidence in Humans for Mirror, General Purpose, and Action-Location Neurons Mirror: Fadiga et al. 1995; Grafton et al. 1996; Rizzolatti et al. 1996; Cochin et al. 1998; Decety et al. 1997; Decety and Grèzes 1999; Hari et al. 1999; Iacoboni et al. 1999; Buccino et al. 2001. General Purpose: Perani et al. 1995; Martin et al. 1996; Grafton et al. 1996; Chao and Martin 2000. Action-Location: Bremmer, et al., 2001.

21. AIP Dorsal Stream: Affordances Ways to grab this “thing” Task Constraints (F6) Working Memory (46?) Instruction Stimuli (F2) Ventral Stream: Recognition “It’s a mug” IT PFC FARS (Fagg-Arbib-Rizzolatti-Sakata) Model AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it. Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model

22. MULTI-MODAL INTEGRATION The premotor and parietal areas, rather than having separate and independent functions, are neurally integrated not only to control action, but also to serve the function of constructing an integrated representation of: Actions, together with objects acted on, and locations toward which actions are directed. In these circuits sensory inputs are transformed in order to accomplish not only motor but also cognitive tasks, such as space perception and action understanding.

23. Modeling Motor Schemas • Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod, Rizzolatti) • Should model coordinated, distributed, parameterized control programs required for motor action and perception. • Should be an active structure. • Should be able to model concurrent actions and interrupts. • Should model hierarchical control (higher level motor centers to muscle extensor/flexors. • Computational model called x-schemas (http://www.icsi.berkeley.edu/NTL)

24. An Active Model of Events • At the Computational level, actions and events are coded in active representations called x-schemas which are extensions to Stochastic Petri nets. • x-schemas are fine-grained action and event representations that can be used for monitoring and control as well as for inference.

25. 3 2 Resource arc 1 Precondition arc Inhibition arc Model Review: Stochastic Petri Nets Basic Mechanism [1] [1] Firing function -- conjunctive -- logistic -- exponential family

26. Model Review 3 2 1 Firing Semantics

27. Model Review 1 2 1 1 1 Result of Firing

28. walker at goal energy walker=Harry goal=home Active representations • Many inferences about actions derive from what we know about executing them • Representation based on stochastic Petri nets captures dynamic, parameterized nature of actions • Generative model: action, recognition, planning , language • Walking: • bound to a specific walker with a direction or goal • consumes resources (e.g., energy) • may have termination condition(e.g., walker at goal) • ongoing, iterative action

49. Preshaping While Reaching to Grasp