A Path to Robot Consciousness via Social Cognition: Agency and Intention

1. A Path to Robot Consciousness via Social Cognition: Agency and Intention Peter Ford Dominey, Robot Cognition Laboratory Stem Cell and Brain Research Institute INSERM U846, France, peter.dominey@inserm.fr ANR Comprendre AMORCES

2. Outline • More questions than answers…. • Motivation – Can robotics clarify our ideas about consciousness? • Context – relation of consciousness and meaning • Sytem architectures • Progress in Meaning and cooperation in robots • Action and language • Shared plans • “Meaning” • Simulation as meaning • Discussion

3. Motivation – cooperative robots • When a robot can do things like …. • Learn the name of an object, • Learn a new action with that object • Tell you what it knows • Ask questions when it doesn’t understand • Anticipate what you are about to do • … can it be useful in the scientific study of consciousness? Dominey, Metta, Nori, Natale (2008) IEEE Int. Conf. On Humanoid Robotics

4. Context – consciousness meaning and robotics • Jeffrey Gray: • Conscious experiences are imbuded with meaning; computers cannot (without human interpretation) compute meaning; therefore, computers cannont be conscious • Robots differ from computers in that they are endowed with just such behavioral dispositions [to compute meaning]… [but] we should remain doubtful whether they are likely to experience conscious percepts… that will depend on how the trick of consciousness is done. • Daniel Dennett: • … set out to make a robot that is theoretically interesting independent of the philosophical conundrum about whether it is conscious. Such a robot would have to perform a lot of the feats that we have typically associated with consciousness in the past, but we would not need to dwell on that issue from the outset.

5. Brain interacts unconsiously with world Constructs a simulation Simulation is consiously perceived Public cognitve space Private cognitive space Private bodily space Real Unperceived External World Cybernetic interactions Constrains The unconscious Brain Simulates Inner cognitive experiences (thoughts, Images…) Private cognitive space Experienced external world, including body from outside Public cognitive space Inner bodily sensations, feelings Private body space Conscious experience Towards a System Architecture of Consciousness The different spaces of conscious experience, from Gray 2004

6. System architecture for a Cooperative Robot External world Real Unperceived External World Cybernetic interactions Constrains The unconscious Brain Simulates Inner cognitive experiences (thoughts, Images…) Private cognitive space Experienced external world, including body from outside Public cognitive space Inner bodily sensations, feelings Private body space Conscious experience Simulates Cybernetic interaction Lallée, Lemaignan, Lenz, Melhuish, Natale, Skachek, van Der Zant, Warneken, Dominey (submitted)

7. Linking Grammar Learning with Vision for Event Description and Interrogation Dominey & Boucher (2005) Artificial Intelligence Vision, "Meaning" Extraction CCD Camera Event(Agent, Object, Recipient) Spoken Language Interface (CSLU RAD) Human narrator Sentence E(A,O,R) Grammatical Construction Model: Sentence to Meaning Gave(moon, cylinder, block) The moon gave the cylinder to the block. The block was gave the cylinder by the moon. The cylinder was gave to the block by the moon.

8. Meaning in Cooperation: Language-Based interaction with the Robot Apprentice Cooperative Table Assembly Scenario • Robot Helps Users to Assemble a Table • Functional Requirements - The robot should: • Respond to human spoken commands with simple behaviors • Open left hand, turn right,.. • Grasp(X): X in <visible> • Learn complex behaviors constructed from the primitives • Give me the orange leg • Hold the table Kawada Industries HRP-2 Platform CNRS-AIST Joint Robotics Laboratory LAAS, Toulouse, France

9. Spoken Language Programming Part of Joint Robotics Laboratory project, AIST/CNRS Dominey, Mallet, Yoshida (2007) IEEE ICRA, IEEE Humanoids, (2009) IJHR

10. Spoken Language Programming • Method • Hand coded « primitives » (postures) • and grasp(x) procedure • Sequenced together via spoken language • Macro Programming • Humanoids 2007 • Procedure with Arguments • ICRA 2007 • Generalizes to different tasks • Assembly, disassembly Part of Joint Robotics Laboratory project, AIST/CNRS Dominey, Mallet, Yoshida (2007) IEEE ICRA, IEEE Humanoids, (2009) IJHR

11. Automatic Learning, and Anticipation • User guides action by spoken language Pseudo-code: • At each command: • If current subsequence is in InteractionHistory • L1 – anticipate speech • L2 – propose next action • L3 – take initiative • Increment L • Else get next command • Execute • Update Interaction History Dominey, Metta, Nori, Natale (2008) IEEE Int. Conf. On Humanoid Robotics

12. Progressive effects of Learning Speech anticipation With leg 2 Action proposition With leg 3 Robot initiative With leg 4 First experiece With leg 1 Mean Execution time for a single action (sec)

13. But,…. Cooperation Requires Shared Plans Tomasello M, Carpenter M, Call J, Behne T, Moll HY (2005) Understanding and sharing intentions: The origins of cultural cognition, Beh. Brain Sc;. 28; 675-735. Dominey PF (2005) Toward a construction-based account of shared intentions in social cognition. Behavioral and Brain Sciences 28:696-+.

14. Learning Shared Plans from Observation • Perceive action • Attribute agency • Form shared plan • Ordered list of (agent, action) pairs • Use it in cooperation • Role reversal • Limitations: robot doesn’t know “why?” Box Toy Larry (left) Robert (right) Lallee, Warneken, Dominey (2009) EpiRob, Humanoids Workshop

15. Approaching Meaning: Linking actions to states • Learn to recognize action via • Dynamic perceptual primitive patterns • Visible, Contact, Moving • Enrich this with knowledge of • Enabling state (initial) • Resulting State (final/goal) • « Derived predicates » • Derived Predicates • On, Under • Has • Reasoning: • Forward chaining from current state to goal • Backward chaining from goal to current state Lallee et al. Submitted Frontiers NeuroRobotics

16. Submitted to IROS 2010

17. Meaning: Linking actions to states Learn the name of a new action Learn the relation between “cover” and “on” Demonstrate transfer to a new enactment

18. Meaning: Linking actions to states « Cover Arg1 with Arg2 » • Learn to recognize action via • Dynamic perceptual primitive patterns • Visble, Contact, Moving • Enrich this with knowledge of • Enabling state (initial) • Resulting State (final/goal) • « Derived predicates » • Derived Predicates • On, Under • Has • Reasoning: • Forward chaining from current state to goal • Backward chaining from goal to current state

20. Language and meaning • Language can augment meaning derived from vision • Explaining derived states • Explaining causal relations Language Action Vision Meaning Initial State –Action – Final State-

21. Brain interacts unconsiously with world Constructs a simulation Simulation is consiously perceived Public cognitve space Private cognitive space Private bodily space Real Unperceived External World Cybernetic interactions Constrains The unconscious Brain Simulates Inner cognitive experiences (thoughts, Images…) Private cognitive space Experienced external world, including body from outside Public cognitive space Inner bodily sensations, feelings Private body space Conscious experience Towards a System Architecture of Consciousness The different spaces of conscious experience, from Gray 2004

22. Hybrid Embodied-Propositional System • Propositional system manipulates compact, « symbolic » representations of actions, plans • Embodied system employs « situated simulations », unpacking the compact representations • Language allows the speaker to « direct the film » that unfolds in the listener’s mind Madden, Hoen, Dominey (2009) A Cognitive Neuroscience Perspective on Embodied Language for Human-Robot Cooperation, Brain and Language

23. Towards Embodiment: Learning to predict the perceptual consequences of a motor action • Grasping requires vision of the hand • The hand has “infinite” postures • How to reduce the visual recognition space?

24. Perceptual-Motor Learning Vision Proprioception (joint angles) … Hand Posture 3 Hand Posture 2 Hand Posture 1 Area 5 MMCM • Associates Distinct Patterns of Joint Angles with the Corresponding Image of the hand Multi Modal Convergence Maps (MMCM) Topographical Organisation (Kohonen SOM like)

25. Training: Vision-Proprioception pairs every 100ms for 8 min 6 joints moved in cyclic pattern ~16 cycles Experimental Effects on Performance iCub (Robotcubproject) Vis-Motor Learning OFF ON • Using Vis-Motor Learning has a significant effect on visual recognition time • F(1,39) = 418, p < 0.0001 Mean Recogntin time (+ SD, SE) S. LALLEE1, G. METTA2, L. NATALE2, U. PATTACINI2, *P. F. DOMINEY1; (2009) Proprioception of the hand contributes to visual recognition speed and accuracy: Evidence from the Multi-Modal Convergence Map model of Parietal Cortex Area 5, Society for Neuroscience Abstract

26. Return to the neurophysiology: of language, action and cooperation • Ventral stream (green) phonological and lexical processing (STS, MTG, PFCv) • Dorsal stream (Blue) grammatical integration/unification and sensorimotor interface, simulation (TPJ, PFCdl, PPC) • Complex Event Recognition (Orange) social cognition, cooperation (STS), Agency, Simulation, Intention,Teleological reasoning, Perspectivie taking

27. Discussion • A Path to Robot Consciousness via Social Cognition: Agency and Intention • Manipulates representations of self, other • Perspective taking • Recognition of agency • Designation of intention based on action recogntion • Use of language to express beliefs • Can robot studies be used to address any aspects of consciousness? • What is the roadmap? • Action and language • Shared plans • “Meaning” • Simulation as meaning • Self – body scheme • How would we define robot consciousness?

28. Acknowledgements • Collaborators • Jocelyne Ventre-Dominey • Michel Hoen • Carol Madden • Felix Warneken • Toshio Inui • Frank Ramus • Anthony Mallet • Eiichi Yoshida • Giorgio Metta • Giulio Sandini • Francesco Nori • Lorenzo Natale • Ugo Pattacini • Research Organizations • CNRS • INSERM • Funding • CHRIS (EU FP7) • Organic (EU FP7) • French ANR • Amorces (PsiRob) • Comprendre (Blanc) • RobotCub (EU FP6) • iCub Open Call • Students/ PostDocs • Jean-David Boucher • Stephane Lallee • Mehdi Khamassi • Xavier Hinaut • Anne-Lise Jouen

29. The Cube Game Boucher,Ventre-Dominey, Dominey (INSERM-RCL, Lyon) Fagel, Bailly (GIPSA-Lab, Grenoble)

30. Real Unperceived External World Cybernetic interactions Constrains The unconscious Brain Simulates Inner cognitive experiences (thoughts, Images…) Private cognitive space Experienced external world, including body from outside Public cognitive space Inner bodily sensations, feelings Private body space Conscious experience The different spaces of conscious experience (From Gray, 2004, Fig 1.1)

31. Integration: Hybrid Telelogical/Embodied Cognitive System Teleological reasoning Embodied representation/simulation Madden, Hoen, Dominey (2009) Brain and Language iCub project – Lyon, June 2009

32. Learning Actions and their Consequences

34. Meaning: Linking actions to states « Cover Arg1 with Arg2 » • Learn to recognize action via • Dynamic perceptual primitive patterns • Visble, Contact, Moving • Enrich this with knowledge of • Enabling state (initial) • Resulting State (final/goal) • « Derived predicates » • Derived Predicates • On, Under • Has • Reasoning: • Forward chaining from current state to goal • Backward chaining from goal to current state

35. RobotCub – A Great Achievement

36. Distinctions and Definitions • Public vs Private • Public – « the red book on the shelf » • Private – thoughts, feelings • Inner vs External

37. Learning from Experience:Automatic Learning, and Anticipation • Replace Explicit Programming • Use On-line, automatic learning of behavior via continuous comparison with the Interaction History Dominey, Metta, Nori, Natale (2008) IEEE Int. Conf. On Humanoid Robotics

38. Real Unperceived External World Cybernetic interactions Constrains The unconscious Brain Simulates Inner cognitive experiences (thoughts, Images…) Private cognitive space Experienced external world, including body from outside Public cognitive space Inner cognitive experiences (thoughts, Images…) Private body space Conscious experience

39. Perceptual-Motor Learning Proprioception (joint angles) Vision … Hand Posture 3 Hand Posture 2 Hand Posture 1 Area 5 MMCM • Associates Distinct Patterns of Joint Angles with the Corresponding Image of the hand Multi Modal Convergence Maps (MMCM) Topographical Organisation (Kohonen SOM like)

40. We want to cooperate with this guy… But How?

41. Plan • Define a Context for Cooperation • Build some basic tools: Spoken Language Programming • Learning • Automatically from one’s own experience • Shared plans from Observation • The meaning of actions • Towards Embodyment • A Hybrid Propositional & Embodied Cognitive System