Socially Intelligent Robots

1. Socially Intelligent Robots Cynthia Breazeal MIT Media Lab Robotic Life Group

2. Robots turn 85 years old Posted May 31st 2006 11:38PM by Ryan Block Filed under: Robots Dear Robots, We're very sorry. It appears we missed your 85th birthday two days ago -- the anniversary of which is marked by the date Czech writer Karel Capek debuted his play R.U.R. (Rossum's Universal Robots) to its first audience in Prague. Yes, we know the concept of the automaton dates back much further, but we think it's well agreed upon that Capek's play marks the robot's entry into mass consciousness (as well as marking the first use of the word "robot"). No matter, we're just saying happy birthday, robots -- not because we fear you'll one day you'll subsume us in some dystopian nightmare of artificial intelligence gone terribly wrong, but because from Asimov to AIBO, from Roomba to Ri-Man, from QRIO to ASIMO, we just love ya. So happy birthday, happy birthday, happy birthday, robots, and when the day of reckoning comes, please remember: Engadget and its readers are your friends. All our love, Engadget

3. Robots have… • explored ocean depths, • mapped subterranean mines, • rescued natural disaster victims, • assisted surgeons with operations, • driven autonomously across the desert, • And even been to Mars…

4. What’s Next? • The next big frontier…society at large

5. Everyday Life with People and Robots…and its implication for design

6. People and Robots • Robots are not perceived as pure tools or appliances, but often as social actors -- over a wide range of morphologies and behaviors

7. Robots Evoke Human Social Responses “The Kismet Effect” New Scientist, 2005

8. Newsmaker: My friend, the robotCNET news.com, May 24, 2006 The PackBots have almost become members of military units, Angle said, recalling an incident when a U.S. soldier begged iRobot to repair his unit's robot, which they had dubbed Scooby Doo. "Please fix Scooby Doo because he saved my life," was the soldier's plea, Angle told the Future in Review conference last week in Coronado, Calif. For many reasons, people bond with robots in a way they don't bond with their lawn mowers, televisions or regular vacuum cleaners. At some point, this could help solve the looming health care problem caused by an enormous generation of aging people. Not only could robots make sure they take their medicine and watch for early warning signs of distress, but they could also provide a companion for lonely people and extend their independence.

9. Social Robots Socio-emotive Factors “Social as relationship” Future applications require robots to address Interactive Toys “Social as entertainment” the socio-emotive and psychological aspects of people, in long-term relations BANDAI “elder toys” Professional Service Robots “Social as interface” NEC “babysitters” OMRON “pets”

10. HRI, An Emerging Discipline An important goal of Human-Robot Interaction (HRI) is synergy of the human-robot system. Robots bring their own abilities that complement human strengths. It is not about equivalence (replacement), but compatibility with a typical human partner

11. Four Cornerstones of Social Robotics in HRI Interdependence Transparent Communication Teamwork Lasting Relationship Social Learning Social Intelligence Cognitive Compatibility Perspective Taking User Studies, Psychology & Social Development

12. Today’s Focus Robots, like humans, should leverage the social and environmental constraints in the real world to foster learning new skills and knowledge from anyone.

13. Personalization agents, Adaptive user interfaces {Lashkari, Metral, Maes, Collaborative Interface Agents, AAAI 1994} {E. Horovitz et al., The Lumiere project, UAI 1998} Active Learning, Learning with Queries {Cohn, Ghahramani, Jordan, Active learning with statistical models, 1995} {Cohn et al., Semi-supervised clustering with user feedback, 2003} Learning by Demonstration, Programming by Example {Voyles, Khosla, Programming robotic agents by demonstration, 1998} {Lieberman, Your Wish is my Command, 2001} {A. Billard, Special Issue of RAS on Robot Programming by Demonstration, 2006} Learning by Imitation {S. Schaal review in TICS 1999} {K. Dautenhahn & C. Nehaniv, Imitation in Animals and Artifacts, 2002} Animal training techniques {Stern, Frank, Resner, Virtual Petz, Agents 1998} {Blumberg et al. Integrated learning for interactive characters, SIGGRAPH 2002} {Kaplan et al., Robot clicker training, RAS 2002} Reinforcement Learning with humans {Isbell et al. Cobot: a social reinforcement learning agent, UAI 1998} {Evans, Varieties of Learning, AI Game Programming Wisdom, 2002} {Clouse, Utgoff, Teaching a Reinforcement Learner, ICML 1992} … and many more

14. How Do Ordinary People Teach a RL Agent? Most people don’t have experience with Machine Learning techniques, they have a lifetime of experience with social learning interactions that they bring to the table. We emphasize the need to consider and design to support the ways that people naturally approach teaching. And then design algorithms and systems that take better advantage of this

15. Experiments inSophie’s Kitchen • A “computer game” - players teach a virtual robot to bake a cake, by sending various messages with a mouse interface. Sophie learns via Q-Learning 30 steps ~10,000 states 2-7 actions/state Allows us to run many subjects on-line

16. Experiments inSophie’s Kitchen • A “computer game” - players teach a virtual robot to bake a cake, by sending various messages with a mouse interface. An object specific reward is about a particular part of the world

17. Initial Experiment • 18 people trained Sophie • They are given a description of the cake task, and told they can’t do actions but can help Sophie by sending FEEDBACK messages with the mouse • System logs time of state changes, agent actions, and any human feedback. We analyze games logs to understand people’s teaching behavior Thomaz & Breazeal RO-MAN 2006

18. Findings: Guidance • People tried to use the object specific rewards as FUTURE directed guidance.

19. Never About Most Recent Object Always About Most Recent Object % % % % % • Many object rewards not about the last object used Each player’s %Object Rewards about lastobject

20. Almost everyone gave rewards to the bowl or tray sitting empty on the shelf...a guidance reward. Number of People Zero rewards to Empty Bowl At least 1 reward to Empty Bowl

21. Findings: People Adapt Teaching to their Mental Model of Sophie • People gave more rewards after realizing their feedback made a difference • Interpreted Sophie’s behavior as being a “staged” learner • Adapted their teaching strategy accordingly

22. (Avg) (Avg) (Avg) Individual Individual Individual human rewards : agent actions

23. Guidance Initial Experiment } Transparency Asymmetry

24. Using Guidance in Sophie’s Kitchen Interactive Q-Learning Algorithm, baseline system slight delay to animate act and receive human reward }

25. Using Guidance in Sophie’s Kitchen

26. GuidanceExperiment Thomaz & Breazeal, AAAI 2006 • Hypothesis: Non-expert teachers can use guidance to improve agent’s performance • 27 subjects trained Sophie in two groups: • Using feedback only • Using both feedback and guidance • Again, system logs game play and logs are analyzed to understand teaching behavior

27. Effects of Guidance + >> only • 1-tailed T-tests show logs in guidance condition are significantly better than non-guidance

28. Transparency Guidance Initial Experiment } Asymmetry

29. Transparency • How can machine learners be Transparent? Teachers structure the environment and the task to help a learner succeed. Learners contribute by revealing internal state; helping the teacher maintain a mental model to make guidance more appropriate.

30. Sophie’s Gaze Behavior Interactive Q-Learning Algorithm modified to incorporate Guidance

31. Sophie’s Gaze Behavior

32. TransparencyExperiment • 52 subjects trained Sophie in an online version: • Feedback and guidance, no gaze • Feedback and guidance, Sophie gazing • Hypothesis: Learners can help shape their learning environment by communicating aspects of the internal process -- gaze will improve the human’s guidance instruction Thomaz et al., ICDL 2006

33. Sophie’s Gaze Behavior Results: Sophie’s gaze significantly improves the guidance received - more when uncertainty high and less when uncertainty is low. Uncertainty high: 3 or more choices Uncertainty low: 3 or less

34. Lessons • People bring their own teaching and learning experience to the task • Social factors of guidance and transparency • Collaborative process between teacher and learner improves performance • Agent can use transparency cues to improve its own learning environment by helping teacher form a better mental model • Adding gaze significantly improves the human’s Guidance

35. Summary