Supporting Interaction in the R OBO C ARE Intelligent Assistive Environment

1. Project “RoboCare: A multi-agent system with fixed and robotic intelligent components” MIUR Law 449/97(yr 00) – 2003-2006 Supporting Interaction in the ROBOCARE Intelligent Assistive Environment Amedeo Cesta, Gabriella Cortellessa, Federico Pecora and Riccardo Rasconi ISTC-CNR [PST] Institute for Cognitive Science and Technology National Research Council of Italy Planning and Scheduling Team http://pst.istc.cnr.it

2. RoboCare initial aims “The objective of the project is to build a distributed multi-agent system which provides assistance services for elderly users at home. The agents are a highly heterogeneous collection of fixed and mobile robotic, sensory and problem solving components. The project is centered on obtaining a virtual community of human and artificial agents who cooperate in the continuous management of an enclosed environment.” Quote from the original project proposal (2001)

3. The goal • A distributed problem • Software Agents • Robotic Agents • Human Agents • All cooperate to provide services for human assistance

4. Different Research Aspects Acceptability Issues Active Supervision Framework Human-Robot Interaction Personalized Intelligent Assistance Robust Mobile Robotic Skills Distributed H/S Infrastructure

5. The robotic start (early 2004) • Pioneer 3DX mobile base with front sonar ring • Custom-built on top structure • Sonar based localization • Omnidirectional camera • 12” touch-screen • On board notebook (AMD 2000+ CPU)

6. The robotic evolution (2005) • Pioneer 3DX mobile base with front sonar ring • Sick Laser Range Finder • Algorithms • Path Planning • Top quality SLAM [Grisetti et al. ICRA-05] • But … difficult to sell as a “robotic companion”

7. behavioral requirements compilation scheduling problem physician family members Intelligent Services beginning: Non-Intrusive Supervision Caregiver specifications compiled into scheduling problems Caregiver vs.Active Supervision Framework interface

8. The silent observer (2005) • First loop between • Sensors (vision) • Intelligent supervision • Connection between position in the apartment and the action in execution • Report at the end of the day for physician and/or family members • T-REX • a basic scheduler • with enhanced modeling functionalities • and schedule execution monitoring features

9. The Interaction Skills The Motion Skills The Interactive Robot (2006) Use of multiagent technology • Endowed with human like I/O channels by engineering state of the art components • Face: Lucia (Piero Cosi, ISTC, Pd) • Voice: Sonic (Univ.Colorado) • Simple Interaction Manager Robust continuous behavior at home with person

10. Input/Output Channels Simple I/O Engine Knowledge for Interaction The Proactive Interactor (2006) • What is behind the interaction? Talking head Speech recognition Interaction Manager An additional set of active services ….

11. The Multiple Intelligent Systems

12. Robot Mobility Robot Interaction PLT T-REX … Service coordination • Loop • DCOP resolution: Asynchronous DCOP with Adopt-N • Read-variables • Application-computation • Write-variables • Soft constraints used to prefer healthy states and avoid dangerous states

13. Robot Mobility Robot Interaction PLT T-REX … Service coordination • Loop • DCOP resolution: Asynchronous DCOP with Adopt-N • Read-variables • Application-computation • Write-variables • Soft constraints used to prefer healthy states and avoid dangerous states

14. Robot Mobility Robot Interaction PLT T-REX … Service coordination • Loop • DCOP resolution: Asynchronous DCOP with Adopt-N • Read-variables • Application-computation • Write-variables • Soft constraints used to prefer healthy states and avoid dangerous states

15. Robot Mobility Robot Interaction PLT T-REX … Service coordination • Loop • DCOP resolution: Asynchronous DCOP with Adopt-N • Read-variables • Computation within applications • Write-variables • Soft constraints used to prefer healthy states and avoid dangerous states

16. Robot Mobility Robot Interaction PLT T-REX … Service coordination • Loop • DCOP resolution: Asynchronous DCOP with Adopt-N • Read-variables • Application-computation • Write-variables • Soft constraints used to prefer healthy states and avoid dangerous states

17. Managing assistant/assisted interaction - basics • On-demand interaction(user takes initiative) • Question / answering • Proactive interaction(RoboCare takes initiative) • Danger • Warning RoboCare as a Mixed-Initiative System

18. Managing interaction in RoboCare On-demand Proactive

19. On Demand Question-Answering • Query to the temporalized knowledge in T-REX • Very simple additional internal query capabilities

20. On Demand Question-Answering

21. Proactive Alarm for Danger • A reactive routine is activated with a precompiled plan • (go-to-place; try-interaction; call-emergency)

22. Proactive Alarm from Danger

23. Proactive Warning • Feedback from sensors is a key activator • Explanation triggered by T-REX temporal knowledge

24. Proactive Warning Which knowledge and reasoning support this?

25. Representing temporal prescriptions as a schedule • Activities and their mutual temporal constraints represented as a Simple Temporal Network • Dispatched for execution and monitored • Constraint violation triggers interaction

26. From Scheduler Knowledge to Interaction (1) the building blocks Constraint Violation Verbalization min Source A_i “Ai is taking place too soon.” max Source A_i “Ai is being delayed too much.” min A_i “Ai was too brief.” A_i “Ai is lasting too long.” max

27. From Scheduler Knowledge to Interaction (2) improving semantic precision Constraint Violation Condition Verbalization min Ai has not been executed (Ai and Aj are swapped) “Shouldn’t Ai be performed first?” A_i A_ j min Ai has been executed “Shouldn’t you wait a little longer before performing Aj?” A_i A_ j max Ai is the SOURCE activity (absolute time limit) “Expedite the execution of Aj .” A_i A_ j max Ai is not the SOURCE activity (relative time limit) “Too much time is passing between Ai and Aj .” A_i A_ j

28. From Scheduler Knowledge to Interaction (3) integrating causal domain analysis max max Source A_1 A_2 A_3 Verbalization: “Commence A2 , as it cannot be executed too late with respect to A1 , and A3 cannot begin later than a certain hour.” max A_3 A_1 A_2 Verbalization: “Stop A1 , as A2 must be immediately executed because it cannot be performed too late with respect to A3 .”

29. Related work • Intelligent assistants • Domain: Autominder [Pollack et-al, 2003] • Capability integration: similarities with CALO, CMradar, Diamond Help, etc. … although project scale quite different! • Specific aspect of interaction through user-oriented explanations • Using scheduling domain ontology to explain failures[Smith, Cortellessa, Hildum & Olher, 2004] • Reasoning on plan temporal network to generate explanations for plan impossibilities[Bresina & Morris, 2006]

30. On-going work: user evaluation

31. Quality movies to study old people

32. Current results

33. Conclusions • RoboCare has addressed the open challenge of integrating diversified intelligent capabilities to create a proactive assistant for everyday life in a domestic environment • This specific work has shown how a “silent observer” system able to passively monitor the execution of activities can be turned into a “proactive interactor” able to perform consistent advice-giving dialogue • Worth highlighting the particular use of the internal knowledge of a constraint-based scheduler as well as its capability of managing changes in the environment • Constraint violations determine when the system has to interact. The analysis and interpretation of the violation contribute to determine how to interact with the user