Integrating Intelligent Assistants into Human Teams

1. Katia Sycara The Robotics Institute Carnegie Mellon University Pittsburgh, PA 15213 (412) 268-8225 katia@cmu.edu www.cs.cmu.edu/~softagents Michael Lewis School of Information Sciences University of Pittsburgh Pittsburgh, PA 15260 (412) 624-9426 ml@sis.pitt.edu www.pitt.edu/~cmlewis Integrating Intelligent Assistants into Human Teams

2. Keith Decker Anadeep Pannu Vandana Verma Team Members CMU Liren Chen Somesh Jha Rande Shern Dajun Zeng Prasad Chalasani Kostya Domashnev Onn Shehory

3. Team Members U. of Pittsburgh Michael Lewis (PI) Terry Lenox Emily Roth

4. Goals Potential Impact for the Navy Approach Research Issues Progress Plan for Next Year Talk Outline

5. Overall Research Goal Increase the effectiveness of joint Command and Control Teams through the incorporation of Agent Technology in environments that are: • distributed • time stressed • uncertain • open (information sources, communication links and agents dynamically appear and disappear) Team members are distributed in terms of: • time and space • expertise

6. Impacts for Navy • Reduce time for a C2 team to arrive at a decision • Allow C2 teams to consider a broader range of alternatives • Enable C2 teams to flexibly manage contingencies (replan, repair) • Reduce time for a C2 team to form a shared model of the situation • Reduce individual and team errors • Support team cohesion and team work skills • Increase overall team performance

7. Transition Opportunities • Maritime Crisis planning • Target identification training • Air campaign planning • Strike planning • Aircraft maintenance

8. Overall Approach • develop an adaptive, self-organizing collection of Intelligent Agents (the RETSINA infrastructure) that interact with the humans and each other. • integrate multimedia information management and decision support • anticipate and satisfy human information processing and problem solving needs • perform real-time synchronization of human actions • notify about significant changes in the environment • adapt to user, task and situation • develop model libraries of individual and team tasks • develop verifiable useful human-agent interaction techniques

9. Overall Research Issues • Agents and Agent Interactions • Human Agent Interaction • Information Filtering and Integration

10. Overall Research Issues: Agents and Agent Interactions • interleaving planning, replanning, execution monitoring and information gathering in a multiagent setting • single agent architecture and self-awareness • agent coordination scheme • finding appropriate agents • agent interoperability • agent-to-agent task delegation protocols • learning through agent interactions

11. Overall Research Issues: Human Agent Interaction • agent-based team aiding • functional allocation between humans and agents (insert agents into military simulations and perform controlled experiments with human subjects to assess utility) • human-agent trust • development of task models (graphical task editor) • user-guided instantiation of agents (agent editor)

12. Insert TeamAiding.ppt

13. Overall Research Issues: Information Filtering and Integration • learning and tracking multiple interests of users • increase relevance of retrieved information (refinement key words, relevance feedback, summary of most important information in documents) • detecting ``interesting'' patterns from multiple data sources • information integration and conflict resolution

14. Retsina Functional Organization

15. Characteristics of RETSINA Agents • Agents act autonomously to accomplish objectives • Goal-directed • Taskable • Running unassisted for long periods • Proactive & Reactive

16. Characteristics of RETSINA Agents (Contd.) • Agents engage in peer-to-peer interactions • Agents are taskable, i.e. users or other agents can delegate tasks to them, user acceptability and trust an important issue • Can interact as cooperative teams or self-interested individuals • Interaction protocols • Coordination Strategies • Negotiation Protocols • Agents adapt to their environment, user, task and each other • Adapt both at the individual level and at the societal level • Employ Alternate Methods • Learn from (and about) users and each other

17. Progress • RETSINA system infrastructure development • Java implementation • RETSINA agent architecture • increased planning sophistication in individual agents • Middle agents • Agent interaction protocols

18. Middle Agent Types Service providers have capabilities and service parameters Service requesters have service request and preferences Service Parameters Initially Known By

19. Retsina Agent Architecture

20. RETSINA Planning Mechanisms • hierarchical task network-based formalism • library of task reduction schemas • alternative task reductions • contingent plans, loops • incremental task reduction, interleaved with execution • information gathered during execution directs future planning • resource and temporal constraints

21. A task Structure (Advertisement Task Structure)

22. Progress (Contd.) • Agent interoperability • language for capability advertisement (Aardvark) • agent name server and distributed matchmakingª • Human Agent Interaction • Task Editor • Agent Editor • Human Agent Trust • Team TANDEM experiments ________________________ ª www.cs.cmu.edu/~softagents/retsina/ans

23. Insert Aardvark.ppt:language for capability advertisement

24. Insert Interact.ppt: Agent Editor

25. Progress (Contd.) • Applications • Information filtering: Webmateª, DVINA • Agents in team aiding: ModSAF, multiagent air patrol, agent-aided aircraft maintenance ___________________________ ª www.cs.cmu.edu/~softagents/webmate  This application is done in collaboration with the CMU wearable computer project.

26. ModSAF Vision

29. Insert AirMain.ppt:Aircraft Maintenance Task

30. Overview of the WebMate System • Use the multiple TF-IDF vectors to keep track of user interests in different domains which are automatically learned • Use the trigger pair model to automatically extract relevant words for refining search • The user can provide multiple pages as relevance guidance for information search

31. Insert WebMate.ppt(more detailed description)

32. Insert WebMateDemo.ppt(detailed description of WebMate demo)

33. Overview of Informedia • One of the six Digital Libraries Initiative projects funded by the NSF, DARPA, NASA and others in collaboration with WQED • A multimedia library that will consist of over one thousand hours of digital video, audio, images, text and other related materials • Uses combined speech, language and image understanding technology to transcribe, segment and index the linear video.

34. Plans for Next Year • Continue enhancing the functionality of individual agents (e.g., more sophisticated planning mechanisms) • Improve the robustness of the RETSINA infrastructure • Finish the implementation of the agent advertisement language (Aardvark) • Refine agent task delegation framework, particularly contingent task delegation • Investigate situation-dependent agent coordination strategies • Investigate information- and action-based conflict resolution • Expand the ModSAF team-aiding scenarios by introducing agents of additional types and functionalities

35. Plans for Next Year (Contd.) • Develop explicit agent tasking mechanisms • Identify appropriate indexing mechanisms for task structure cases • Expand the functionalities of agent editor • Automatically learn individual and team coordination patterns from team activity traces

36. Plan for Integrating the Parts of CMU MURI • Work with U. of Pittsburgh to identify additional agent requirements needed for agent-based team aiding • U. of Pittsburgh will test the effectiveness of agent-based team aiding in ModSAF scenarios with human subjects • Incorporate multimedia information from Informedia into agent-based team aiding • Use the wearable computers as the platform for running the collaborative aircraft maintenance agents