Supporting Technology for Group Interaction

1. Supporting Technology for Group Interaction Howard Shrobe MIT AI Lab Oxygen Workshop, January, 2002

2. A Three Pronged Attack on Collaborative Knowledge Management • Ubiquitous human-centered, perceptually enabled environments. • An adaptive infrastructure that understands the context and content of problem solving discourse (at least a little) • It accurately indexes all information. • It distributes information to those who can use it. • It can find resources (information and people) to help a collaboration. • Tools informed by an understanding of the domain and the organization. • An “almost expert” (apprentice) system • Which knows when to ask questions: • What it doesn’t understand is often what’s most important • What isn’t obvious is what needs to be documented and distributed • Documentation almost for free by embedding a helpful computational system in our normal work flow

3. I think I can make it work that way, but does it affect anything else! The side-tracker interface would be much clearer if it worked this way! Clarify Interface That should work, as long as it doesn’t break the special hack for managers. I’ll look into that, you all can start the coding and doc updates Critique Propose Code Brainstorm Document An Example of Group Interaction and Shared Plans Plan

4. Shared Semantic Web Knowledge Base Shared Plans Shared Info Knowledge acquisition Inference Facilitator Resource Discovery KB Goal Plan Plan Haystacks Knowledge-Based Infrastructure Helps People Collaborate: • By acting as an assistant to meeting facilitators it can help to capture important information such as: • Issue, positions, arguments for and against positions • Commitments, action items • Video and audio transcripts of meetings • By understanding the role of individuals within the organization it can help to decide who should see what information. • By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”. Facilitator Agent

5. Shared Semantic Web Knowledge Base Shared Plans Shared Info Knowledge acquisition Inference Facilitator Resource Discovery KB Goal Plan Plan Haystacks The Oxygen Collaboration Infrastructure:Knowledge Based Collaboration Webs • Information nodes: • Goals of the project • Proposed methods for the goals • Arguments in favor and against methods • Documents supporting these arguments • Node format: • Much of the content is opaque to the system (e.g. Multi-media fragments) • Some slots are understood by the system • And, short natural language annotations can be attached, parsed and understood by START • Formal representations are also possible • Relational information: • Links showing the relationships between the information nodes • Link types are meaningful to the system • Organizational descriptions: • Resource descriptions including capabilities, roles and interests (for personnel) • Process goals and plans Meeting Manager email web

6. Oxygen Can Support Human Collaboration by Making Simple Inferences • Oxygen deduces new information using: • The background knowledge base • The types of the links • The slots of node structures that are understood • The semantic content of the node annotations • Descriptions of the resources and people in the organization • Oxygen routes information and discussion topics to those whose: • Organizational role requires it • Interests suggest it • Capabilities and skills might be useful • Oxygen posts new goals and initiates new discussion processes to address these goals. • Oxygen supports human interactions in a manner relevant to the organization’s problem solving context

7. Key Oxygen Components Make This Much Easier by Bringing Computers Into Our World Pointing Camera Pointing Camera Video Displays Microphone Array Tracking Cameras

8. People Meeting Structure Agenda Item Start time; End time: Agenda Item Start time; End time: Action Items Agenda Item Start time; End time: Commitment Start time; End time: Who: Deadline Discourse Supporting Argument Start time; End time: Commitment Start time; End time: Who: Deadline Discourse Discourse Structure Position 1 Start time; End time: Commitment Start time; End time: Who: Deadline Discourse Issue 1 Start time; End time: Position 2 Start time; End time: Refuting Argument Start time; End time: Issue 2 Start time; End time: A Meeting Web Video/Audio Transcript (quicktime movie)

9. 5 Keys Challenges for Adaptive Interfaces • Providing a practical level of knowledge representation that enables groups interactions and grounding in the real world of space and time • Providing services in a multi-user environment while making optimal use of the currently available resources • Recovering from equipment failures, information attacks, etc. • Coordinating and fusing information from many sensors and modalities • Capitalizing and recognizing context • Maintaining Security and Privacy and trading these off against other goals

10. Challenge 1: Grounding in Real-World Semantics • We want to build applications that service many individuals and groups of individuals • These people will move among many physical spaces • The devices and resources they use change as time progresses • The context shifts during interactions • The relevant information base evolves as well.

11. Research Agenda: Knowledge Representations • People • Interests, skills, responsibilities, organizational role • Organizations • Members, structure • Spaces • Location • Subspaces • Devices and resources • Resources • Information nodes • Topic area, place in ontology, format • Services • Methods, parameter bindings, resource requirements • Agents • Capabilities, society, acting on behalf of whom • Events • People identification, motion in new space, gestures

12. Challenge 2: Adaptive Resource Management • In most systems, applications are written in terms of specific resources • (e.G. The left projector in michael’s office, or worse yet physical address). • This is in conflict with • Portability across physical contexts • Changes in equipment availability across time • Multiple applications demanding similar resources • Need to take advantage of new resources • Need to integrate mobile devices as they migrate into a space • Need to link two or more spaces • What is required is a more abstract approach to resources in which no application needs to be tied to a specific device.

13. Each method requires different resources Each method binds the settings of The control parameters in a different way Resource1,1 Service Control Parameters Resource Cost Function Method1 Resource1,2 User’s Utility Function Abstract Service Method2 Resource1,j The binding of parameters has a value to the user Methodn User requests A service with certain parameters The resources Used by the method Have a cost Each service can be Provided by several Methods Net benefit The system adapts by having many plans for each service The system selects the method which maximizes net benefit

14. Resource1,1 Security Policies Resource1,2 Resource Cost Function Method1 Resource1,j Method2 User’s Utility Function Abstract Service Methodn Net Benefit Challenge 6: Security and PrivacyIs Addressed by This Infrastructure Security and Privacy Issues Are Addressed by Factoring in the Cost of Violating a Security or Privacy Policy

15. Challenge 3: Robustness and Recovery From Failures • The intelligent room renders services by translating them into plans involving physical resources • Physical resources have known failure modes • Each plan step accomplishes sub-goals needed by succeeding steps • Each sub-goal has some way of monitoring whether it has been accomplished • These monitoring steps are also inserted into the plan • If a sub-goal fails to be accomplished, model-based diagnosis isolates and characterizes the failure • A recovery is chosen based on the diagnosis • It might be as simple as “try it again”, we had a network glitch • It might be “try it again, but with a different selection of resources” • It might be as complex as “clean up and try a different plan”

16. I need to ask a question of a systems wizard Recovering From Failures I don’t see light on the screen • Plan 1: • Locate A systems wizard in the E21 • Monitor: check that person is still there • Turn on the selected projector • Monitor: check that projector turned on • Project the message • Done • Monitor: check that the person noticed the message Plan Breakdown Projector-1 must be broken. We’ll try again, but using Projector-3. I see sally by the screen

17. Frobulate the sidetracker Challenge 4: Exploitation of Context • The attention of the system should be focused by what people in the facility do and say. • Speech recognition should be biased in favor of things going on in the facility • Speech system is made up of many “ grammar fragments” • Grammar fragments are activated (and deactivated) when perceptual events (visual or speech) suggest they should be • Visual Interpretation should also be biased by context Activate the Drawing Grammar If A Person approaches A device ?x and Grammar ?y is relevant to ?x Then Activate Grammar ?y Sally is near The Whiteboard White Board Place Manager The Drawing Grammar Is Relevant to the Whiteboard

18. Challenge 5: Perceptual Coordination • Separate the implementation of perceptual tasks from the uses to which perception is put • All perceptual modules advertise the class of “behavioral events” they are capable of recognizing and signaling • These events are organized into a taxonomy • Some events are “ close to the physics” (e.g. an object was observed at location <x,y>) • Some events are more abstract (e.g. a person is near the white-board) • The same event can be signaled by quite different perceptual modules (e.g. both face and voice recognition can localize a person). • Other modules register their interest in certain classes of events • Requests at a higher level in the taxonomy subsume lower level events • Modules which receive low-level events may register for and collate many different classes of events • They combine these and signal higher-level events

19. Interested in the location of individuals Signal people approaching the whiteboard White Board Context Manager Interested in face location I signal the location of individuals Face Recognition Interested in body motion Signal the location of individuals Interested in body motion I signal face location Voice Identification Face Spotter Signal body motion Visual Tracker Research Agenda: A Dynamic Event Bus For Perceptual Coordination

20. Shared Semantic Web Knowledge Base Shared Plans Shared Info Knowledge acquisition Inference Facilitator Resource Discovery KB Goal Plan Plan Haystacks Facilitator Agent Oxygen’s Knowledge-Based Infrastructure Helps People Collaborate: • By acting as an assistant to meeting facilitators it can help to capture important information such as: • Issue, positions, arguments for and against positions • Commitments, action items • Video and audio transcripts of meetings • By understanding the role of individuals within the organization it can help to decide who should see what information. • By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”. • By understanding the context within which people act, it can react in appropriate ways.