Learning Agents Center Computer Science Department George Mason University

1. IT 803 Spring 2004 – Mixed-Initiative Intelligent Systems – Prof. G. Tecuci COLLAGEN: Mixed-Initiative Interaction with a Collaborative Agent Dorin Marcu 02-09-2004 Learning Agents Center Computer Science Department George Mason University

2. Presentation Outline • A brief introduction to COLLAGEN • An air travel application built with COLLAGEN • A more detailed look at COLLAGEN: • architecture • communication • shared awareness • tasks • control • evaluation • Summary of the main ideas in COLLAGEN • Lessons learned • References

3. Research Motivation • current systems lack support for the user’s problem solving process during extended periods of time: • the order in which actions must be performed by the user and the system is often inflexible • it’s hard to recover from mistakes • each system has its own interaction conventions

4. Explored Solution • support the problem-solving level of human-computer interaction via a collaborative software agent • develop a new paradigm for human-computer interaction which explicitly supports the user’s problem-solving process based on current theories of collaborative discourse

5. Assumption A human-computer interface based on familiar human discourse rules and conventions will be easier for people to learn and use than one that is not.

6. Collaborative Interface Agents From (Rich and Sidner, 1998), page 317

7. COLLAGEN: A Collaboration Manager • A collaboration manager is a software component that: • mediates the interaction between a software interface agent and a user • keeps track of the linguistic and attentional state of a discourse (similarly to a discourse manager) • keeps track of the collaborative intentions of the participants

8. COLLAGEN: A Collaboration Manager (contd.) • A collaboration manager is less than a fully automated planning system because: • it does not by itself decide what the agent should do or say next (though it may provide some candidates) • it provides a representation for recording the decisions that the agent has made and communicated.

9. Definition of Mixed-Initiative in COLLAGEN Mixed-initiative reasoning in COLLAGEN can be defined as a discourse-based collaboration between a human user and an interface agent that attempt to achieve shared goals by decomposing them into sub-goals and primary actions for which they have complementary solving capabilities.

10. Characteristics of Mixed-Initiative in COLLAGEN • one human – one agent, discourse-oriented collaboration • partial, task-oriented mixed-initiative system: • some tasks have fixed-initiative flags, specifying which participants can solve them • the human participant has precedence over the agent, and can ignore the contributions of the agent • initiative taking: • proposing a goal • proposing a recipe for solving a goal • solving a (part of a) recipe • delegation

11. Main Features of COLLAGEN • both participants know and intend that all their actions are observed: • reporting communication (‘I have done x’) • direct observation

12. Main Features of COLLAGEN (contd.) • the mixed-initiative capabilities of the agent arise from the interplay of two sources: • application-independent algorithms and data structures in COLLAGEN • application-specific code and libraries in the agent • a library of recipes that specify the typical steps and constraints for achieving certain goals • arbitrary pattern-action rules

13. Main Features of COLLAGEN (contd.) • supports mixed-initiative by: • interpreting discourse acts • maintaining a model of the achieved and expected tasks and goals of the user and agent • the user makes the final decision for the problem-solving process and can ignore the contributions of the agent

14. Main Features of COLLAGEN (contd.) • the interaction model is based on a formal representation of the mutual beliefs about the goals and actions to be performed, and the capabilities, intentions, and commitments of the participants

15. Presentation Outline • A brief introduction to COLLAGEN • An air travel application built with COLLAGEN • A more detailed look at COLLAGEN: • architecture • communication • shared awareness • tasks • control • evaluation • Summary of the main ideas in COLLAGEN • Lessons learned • References

16. Air Travel Planning System You are a Boston-based sales representative planning a trip to visit customers in Dallas, Denver, and San Francisco next week. You would prefer to leave on Wednesday morning, but can leave on Tuesday night if necessary. Your customer in Denver is only available between 11 a.m. and 3 p.m. on Thursday. You would prefer to fly as much as possible on American Airlines, because you have almost enough frequent-flier miles to qualify for a free trip this summer. You absolutely must be home by 5 p.m. on Friday to attend your son’s piano recital.

17. Air Travel Planner Interface From (Rich and Sidner, 1998), page 319

18. Air Travel Planner Interface (contd.) From (Rich and Sidner, 1998), page 320

19. Traditional Use of Planner Seven visitors and staff members were asked to solve this and similar problems and their behavior was recorded via informal notes and the logging facilities built into the application A typical problem solving session lasted about 15 minutes and entailed about 150 user actions (mouse clicks).

20. User Problems with the Traditional Planner Various forms of getting stuck and getting lost: • trouble knowing what to try next when the trip has been over- or under-constrained • trouble keeping track of which combinations of routes and constraints were already examined • workflow interruptions caused by the use of application’s functions (e.g. “Snapshot” – save context)

21. Collaborative Interaction Example (Rich and Sidner, 1998), page 323

22. Collaborative Interaction Example (contd.) (Rich and Sidner, 1998), page 323

23. Presentation Outline • A brief introduction to COLLAGEN • An air travel application built with COLLAGEN • A more detailed look at COLLAGEN: • architecture • communication • shared awareness • tasks • control • evaluation • Summary of the main ideas in COLLAGEN • Lessons learned • References

24. The Architecture of COLLAGEN (Rich and Sidner, 1998), page 335

25. The Interface Agent COLLAGEN does not provide tools for building a complete agent – different types of agents can be supported (rule-based expert systems, neural nets, or a completely ad hoc collection of code) COLLAGEN provides a generic framework for recording the decisions made and communicated by the agent (and the user), but not for making them.

26. Question Why would such a generic framework be useful? Effective?

27. Answer Why would such a generic framework be useful? Effective? Because it will allow the reuse of components for: - building collaborative agents - application-independent discourse manager

28. The Execution Cycle • a communication or observation event arrives at the discourse interpretation module • the discourse interpretation module updates the discourse state • a new agenda of expected communication and manipulation acts is computed by the discourse generation module • the agent may decide to select an entry in this new agenda for immediate execution (according to the agent’s initiative strategy) • the user communication menu is updated with all the communication actions in the agenda for which the actor is either unspecified or the user

29. The Default Agent Initiative Strategy The default agent implementation that is included in COLLAGEN always chooses to perform the highest priority action in the current agenda for which the actor is either unspecified or itself. The priorities are manually associated with actions. Previous experiments also used pair-wise comparison rules. (Rich, 2002)

30. The Architecture Issue • COLLAGEN has a generic framework for developing interface agents that can collaborate with human users to solve shared goals. • The generic framework must be customized for a specific application by developing application-dependent : • recipes • action models • methods for performing actions and for observing actions performed by the user • collaboration behavior

31. Presentation Outline • A brief introduction to COLLAGEN • An air travel application built with COLLAGEN • A more detailed look at COLLAGEN: • architecture • communication • tasks • shared awareness • control • evaluation • Summary of the main ideas in COLLAGEN • Lessons learned • References

32. Window Sharing • both the user and the agent have a dedicated “home” window that is used for communication between them • each home window contains an identifying face and has an associated cursor • the home windows are serviced by separate processes to support asynchronous mixed-initiative interaction (Rich and Sidner, 1998), page 321

33. Types of Agent Communication • printing English text in the agent’s home window • acting on the application’s interface with its cursor while approved and observed by the user The user can ignore the suggestions proposed by the agent. (Rich and Sidner, 1997), page 290

34. Types of User Communication • selecting from a menu of communications expected by the discourse interpretation algorithm (that were generated from the underlying model of discourse) • acting directly on the application with his/her cursor (the agent always observes the user’s actions through a generic layer in the application that mirrors semantic actions into the input buffer of the agent process) (Rich and Sidner, 1998), page 325 (Rich and Sidner, 1998), page 319

35. The Communication Issue COLLAGEN is based on a discourse theory of collaboration in which: • the agent communicates by: • generating natural language text or user-selectable menus/buttons • performing direct actions in the interface that the user can observe • the user communicates by: • selecting agent generated menus/buttons • performing direct actions in the interface that the agent can observe

36. Presentation Outline • A brief introduction to COLLAGEN • An air travel application built with COLLAGEN • A more detailed look at COLLAGEN: • architecture • communication • shared awareness • tasks • control • evaluation • Summary of the main ideas in COLLAGEN • Lessons learned • References

37. Collaborative Discourse Theory Collaborationis a process in which two or more participants coordinate their actions toward achieving shared goals. Most collaborations between humans involves communication. Discourseis a technical term for an extended communication between two or more participants in a shared context, such as a collaboration.

38. Collaborative Discourse Theory (contd.) • Three interrelated types of collaborative discourse structures (Grosz and Sidner, 1986): • intentional structure (formalized as partial SharedPlans) • linguistic structure (includes the hierarchical grouping of actions into segments) • attentional structure (captured by a focus stack of segments)

39. Intentional Structure for the Air Travel Planner • the user knows the constraints on travel • the agent has access to a data base of all possible flights • both participants have a common goal (to find an itinerary that satisfies the constraints)

40. Intentional Structure for the Air Travel Planner (contd.) • both participants: • have agreed on a sequence of actions (a recipe) to accomplish the common goal (e.g., choose a route, specify some constraints on each leg, etc.) • are capable of performing their assigned actions • intend to do their assigned actions • are committed to the overall success of the collaboration (not just the successful completion of their own parts).

41. SharedPlans SharedPlans is a formal representation of the participants’ mutual beliefs about the goals and actions to be performed, and of their capabilities, intentions, and commitments.

42. Representation of SharedPlans in COLLAGEN (Rich and Sidner, 1998), page 331

43. Features of SharedPlans • partial (due to incomplete knowledge) – communication is required to fully specify them • recursive • their planning and execution is usually interleaved for each participant and among participants

44. SharedPlans in COLLAGEN COLLAGEN provides a generic framework only for recording the order in which planning and execution occur. COLLAGEN does not currently provide a generic framework for execution (interleaving planning and execution).

45. Discussion No generally accepted domain-independent theory of how people manage the interleaving of planning and execution. Best candidate: the BDI (belief/desire/intention) frameworks.

46. Discourse Segments There is general agreement that discourse in human-human interactions has a natural hierarchical structure, the elements of which are called segments. A segment is a contiguous sequence of communicative actions that serve some purpose (e.g. to achieve shared knowledge of some fact).

47. Example of Discourse Segments (Rich and Sidner, 1998), page 329

48. Focus Stack The focus stack contains discourse segments in the order in which they are created during the natural flow of a collaborative discourse. It captures the shifting focus of attention in a discourse. New segments and sub-segments are created, pushed onto the focus stack, completed, and then popped off the stack as the SharedPlan unfolds in the conversation.

49. Example of Focus Stack (Rich and Sidner, 1998), page 331

50. Discourse State in COLLAGEN (Rich and Sidner, 1998), page 335