640 likes | 761 Views
C onversational A gents in M ulti- P arty I nteractive S ituations. Rohit Kumar. Committee Carolyn P. Rosé (Chair) Alan W. Black Ian R. Lane Jason D. Williams (AT&T Research). Friday, June 11, 2010. Bridges. Multi-Party Interaction. Social / Trust Games. Collaborative Learning.
E N D
Conversational Agents in Multi-Party Interactive Situations Rohit Kumar Committee Carolyn P. Rosé (Chair) Alan W. Black Ian R. Lane Jason D. Williams (AT&T Research) Friday, June 11, 2010
Bridges Multi-PartyInteraction Social /Trust Games CollaborativeLearning TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication CommunicationStudies SoftwareArchitecture
... Conversational Agent, that is, an agent that can participate in fully natural dialog ... - James Allen Conversational Agents exploit natural-language technologies to engage users in text-based information-seeking and task-oriented dialogs for a broad range of applications. - James Lester Conversational agent is a system that exchanges information between user and agent using natural language dialogue - Sofus Macskassy Conversational agents are communication technologies that use natural language and computational linguistic techniques to engage users in human-like, Web-based dialogs. - Wikipedia Conversational Agents (CAs) • General Definition Conversational Agents are automated agentsthat extend conversation as a medium ofinteraction with machines. • Many Applications > Background > Conversational Agents
Some Conversational Agents • Many studies have shown effectiveness of CAs • Information Access > Raux et. al., 2005 • Intelligent Tutoring > Kumar et. al. 2006/2007a • Therapy > Bickmore et. al., 2005 > Background > Conversational Agents
Bridges Multi-PartyInteraction Social /Trust Games CollaborativeLearning TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication CommunicationStudies SoftwareArchitecture
Multi-Party Interactive Situations (MPIS) • Multi-Party Interactive Situations • Meetings, Dinner, Games, Classrooms • Groups more effective than Individuals at Intellective Tasks Review in McGrath, 1984 • Increasing Computer Mediated Multi-Party Interaction • Instant Messaging / Chat • Video Conferencing • Collaborative Learning • Beneficial pedagogical approach • Multi-player Online Games • Collaborative Work • Online Auctions • Social Networking > Background > Multi-Party Interactive Situations
Bridges Multi-PartyInteraction CollaborativeLearning Social /Trust Games TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication CommunicationStudies SoftwareArchitecture
CAs in Multi-Party Interactive Situations • One Agent + Two or More Users • E.g.: Tutor supporting Collaborative Learning Moderator in Group Interaction • Two or More Agents + One User • E.g. TeamTalk Harris & Rudnicky, 2007 • Two or More Agents + Two or More Users • E.g.: Multi-player Games (with multiple NPCs) > Background > Conversational Agents in Multi-Party Interactive Situations
CAs in Multi-Party Interactive Situations • One Agent + Two or More Users • E.g.: Tutor supporting Collaborative Learning Moderator in Group Interaction Existing Work > Background > Conversational Agents in Multi-Party Interactive Situations
CAs in MPIS: Two Challenges • Building / Implementing Agents for Multi-Party Interactive Situations • Technical Challenge • Engineering / Development Issues • Designing Suitable / Useful Agent Behaviorfor such Situations • Scientific Challenge • Design Principles / Guidelines > Problem Statement
Technical Challenge: Building CAs • Existing approaches • Architectures • Backboard, Pipeline, Hub, Multi-Expert, … • Representations • State, Frame, Schema, Agenda, Plan, … • Multi-Expert Architectures • Jaspis Turunen & Hakulinen, 2003 • RIME Nakano et. al., 2008 • Event-driven Dialog Management • Olympus 2.0 > Interaction ManagerRaux & Eskenazi, 2007 > Problem Statement > Building CAs
Technical Challenge: Building CAs • Shortcomings of Existing approaches • Poor Representational Capability • Inflexibility to handle Complex Interaction Dynamics > Problem Statement > Building CAs
Technical Challenge: Building CAs • Poor Representational Capability • E.g. Scheduling policy for simultaneous responses from multiple users • Very High Level Languages • Restricted to small set of operators and control structures • Inability to freely combine representations in existing architectures > Problem Statement > Building CAs
Even Participation Assumption Single User Interaction Multi-Party Interaction System Technical Challenge: Building CAs • Inflexibility to handle Complex Interaction Dynamics • E.g. • Ensuring all users get an equal opportunity to contribute • Two Assumptions that Fail > Problem Statement > Building CAs
Known Addressee Assumption • Schisming Student1 OK, lets start Tutor What would happen to the power output of a Rankine Cycle at a higher operating temperature? Student2 hmmm … Can you answer that? Student1 I think it will increase. Tutor The correct answer is that at a higher operating temperature, more heat is added to the cycle and hence the power output increases too. What about the heat rejected by the cycle though? ? ? Tutor You are right S1. It increases too. Lets move on to the next topic. Technical Challenge: Building CAs • Inflexibility to handle Complex Interaction Dynamics • E.g. • Ensuring all users get an equal opportunity to contribute • Two Assumptions that Fail > Problem Statement > Building CAs
Technical Challenge: Contributions • Basilica A novel software architecture (and development tools) for building CAs in MPIS that provides • Rich Representational Capability • Using High-Level Language • Flexibility to address Complex Interaction Dynamics • Does not make Even Participation and Known Addressee assumptions • An array of CAs built using this architecture that provide • Re-useable components > Problem Statement > Building CAs > Contributions
Second Life Middleware OutputCoordinator SLActor SLListener TutoringActor TurnTakingCoordinator MessageFilter GreetingActor TutoringManager TouchFilter Basilica: Software Architecture • Event-driven Architecture • Agent Representation Kumar & Rosé, 2009 • Network of Behavioral Components • Components are programmable using High-level Languages • Analogy: UI Toolkits • E.g.: Java Swing > Building CAs > Basilica
Basilica: Software Architecture • Core Classes: Provide implementations for • Agent & Component Management • Event Propagation • Observer Interfaces • Agent Factory: Runtime Agent Assembly (from XML) • Utilities & Development Tools • Re-Usable Components > Building CAs > Basilica
Basilica: Four Agents • Four Agents built using Basilica • CycleTalk Tutor Agent • PsychChallenge Peer Agent • WrenchTalk Tutor Agent • Emergency Response (911) Interpreter Agent • Four different Multi-Party Interactive Situations > Building CAs > Basilica > Agents Developed
CycleTalk Tutor Agent > Building CAs > Basilica > Agents Developed
CycleTalk Tutor Agent • Implements Turn-taking rules suitable for Tutoring situation • Wait for responses from students • Prioritize between correct/incorrect/irrelevant responses ConcertChat Server OutputCoordinator ConcertChatActor ConcertChatListener PromptingActor AttentionGrabbingActor AttentionGrabbingFilter TutoringActor TutoringManager RequestDetector TurnTakingCoordinator MessageFilter HintingActor HintingManager > Building CAs > Basilica > Agents Developed
PsychChallenge Peer Agent On Learning Portal of major Publishing Company > Building CAs > Basilica > Agents Developed
HTTP PsychChallenge Web Interface (API) Generic HTTP Middleware (Servlet) M.WareActor OutputCoordinator M.WareListener GreetingActor StatusFilter RoleActor RoleFilter HintingActor ScoreMemory GuessingActor WordMemory PsychChallenge Peer Agent • Agent has the same role as users > Building CAs > Basilica > Agents Developed
WrenchTalk Tutor Agent > Building CAs > Basilica > Agents Developed
WrenchTalk Tutor Agent ConcertChat Server ConcertChatActor ConcertChatListener MessageFilter PresenceFilter DiscourseMemory AnnotationFilter OutputCoordinator SocialController ActivityDetector ProgressDetector PlanExecutor RequestDetector T.TakingCoordinator IntroductionsManager PromptingManager TutoringManager TutoringActor IntroductionsActor PromptingActor > Building CAs > Basilica > Agents Developed
NineOneOne Interpreter Agent V I D E O > Building CAs > Basilica > Agents Developed
XMPP Connector XMPPActor XMPPListener PresenceFilter Caller Proxy Live Audio ASR TTS MessageFilter TellCallerActor CallerTurnMemory DistressedCaller NLUFilter MetaActor RequestTypeFilter IQAActor MetaActManager Parser XMPP Server (Communication Backbone) ActClassifier IQAActManager SlotConfirmationActor SlotConfirmationManager GenericsDetector RequestDetector InformDispatcherActor Emergency Dispatcher ReportDetector SlotValueDetector NineOneOne Interpreter Agent > Building CAs > Basilica > Agents Developed
CAs in MPIS: Two Challenges • Building / Implementing Agents for Multi-Party Interactive Situations • Technical Challenge • Engineering / Development Issues • Designing Suitable / Useful Agent Behaviorfor such Situations • Scientific Challenge • Design Principles / Guidelines > Problem Statement
Scientific Challenge: Agent Behavior • Unlike individual learners,Teams of students ignore / abuse automated tutorsKumar et. al. 2007a • Agents lack Social Communication Skills • Need to be better communicators > Problem Statement > Agent Behavior
Interaction & PerformancePhase MovementConflict Resolution Bridges Multi-PartyInteraction CollaborativeLearning Social /Trust Games TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication SoftwareArchitecture CommunicationStudies
Scientific Challenge: Agent Behavior • Small Group Communication • Two Fundamental Processes • Bales, 1950 (Problem Solving Groups) • Instrumental (task-related) vs.Expressive (social-emotional) • Need for an Equilibrium • Developed Interaction Process Analysis (IPA) • Bion, 1961 & Thelen, 1956 (Therapy Groups) +ve Expressive Instrumental -ve > Problem Statement > Agent Behavior > Small Group Communication
Scientific Challenge: Agent Behavior • CAs must perform both Task-related as well as Social Interaction • Task-related Interaction strategies • Our earlier work Chaudhuri et. al., 2008, 2009 • Social Behavior > Problem Statement > Agent Behavior
Scientific Challenge: Social Behavior Related Work: • Verbal Social Behavior by CAs • Affective Computing • Non-Verbal Social Behavior by ECAs Multi-Party > Problem Statement > Agent Behavior > Related Work
Scientific Challenge: Contributions • Development of Social Interaction Strategies motivated from research in Small Group Communication • Investigation of effectiveness of Social Interaction Strategies in MPIS • Experiments on amount, timing and type of behavior • Studied in multiple task domains • Collaborative Learning & Trust Games • Methodology & Materials to further the development of CAs as Good Communicators • Towards incorporating research from Small Group Communication in CA design > Problem Statement > Agent Behavior > Contributions
Social Interaction Strategies • Eleven Social Interaction Strategies developed • Based on Three Positive Social-Emotional Interaction Categories > Social Behavior > Design
ConcertChat Server ConcertChatActor ConcertChatListener MessageFilter PresenceFilter DiscourseMemory AnnotationFilter OutputCoordinator SocialController ActivityDetector ProgressDetector PlanExecutor RequestDetector T.TakingCoordinator IntroductionsManager PromptingManager TutoringManager TutoringActor IntroductionsActor PromptingActor Implementation: WrenchTalk Tutor Agent • Two primary controllers • PlanExecutor > Executes Task-related steps • SocialController > Triggers Social Behavior • Controllers regulate each other > By Blocking > Social Behavior > Implementation
1d. 2b. 2b. Implementation: Social Controller • Social Behavior Triggering • Hand Crafted Rules • Four Features • Last executed plan step • Annotations of student turns • Dictionary Lookup • Activity Levels • Groups & Individual • Strategy: 1e. (Encourage) • Social Ratio • Ratio of Social Turnsto Task-related turns • Threshold: 20% > Social Behavior > Implementation
Research Questions: Experiments • Effectiveness of Social Behavior Vs. • Gold Standard (Human performance) • Baseline (No Social Behavior) • Right Amount of Social Behavior • Contribution of individual behaviors • Effect of the Triggering Model • Generalizability to other Interactive Situations Experiment 1 Experiment 2 Experiment 3 ProposedTask 1 & 2 ProposedTask 3 & 4 > Social Behavior > Experiments
Experiment 1-3: Interactive Situations • Collaborative Design Labs • Mechanical Engineering Courses • Freshmen: Wrench Design • Teams of 3-4 students • Underlying Concepts • Force, Moment, Stress, Strength, … • Sophomore: Power plant design • Teams of 2 students • Underlying Concepts • Relationships between 4 System Properties & 5 Response Variables • Tasks structured using Worksheets > Social Behavior > Experiments > Situations
Experiment 1-3: Methodology • Controlled Experiments • Between subjects • Conducted over multiple sessions • 35-40 minutes per session • Different students in each session • Students randomly assigned to teams on the spot • Team mates not seated next to each other • Communicate using ConcertChat • Teams randomly assigned to conditions • Nearly even distribution of conditions in each session • Incentive • Class Credit • Gift Certificates for best team designs > Social Behavior > Experiments > Methodology
Experiment 1-3: Labs > Social Behavior > Experiments > Metrics
Experiment 1-3: Metrics • Performance: Learning Outcomes • Pre & Post Tests • Multiple choice questions (MCQs) • Short Essay Questions (SEQs) • Perception: Survey Burke, 1967 • 7-point Likert-scale (1-Strongly disagree, 7-Strongly Agree) • 9 items • Ratings about Tutor (Agent): Likeable, Friendly, … • Satisfaction Ratings: Task & Group Discussion • Legitimacy > Social Behavior > Experiments > Metrics
Research Questions: Experiments • Effectiveness of Social Behavior Vs. • Gold Standard (Human performance) • Baseline (No Social Behavior) • Right Amount of Social Behavior • Contribution of individual behaviors • Effect of the Triggering Model • Generalizability to other Interactive Situations Experiment 1 > Social Behavior > Experiments
Experiment 1: Evaluating Effectiveness • Experimental Design • Three Conditions • Benefit of the Collaborative Design Activity • No significant difference between pre-tests for the three conditions • Significant improvement from pre-test for post-test for all groups • F(1,190) = 16.67, p <0.001, Effect size = 0.51σ > Social Behavior > Experiment 1
Experiment 1: Results > Performance Learning Outcomes • ANCOVA post-test = Fn(1, pre-test, condition, day) • Significant effect of Condition F(2, 93) = 10.56, p < 0.001 • Post-hoc Analysis • Task vs. Social { p < 0.01, 0.71σ } • Task vs. Human { p < 0.001, 0.93σ } • Social vs. Human { p = 0.509 } • Tests: 8 MCQs & 3 SEQs Analysis done on total score > Social Behavior > Experiment 1 > Results
Experiment 1: Results > Perception • Human & Social conditions better than Task conditions • Human condition significantly better on (Q1 - Q5, Q8) • Social condition significantly beter on Q2, marginal on Q4, Q5 > Social Behavior > Experiment 1 > Results
Experiment 1: Summary • Significant benefits of employing Social Interaction Strategies • Implementation of Social Tutors not as good as Human tutors • Right Amount of Social Behavior • Significantly more Social Behavior in Human condition Average (Human) = 22.17 Average (Social) = 16.83 • Human Social Behavior Triggering Experiment 2 ProposedTask 1 & 2 > Social Behavior > Experiment 1
Experiment 2: Amount of Behavior • Manipulated by changing Social Ratio • Experimental Design • Three Conditions • Benefit of the Collaborative Design Activity • No significant difference between conditions on pre-tests • Significant improvements from pre- to post-test • Effect size = 0.79σ (Total), p < 0.001 > Social Behavior > Experiment 2
Experiment 2: Results Performance > Learning Outcomes • Tests: 22 MCQs & 6 SEQs • ANCOVA post-test = Fn(1, pre-test, condition, session) • Significant effect of Condition on MCQs F(2, 97) = 3.48, p < 0.05 • Post-hoc Analysis (for MCQs) • None vs. Low { p < 0.07, 0.69σ } • Low vs. High { p < 0.07, 0.55σ } • None vs. High { not significant } Perception > Survey • No Significant Differences between conditions Use moderate amount of Social Behavior > Social Behavior > Experiment 2 > Results