Chap. 13 Expressive Behaviors for Virtual Worlds

Chap. 13Expressive Behaviors for Virtual Worlds Stacy Marsella, Jonathan Gratch, and Jeff Rickel November22, 2004

Outline • Introduction • Steve • Jack and Steve • Carmen’s Bright IDEAS (CBI) • Mission Rehearsal Exercise (MRE) • Conclusion

Introduction • Goal of this chapter • To create virtual humans that convey much information to humans while interacting with them in virtual worlds • Information includes non-verbal behaviors concerning emotional state • Requirements for virtual human design • Believable: they must provide human-like behaviors • Responsive: they must respond to the events surrounding them • Interpretable: the user must be able to interpret their response to situation • This chapter describes • The progress toward a model of outward manifestations of an agent’s cognitive and emotional state • The review of three prior systems that influenced model of this chapter • The model of this chapter is an integration of prior systems

Steve Contribution • Previous systems provides impressive capabilities in its area of research focus, but they had some limitations • Limitation of previous systems • Agents are modeled only for conversation between two of them • Collaboration of users and agents on one task is not allowed • Agent’s presence was partially limited to a 2D • Agent’s movement and relationships are limited • Steve’s contribution • Only Steve can interleave task-related behaviors and face-to-face dialogue with humans and virtual humans in dynamic virtual worlds • Interleaving은 성능을 높이기 위해 데이터가 서로 인접하지 않도록 배열하는 방법

Steve The Architecture • Steve consists of three modules: perception, cognition and motor control

Steve Primitives • The cognition module generates Steve’s communicative behavior by dynamically selecting next action from a list of primitives • Primitives • Speak • Move to an object • Manipulate an object • Visually check an object • Point at an object • Give tutorial feedback • Offer turn • Listen to student • Wait for someone • Acknowledge an utterance • Drop hands • Attend to Action

Steve React to Interruption & Context (1) • To react to interruption • In dynamic virtual worlds, to react to the unexpected events is important for task-oriented collaboration • It is important to maintain coherenceof an agent’s behavior • Steve considers 2 separate but complementary types of context 1. Task context • Continually monitors the virtual world • Uses the task model to plan to complete the task • Uses variant of partial-order planning techniques 2. Dialogue context • Represents the state of interaction between a student and Steve • Uses focus stack: represents the hierarchy of tasks, subtasks, and currently engaged actions • Maintains a record, Steve’s answer etc.

Steve React to Interruption & Context (2) • Based on the current task and dialogue contexts, Steve can choose next action • Steve can choose an action of following three roles • Steve can respond to the student to know a student’s request • Steve can choose for himself how to advance the collaborative dialogue • Steve can choose a turn-taking or grounding act that helps regulate the dialogue without advancing the task • In case several actions are appropriate at one moment, • Steve choose next action considering priorities • Actions with low priority will be performed later, not be deferred • Summary • Steve does not include an emotion model, but • It serves a valuable foundation for MRE model

Jack and Steve Jack and Steve vs. Steve • The Jack and Steve system was predicated on two basic claims: • Plans and plan reasoning can mediate expressive behavior • Small biases in how plans are evaluated and generated can result in large systematic differences in agent behavior • Jack and Steve vs. Steve

Jack and Steve Motivating Example • This example shows two separate runs of system where the only difference is a change in the personality of Steve (rude / fair) • Jack: I want to make some big money • Steve: I want to catch some waves Mental state shows individual action, plan, emotion etc. Mental state of each agent at specific point in the interaction

General purpose planning system Social reasoning layer Jack and Steve Plan-Based Social Appraisal (1) • Jack and Steve supports expressive and flexible interactions by implementing social reasoning as a layer atop a general-purpose planning system • Planning system provides • Domain-independent representations of world actions • General reasoning mechanisms that construct partial plans, repair interactions among them, and oversee plan execution • The social layer • Manages communication • Biases plan generation and execution according to social context

Jack and Steve Plan-Based Social Appraisal (2) • To support various social interactions, the social reasoning layer must provide a rich model of the social context • The social situation is described in terms of many static and dynamic features • Static features • include innate properties of the character • social role and small set of personality variables • Dynamic features • Are derived from inference procedure that operate on the current mental state of agent • Include - current communication obligation - various relations between plans in memory (e.g. your plans threaten my plans) - a model of the emotional state of the agent

Jack and Steve Plan-Based Social Appraisal (3) • One novel aspects of this system is that the social layer alters the planning process fundamentally • In terms of planning, the social layer can bias planning to be more or less considerate to the goals of other participants • In terms of communication, agents can vary from bossy agents that try to tell others what to do to passive agents that avoid interactions or social conflicts

Jack and Steve Social Context, Operator, Rule • Jack and Steve system maintains a rich representation of the social context • The social context of Jack and Steve is divided into 3 categories of plan context, emotional context, and communicative context • Social operators are actions that occur at the social level • Social operators are subdivided into meta-planning operators and communicative operators. • Distinct personalities are implemented via a set of social rules • Social rules execute sequences of social operators based on appraised features of the social context • Jack and Steve system include 30 social rules

Carmen’s Bright IDEAS Introduction • Carmen’s Bright IDEAS (CBI) • An agent-based system designed to realize an Interactive Pedagogical Drama • The pedagogical goal is to help mothers of pediatric cancer patients • The drama mirrors the mother’s own problems • It allows the learner to interactively influence how Carmen copes with problems • In CBI • to model the causes of emotions is important • agents needed effective ways to convey the impact of emotion on both the agent’s dialogue and physical behavior

Carmen’s Bright IDEAS The Drama of Carmen’s Bright IDEAS • CBI is an interactive pedagogical drama • Carmen discusses her problems with a clinical counselor, Gina • Gina suggests a problem solving technique called Bright IDEAS • Bright IDEAS means • Bright: positive attitude • I: Identify a solvable problem • D: Develop possible solution • E: Evaluate your options • A: Act on your plan • S: See if it worked • User, the human mother, interacts with the drama by making choice for Carmen • Both Gina’s dialogue and the user’s choices influence the cognitive and emotional state of Carmen (in conflicting ways)

Carmen’s Bright IDEAS Interaction Model • Learner influences Carmen by selecting thought balloons • In this interaction model, the user gets a vivid demonstration

Carmen’s Bright IDEAS CBI Agent Model • IPD have 5 main components, but the discussion will focus on the on-screen character agents • Each on-screen character is realized by an agent architecture that has modules for problem solving, dialogue, emotional appraisal, and behavior generation

Carmen’s Bright IDEAS Emotional Model • The pedagogical goal was for the mothers to learn how to choose and carry out the right coping strategy for a given situation • Cognitive appraisal theory by Richard Lazarus influences CBI • This organizes human behavior around appraisal and coping • Appraisal leads to emotion by assessing the person-environment relationship • Coping and appraisal interact and unfold over time, supporting the temporal character of emotion evident in human behavior • Coping is the process of dealing with emotion • Problem-focused coping • Emotion-focused coping

Carmen’s Bright IDEAS Remarks • The design of dialogue, emotions, and expressive behavior systems was driven by a need to convey deep inner conflicts and how those conflicts play out expressively

Mission Rehearsal Exercise Introduction • The Mission Rehearsal Exercise (MRE) system • Brings ideas from each of the preceding systems • Createsa broader and more flexible array of expressive behaviors • The goal is to teach leadership skills in high-stakes social situation • To model such dramatic and interactive scenario, the MRE • Combines Steve’s ability to flexibly interact with a human user • Augment it with the richer social and emotional behaviors of CBI and Jack and Steve • The MRE combines a variety of capabilities in service of realistic and natural collaboration with virtual humans • Example

Mission Rehearsal Exercise Cognition and Emotions • Our virtual humans must • Provide realistic expressive behaviors • Require cognitive machinery to recognize which behaviors are appropriate in the course of an unscripted interaction with a human user • The Steve system is selected as a starting point since it supports flexible face-to-face interactions with a human user • Integrate plan-based appraisal model of the Jack and Steve system into the Steve system • Integrate CBI’s coping model • Coping strategies were recast explicitly into procedures that updated Steve’s task representations and reasoning processes

Mission Rehearsal Exercise Cognition and Emotions Steve Steve supports flexible face-to-face interaction with human users + This allows agents to negotiate over tasks and express emotions, i.e. task model is extended in a number of ways to represent the socio-emotional context Plan-based appraisal modelof Jack and Steve Integrating CBI’s coping model results in a tight integration between appraisal, coping and task reasoning that closely follows the cognitive appraisal theory of Richard Lazarus + Coping model of Carmen’s Bright IDEAS

Mission Rehearsal Exercise Physical Behavior • Virtual humans in the MRE attempt to manifest the rich dynamics of this cognitive and emotional inner state through each character’s external behavior • The key challenge is the range of behaviors that must be integrated: Each character’s body movements must reflect • Its awareness of events in the virtual world • Its physical actions • The number of non-verbal signals that accompany speech during social interaction • Its emotional reactions • Expressive physical behavior in the MRE agents integrates the task-related non-verbal behaviors of the Steve system and the coping behaviors of CBI

Conclusion • Steve serves as the foundation for the virtual humans in MRE system as his ability to interleave task-related behaviors and face-to-face dialogue in dynamic virtual worlds • The Jack and Steve system contributed a model of task-oriented emotional appraisal and a model of socially situated planning • The CBI system contributed a complementary model of emotional appraisal focusing on social relationships • MRE virtual humans integrate many of the ideas from these three prior systems

Chap. 13 Expressive Behaviors for Virtual Worlds