SMART ANIMATED AGENTS

1. SMART ANIMATED AGENTS • Norman I. Badler -- Course #24 Co-Organizer (with John Funge) • Center for Human Modeling and Simulation • University of Pennsylvania • Philadelphia, PA 19104-6389 • http://www.cis.upenn.edu/~badler Smart Animated Agents -- SIGGRAPH Course #24

2. Course Speakers • Norman Badler (U. Pennsylvania) • Justine Cassell (MIT Media Lab) • John Funge (Sony Computer Entertainment America) • Jeff Rickel (ISI / U. So. California) • Bruce Blumberg (MIT Media Lab) Smart Animated Agents -- SIGGRAPH Course #24

3. Course Schedule (a.m.) • 8:30-8:35 Badler (Introduction) • 8:35-10:00 Badler • 10:00-10:15 (break) • 10:15-11:45 Cassell • 11:45-12:00 Questions and Issues • 12:00-1:30 (lunch) Smart Animated Agents -- SIGGRAPH Course #24

4. Course Schedule (p.m.) • 1:30-2:30 Funge • 2:30-3:00 Rickel • 3:00-3:15 (break) • 3:15-3:45 Rickel • 3:45-4:45 Blumberg • 4:45-5:00 Questions and Issues Smart Animated Agents -- SIGGRAPH Course #24

5. Course #24 Topics • Action Primitives and Action Representation • Natural Language Interfaces • Conversational and Communicative Agents • Cognitive Modeling • Pedagogical Agents • Task-Oriented Collaboration • Learning Smart Animated Agents -- SIGGRAPH Course #24

6. Building Smart Agents (Badler) • Introduction and Applications • Smart Avatars • Parameterized Action Representation (PAR) • Natural Language Instructions • Automating Attention • Agent Manner • Building PARs Smart Animated Agents -- SIGGRAPH Course #24

7. Building Smart Agents • Introduction and Applications • Smart Avatars • Parameterized Action Representation (PAR) • Natural Language Instructions • Automating Attention • Agent Manner • Building PARs Smart Animated Agents -- SIGGRAPH Course #24

8. Introduction to and Applications for Embodied Agents: • Engineering Ergonomics. • Design and Maintenance Assessment. • Games/Special Effects. • Military Simulations. • Job Education/Training. • Medical Simulations. Smart Animated Agents -- SIGGRAPH Course #24

9. Virtual Human “Dimensions” • Appearance • Function • Time • Autonomy • Individuality Smart Animated Agents -- SIGGRAPH Course #24

10. Appearance: • 2D drawings > 3D wireframe > • 3D polyhedra > curved surfaces > freeform deformations > • accurate surfaces > muscles, fat > biomechanics > clothing, equipment > physiological effects (perspiration, irritation, injury) Smart Animated Agents -- SIGGRAPH Course #24

11. Function: • cartoon > jointed skeleton > • joint limits > strength limits > • fatigue > hazards > injury > skills > effects of loads and stressors > psychological models > • cognitive models > roles > teaming Smart Animated Agents -- SIGGRAPH Course #24

12. Time (to create / move each): • off-line animation > • interactive manipulation > • real-time motion playback > parameterized motion synthesis > multiple agents > • crowds > coordinated teams Smart Animated Agents -- SIGGRAPH Course #24

13. Autonomy: • drawing > scripting > • interacting > reacting > • making decisions > • communicating > intending > • taking initiative > leading Smart Animated Agents -- SIGGRAPH Course #24

14. Individuality: • generic character > • hand-crafted character > • cultural distinctions > • sex and age > personality > psychological-physiological profiles > specific individual Smart Animated Agents -- SIGGRAPH Course #24

15. Comparative Graphical Agents Application Appear. Function Time Autonomy Individ. Cartoons high low high low high Sp. Effectshigh low high low med Medicalhigh high med med med Ergonomicsmed high med med low Gameshigh low low med/high med Military med med low med/high low Educationmed low low med/high med Training med low low high med Smart Animated Agents -- SIGGRAPH Course #24

16. Building Smart Agents • Introduction and Applications • Smart Avatars • Parameterized Action Representation • Natural Language Instructions • Automating Attention • Agent Manner • Building PARs Smart Animated Agents -- SIGGRAPH Course #24

17. Why Smart Avatars? For Motion Control • Point-and-click (menu or direct 2D manipulation). • Directly sensed (3D motion capture). • Language commands (text or speech). • Use instructions -- as if the agent were oneself or another real person: • A Smart Avatar Smart Animated Agents -- SIGGRAPH Course #24

18. Smart Agent Requirements • Actions to Execute: • Action Representation - What it can do. • Behavior Model: • The agent’s decision-making, “thought,” and reaction processes - What it should do or wants to do. • Inputs to Effect Behavior: • Incoming knowledge about the outside world - What it needs to know. Smart Animated Agents -- SIGGRAPH Course #24

19. “Classic” AI: Agent Action Cycle Control World model • Messages • Sensors • Situation Sense Act Agent model Smart Animated Agents -- SIGGRAPH Course #24

20. Smart Agent Requirements • Actions to execute: • Action Representation - What it can do. • Behavior Model: • The agent’s decision-making, “thought,” and reaction processes. • Inputs to Effect Behavior: • Incoming knowledge about the outside world. Smart Animated Agents -- SIGGRAPH Course #24

21. 4 Levels of Action Representation • 0: Basic Motion Generators • 1: Parallel -Transition Networks • 2: Parameterized Actions • 3: Natural Language Instructions Smart Animated Agents -- SIGGRAPH Course #24

22. Level 0: Basic Human Movement Capabilities • Gesture / Reach / Grasp. • Walk / Turn / Climb. • Posture Transitions (Sit / Stand) • Visual Attention / Search. • Pull / Lift / Carry. • Motion playback (captured or scripted). • ‘Noise’ or secondary movements. Smart Animated Agents -- SIGGRAPH Course #24

23. Synthesized Motions -- Leverage Economy of Expression • Few parameters controlling many: • Inverse kinematics for arms, legs, spine. • Paths or footsteps driving locomotion. • Balance constraint on whole body. • Dynamics control from forces and torques. • Facial expressions Smart Animated Agents -- SIGGRAPH Course #24

24. Smart Agent Requirements • Actions to execute: • Action Representation. • Behavior Model: • The agent’s decision-making, “thought,” and reaction processes - What it should do or wants to do. • Inputs to Effect Behavior: • Incoming knowledge about the outside world. Smart Animated Agents -- SIGGRAPH Course #24

25. Raise Level of Behavioral Control from Level 0 • AnimNL project (~1988-1994): • “Go into the kitchen and get me the coffee urn” (manual scripting of actions) • SodaJack (action planner + object specific reasoner) • Needed a better underlying paradigm upon which to build smarter agents. Smart Animated Agents -- SIGGRAPH Course #24

26. Level 1: Parallel Transition Networks (PaT-Nets) • A Virtual Parallel Execution Engine for agent actions (a.k.a. Finite State Machines): • Processes are nodes. • Instantaneous (conditional or probabilistic) transitions are edges. • Hierarchic. • Message passing and synchronization. • Emerging common paradigm for agent control. Smart Animated Agents -- SIGGRAPH Course #24

27. PaT-Net Applications • Conversational agents. (SIGGRAPH ‘94) • Hide and seek. (VRAIS ‘96) • MediSim: Physiological models. (Presence ‘96) • Jack Presenter. (AAAI-97 Workshop/IEEE CG&A) • Delsarte Presenter. (Pacific Graphics ‘98) • JackMOO. (WebSim ‘98, VR ‘99) • AVA (Attention). (Autonomous Agents ‘99) Smart Animated Agents -- SIGGRAPH Course #24

28. What’s Missing? • PaT-Nets are effective but hand-coded. • No matter what artificial language we introduce it is not the way people conceptualize the situation. (Badler/Webber) • Connect language and animation through an intermediate level --- Smart Animated Agents -- SIGGRAPH Course #24

29. Level 2: Parameterized Action Representation (PAR) • Derived from BOTH Natural Language analyses and animation requirements: • Agent, Objects, Sub-Actions. • Preparatory Specifications, Postconditions. • Applicability and Termination Conditions. • Purpose (Achieve, Generate, Enable). • Path, Duration, Motion, Force. • Agent Manner. Smart Animated Agents -- SIGGRAPH Course #24

30. Level 3: Natural Language Instructions • Instructions say what to do. • Instructions depend on underlying action skills. • Instructions build agent behaviors (future actions or standing orders). Smart Animated Agents -- SIGGRAPH Course #24

31. Integrated Approach: The JackMOO Testbed • JackMOO goal: Create a multi-user, shared, 3D virtual environment with full body avatars and autonomous human agents, language-based commands, and low network bandwidth. • Based on lambdaMOO engine with Jack 3D environment and simple imperative commands. Smart Animated Agents -- SIGGRAPH Course #24

32. Smart Animated Agents -- SIGGRAPH Course #24

33. JackMOO Smart Avatar Experiments • Greetings (Gender and culture specific) • Go to … (Sit in chair; Go to bed; Leave) • Do unspecified but necessary preparatory actions. • Relationships (Follow me) • Mutual agreement. • Autonomous Agents (Waiter) • Reacts to environment & states of other agents. Smart Animated Agents -- SIGGRAPH Course #24

34. Expanding the Agent Model • Create individuals or specific people. • Link perceptions of context to action. • Embed action planning capabilities. • Add emotional planner. • Agent = • Intentions X Personality X Emotions X Context X History X Capabilities X … Smart Animated Agents -- SIGGRAPH Course #24

35. Adapt the OCC Model of Agent Emotional Response • Consequences of events: • Consequences for self • Consequences for others • Actions of agents: • Self • Others • Actions of objects. Smart Animated Agents -- SIGGRAPH Course #24

36. OCC Model of Emotions Joy Distress WELL-BEING Pride Admiration Shame Reproach ATTRIBUTION Love Hate ATTRACTION Happy-for Gloating Resentment Pity FORTUNES OF OTHERS Hope Fear Confirmed Disconfirmed Satisfaction Relief Fears-confirmed Disappointment PROSPECT-BASED Gratification Gratitude Remorse Anger WELL-BEING/ATTRIBUTION COMPOUNDS Valenced Reaction To Consequences of Events Actions of Agents Aspects of Objects pleased, displeased approving, disapproving liking, disliking FOCUSING ON FOCUSING ON Consequences for Others Consequences for Self Self Agent Other Agent Prospects Relevant Prospects Irrelevant Undesirable Desirable Smart Animated Agents -- SIGGRAPH Course #24

37. Smart Agent Requirements • Actions to execute: • Action Representation. • Behavior Model: • The agent’s decision-making, “thought,” and reaction processes. • Inputs to Effect Behavior: • Incoming knowledge about the outside world - What it needs to know. Smart Animated Agents -- SIGGRAPH Course #24

38. Response Requires Input • Sensing the state of events: • Self (action postconditions) • Others (messages; observations) • Sensing the actions of agents: • Self knowledge (what am I doing) • Others (messages; observations) • Sensing the actions of objects. • Smart objects Smart Animated Agents -- SIGGRAPH Course #24

39. Input Sensing • Message passing. • Explicit transfer or direct knowledge of state information between agents. • Artificial perception. • Visual/auditory/haptic [collision detection] sensing to attend to and observe local context. • Situation awareness. • Recognizing complex relationships. Smart Animated Agents -- SIGGRAPH Course #24

40. Training the Agent Model Control • Hand-coded procedures. • Rule-based systems. • Natural Language instructions. • By example (demonstration). Sense Act Agent model Smart Animated Agents -- SIGGRAPH Course #24

41. Building Smart Agents • Introduction and Applications • Smart Avatars • Parameterized Action Representation • Natural Language Instructions • Automating Attention • Agent Manner • Building PARs Smart Animated Agents -- SIGGRAPH Course #24

42. Recall: Parameterized Action Representation (PAR) • Representation derived from BOTH NL analyses and animation requirements: • Agent, Objects, Sub-Actions. • Preparatory Specifications, Postconditions. • Applicability and Termination Conditions. • Purpose (Achieve, Generate, Enable). • Path, Duration, Motion, Force. • Agent Manner. Smart Animated Agents -- SIGGRAPH Course #24

43. Smart Animated Agents -- SIGGRAPH Course #24

44. Examples of PAR Action Fragments • Preparatory Specifications: • If not at proper location to execute action, get there and get into correct pose to continue. • Applicability Conditions: • In order to use a gun, the agent must have one; he does not have to go find one. • Termination Conditions: • “Draw gun” terminates when the gun is no longer in the holster. Smart Animated Agents -- SIGGRAPH Course #24

45. Examples of PAR Action Fragments • Path parameters: • Walk to a given location. • Reach to a given place. • Agent manner: • Walk style. • Expressive content (EMOTE parameters). • Postconditions: • After “receive object” action terminates, agent has object. Smart Animated Agents -- SIGGRAPH Course #24

46. Case Study: The Virtual Reality Checkpoint Trainer • Joint ONR Project between UPenn, UHouston, and EAI. • Multi-agent and/or avatar situation. • Process simulator for traffic. • Autonomous agents. • Real-time behaviors and reactions. • Natural Language input for “standing orders”. • (Next step: Live trainees in VR.) Smart Animated Agents -- SIGGRAPH Course #24

47. Virtual Environment Smart Animated Agents -- SIGGRAPH Course #24

48. The Checkpoint Scene Smart Animated Agents -- SIGGRAPH Course #24

49. Components of the Checkpoint Scenario • The PAR system architecture. • Agents and behavior rules. • The Actionary. • Actions currently represented. • Python implementation. • Natural Language inputs. • But first, the video. Smart Animated Agents -- SIGGRAPH Course #24

50. Execution Engine NL2PAR Rule Manager Actionary Jack Toolkit Actions Objects Process Manager Motion Generators Visualizer PAR SYSTEM ARCHITECTURE Agent Proc 1 Queue Manager Process Manager Agent Proc 2 Queue Manager Process Manager Agent Proc N Queue Manager Smart Animated Agents -- SIGGRAPH Course #24