Modeling Emotions in Game Characters: Theoretical Foundations & Practical Guidelines

1. Modeling Emotions in Game Characters:Theoretical Foundations &Practical Guidelines Eva Hudlicka Joost Broekens SEP 10, 2009 ACII 2009 Amsterdam

2. Outline • Emotions & Games: Affective Gaming • Theoretical Foundations • Practical Guidelines • Conclusions

5. Where We Are Now • Tremendous advances in gaming technologies • From dots to cinematic characters • From single screen to virtual world • Focused primarily on: • Physical realism of game characters & environments • Complexity & performance of simulations & networking • Rich gameplay/interaction • Today’s games still limited in: • Affective realism for game character (NPC) • Social complexity & realism of interactions between and with NPC • Ability to adapt to player’s state: personalization of the experience (see Georgios’ talk)

6. In terms of the full potential of gaming.. We are about here…

7. To achieve the “next big leap” • ..in engagement & effectiveness • Games would benefit from: • Adapting to players’ affective states • Enhancing social & affective complexity & realism of: • Game characters • Their interaction with each other & the players • Game narrative as a whole (interactive storytelling)

8. Current focus of Affective Gaming AC Methods & Techniques Relevant for Affective Gaming • Sensing & recognition of players’ emotions • Adaptive gaming • Game control • Expression of emotions by game characters • More realistic visuals • Models of emotion in game characters • To support complex, autonomous behavior • To support adaptive behavior • Models of players’ emotions • Affective user models to support game personalization (Yannakakis) • Affective game evaluation • Use affective feedback to develop games with desired affective profiles

9. Emotion Models in Affective Gaming • Game characters’ emotions • To generate • realistic & affectively-complex character behavior & believable affective expressions • You need • Emotion generation & emotion effects on cognition & behavior • … responsive to dynamically-evolving game scenario & player behavior • … in real-time (practical issues such as scalability and performance) • Players’ emotions • Help recognize player emotions in real-time • Develop gameplay adaptations • Generate more realistic player avatar behavior

10. Outline • Emotions & Games: Affective Gaming • Theoretical Foundations • Practical Guidelines • Conclusions

11. Emotion and affect in human behavior • Basic emotions: fear, anger, happiness, sadness, surprise, disgust • Short episode of multimodal activity triggered by event: • subjective feelings (the emotion we normally refer to), • tendency to do something (action preparation), • facial expressions, • evaluation of the situation (cognitive evaluation, thinking), • physiological arousal (heartbeat, alertness). • Affect = related to emotion, mood and attitudes: • emotion : object directed, short term, high intensity, action oriented, differentiated. • mood : usually unattributed and undifferentiated, longer term, low intensity. • attitude : affect permanently associated with an object/person • affect : abstraction of emotion/mood in terms of, positiveness/negativeness and activation/deactivation (e.g., Russell, Rolls).

12. Emotion and affect in human behavior • Situational evaluation (intrapsychic) andcommunication (interpersonal). • Heuristic relating events to actions through an evaluation of personal relevance (e.g., goals, needs, drives, motivations): • Evaluation of personal relevance of event • Speeds-up decision-making • fast reactions and action preparation • influence information processing • Learning & adaptation, attention, mental search/planning, creativity, etc.. • Communication medium: • communicate internal state and intention • alert others • show empathy (understanding of situation of others)

13. Emotion: dimensions (Breazeal, 2003) adapted from Russell (1997) • Set of underlying common factors of emotion • E.g. Russell, Mehrabian, Wundt

14. Emotion: categories • A small number of hardwired basic emotions exist • E.g. Tomkins, Izard, Ekman, Panskepp • Category is a typical “emotion syndrome” • A complex of physiology, expression, behavior, and feeling • Anger: • High arousal • Face: angry • Approach • Bad feeling • Joy: • High arousal • Face: happy • Play • Good feeling • Sadness: • Low arousal • Face: sad • Avoid • Bad feeling

15. Emotion: components Novelty Pleasantness Goal/Need conduciveness Coping potential Sensory-Motor level Sudden, intense stimulation Innate preferences/ aversions Basic needs Available energy Schematic level Familiarity: schema matching. Learned preferences or aversions Acquired needs motives Body schema Conceptual level Expectations: cause/effect, probability Recalled, anticipated, or derived positive-negative estimates Conscious goals, plans Problem-solving ability. • Parallel evaluation in terms of appraisal dimensions • (E.g. Scherer: stimulus checks)

16. Emotion: summary

17. Outline • Emotions & Games: Affective Gaming • Theoretical Foundations • Practical Guidelines • Conclusions

18. Main questions • I want to build an emotional agent, now • Where do I start? • Where do I end? • When did I succeed? • A tree has no emotions • A mosquito has “emotional behavior” in the eye of the emotional beholder (fear, frustration). • A lizard has emotions, although probably only a couple of them (fight, flight, satisfaction) • A grown buffalo has emotions, and probably quite a lot of them (fear, excitement, joy, attraction, satisfaction). • Agent? • Emotion? • Let’s dive into the details…

19. Agent related issues • Really understand the environment & agent / NPC / Virtual Character / etc. • What is the observable behavior of my agent? • What are its potential actions? • What is the environment of the agent? • What are the observable events for that agent? • What are the beliefs and motivations for the agent? • What does the agent want, like, dislike etc… • Do emotions make sense, what are they supposed to add? • HCI aspects: Believability and effectiveness (serious games), fun (games), interaction • Agent autonomy aspects: intelligence, planning, learning and adaptation, etc.?

20. Emotion related issues • How do I select an emotion theory that best matches the: • Internal workings of my agent (motivations, beliefs, etc.) • Observable behavior of my agent (actions) • Environment (events) • How do I value events in relation to the emotion theory? • Direct versus indirect mapping. • How do I represent emotion internally? • States, traits, moods, attitudes • Factors, categories, components • Level of complexity of the “emotion object” (level of detail) • How do I represent emotion dynamics? • Onset, decay, mixed emotions (similar versus opposing), etc. • How do I link emotion to emotion effects, expression and behavior? • Direct vs Indirect, feature vs expression • Facial expression, body posture, movement, behavior. • Effects on problem solving, planning, decision making, learning, etc.

21. Emotion architecture for an NPC Emotion generation Emotion representation Emotion effects Environment Interpretation Affect Type Effects Behavior Internal Events Social Trait Social Face Psychological Emotion Psychological Posture External Events Biological Mood Biological Actions Physiological Attitude Physiological Behaviors Thoughts Guideline to make your choices Not everything is needed in your model!

22. Outline • Emotions & Games: Affective Gaming • Theoretical Foundations • Practical Guidelines: example integrating generation and effects • Conclusions

23. First Person Shooter NPC Source: resident evil • The agent • Monster (zombie) • Needs: • Health (hit-points) • Hunger (eat flesh, drink) • Potential events • see_human, see_animal, see_water, • health_high, health_low • energy_high, energy_low • Hunger_high, hunger_low • attacked • Potential behaviors • wander, chase_human, chase_animal, move_to_water, search_X, flee • eat_human, eat_animal, drink, sleep

24. First Person Shooter NPC Source: resident evil • The emotions • Why? • Add realism • Influence action choice • Influence attention • Expression: • Fear, aggression, happiness, panic. • Behavior: • Trigger search, chase and flee • Attention • Field of view (broad-narrow) • Object fixation • Self, other oriented

25. First Person Shooter NPC: Select theory

26. Homeostasis: why? • Drives & needs • Drive: motivator for particular behavioral “programs” • Needs: biological drives • Sleep • Hunger • Thirst • Etc. • When drives not met (or met too much) this influences: • Emotion, behavior and attention to regulate drives • Process is called homeostasis: • Behavior is aimed at satisfying needs and regulates emotion • Emotion is a signal that biases organisms towards interaction that satisfies current needs.

27. First Person Shooter NPC: Select affect types

28. First Person Shooter NPC: expression

29. First Person Shooter NPC Environment Interpretation Affect Type Behavior Effects attention Emotion (categories) Fear Aggression Happiness Panic Health_low Health_high Hunger_low Hunger_high Fear Anger Happiness Panic Biological Needs: Health Hunger See_human See_animal See_water Search_X chase_X Flee, etc. Emotion elicitation Emotion representation Emotion effects Architecture

30. First Person Shooter NPC • Instrumentation • Direct mapping based on drives/needs and anticipated effect.

31. First Person Shooter NPC • Affect dynamics • FeltDrive=drive+1/2*anticipation • Expression • Based on mapping of FeltDrive combination to emotion category

32. Emotion effects • Fewer theories exist than for emotion generation • Specific mechanisms of emotion effects not as well developed • Some available theories: • Distinct modes of processing associated with different emotions (Oatley & Johnson-Laird, 1987) • Spreading activation & priming (Bower, 1984; Derryberry, 1988) • Emotions as patterns of parameters modulating processing (Fellous, Matthews, Ortony et al., Hudlicka, Ritter) • Componential model of effects (Scherer et al., Lerner & Tiedens)

33. Emotion effects • Many relationships between affect and cognition: • Mood influences information processing style • Top-down (positive) versus bottom-up (negative) • Heuristic/generic/assuming/creative processing (positive) versus detail/feature/critical/procedural processing (negative) • Mood influences learning • Flow, boredom, frustration , etc. • Emotion influences information processing, e.g., • anxiety  threat bias (Mineka et al., 2003) • aggression  higher risk tolerance (Lerner & Tiedens, 2006) • arousal is related to attention capacity

34. Emotion Effects in NPC Context • Zombie tasks: • Maintain health • Maintain energy • Stay fed • NPC behavior changes depending on emotion • homeostatic interpretation: specific emotions favor specific behaviors to satisfy needs. • Happiness: increase tendency to…wander, sleep. • Panic: increase tendency to…search_animal, chase_animal, flee_human • Aggression: increase tendency to… search_X, chase_X, eat_X • Fear: increase tendency to… wander, flee_X.

35. Emotion Effects in NPC Context (2) • Zombie attention: • Nr of humans/animals tracked (parameter for field of view: detail vs. global) • Object fixation (parameter for stability & capacity working memory) • Focus on self-related needs vs. other (parameter for self-other directedness) • Effects (provided as example!): • Pos. valence: global attention, distractible (e.g., Dreisbach & Goshke, 2004) • Neg. valence: focus on details, tunnel vision (e.g., Clore, Isen) • Anger: Attribution of hostility in others (Lerner & Tiedens, 2006) • Fear: Threat-directed attention (Mineka et al., 2003) • Emotion influences attention parameters (parameter-based effect) • Happy: Large field of view, weak object fixation, balanced needs • Panic: Small field of view, weak object fixation, balanced needs • Aggression: Small field of view, strong object fixation, hunger • Fear: Small field of view., strong object fixation, health

36. Outline • Emotions & Games: Affective Gaming • Theoretical Foundations • Practical Guidelines • Conclusions

37. Conclusions • Affect-focused game design • emotion plays a central role in: • Gameplay design - adapt to player emotions • Game character design - more affective realism • “Assist Me, Challenge Me, Emote Me” (Gilleade, Dix & Allanson 2005) • Affective game engines (Hudlicka, 2009): • Game development tools that support: • Sensing & recognition of player emotions • Gameplay adaptation to player affect • Design of affective game characters

38. Theoretical Perspectives Modeling Emotions in Game Characters • Affective gaming • Emotion recognition for adaptive game control • Emotion in NPC for adaptation, realistic behavior & expression • Models of players’ emotions for game personalization • Affective feedback for game evaluation and development • NPC emotionsGenerate realistic affective character behavior & expressions • Emotion effects on cognition & behavior • … responsive to dynamic scenario & player behavior • … in real-time Affect Type Alternatives What does my NPC need? Agent issues observable events, actions, architecture complexity, motivation Game issues Computational complexity available for emotion,Intended player experience Emotion issues Theory selection, Emotion representation and dynamics, Emotion effects on cognition, expression, behavior Implementation Alternatives Emotion Architecture Overview • Requirements for Affective NPCs • Affect-focused game design perspective • Tools for analysis of affective requirements within game • Tools for design and development of affective NPCs - Affective Game Engines