Affect in games

1. CIG 2012 tutorial Kostas Karpouzis National Technical University of Athens kkarpou@cs.ntua.gr Affect in games

2. what I’ll talk about • Concepts of affect/emotion/behaviour • (hopefully!) related to games • Engagement/attention/flow • How to detect/make sense of them • Modelling/detecting player experience and sattisfaction • and adapting to

3. what I won’t talk about • Designing affective NPCs • which show happiness, sadness, contempt, etc. • Affect and in-game behaviour • other than attention/engagement • e.g. fatigue and game play, how happy/sad players perform, etc. • Particular affect detection techniques • But we can discuss those later

4. yours truly • Post-doc in Humaine Network of Excellence • Since: • CALLAS (emotion in arts and entertainment) • Feelix Growing (affect-aware robots) • Currently: • Siren (serious games for conflict resolution in school environments) • ILearnRW (serious games for children with dyslexia and dysorthographia) • Also: • Humaine Games SIG (w/ G. Yannakakis), IEEE TAC SI (Affect in Games) • IEEE Aff. Comp TF, IEEE CIS Games TC TF on Education

5. affective games • Toys provide our first human-“machine” interface! • Games are everywhere these days! • Computers, consoles/TV, mobile phones • Browser and Facebook/Google+ do not require installation • ‘Freemium’ business model attractive to both gamers and developers • Huge interest in industry and academia • Lots of funding from EU projects 

6. affective games • Constrained environment  easier to track people • Novel modalities: gestures, body movement, speech (lexical and prosody) • Ideally associated with what happens in the game • (Possibly) novel interaction paradigms, i.e. not WIMP

7. affective gamers

8. or(if things go wrong)…

9. affective gamers

10. let’s make use of it 

11. a bit of theory theories of emotion

12. terminology • Emotions, mood, personality • Can be distinguished by • time (short-term vs. long-term) • influence (unnoticed vs. dominant) • cause (specific vs. diffuse) • Affect classified by time • short-term: emotions (dominant, specific) • medium-term: moods (unnoticed, diffuse) • and long-term: personality (dominant)

13. terminology • what we perceive is the expressed emotion at a given time • on top of a person’s current mood, which may change over time, but not drastically • and on top of their personality • usually considered a base line level • which may differ from what a person feels • e.g. we despise someone, but are forced to be polite

14. terminology • Affect is an innately structured, non-cognitive evaluative sensation that may or may not register in consciousness • Feeling is defined as affect made conscious, possessing an evaluative capacity that is not only physiologically based, but that is often also psychologically oriented. • Emotion is psychosocially constructed, dramatized feeling

15. how it all started • Charles Darwin, 1872 • Ekman et al. since the 60s • Mayer and Salovey, papers on emotional intelligence, 90s • Goleman’s book: Emotional Intelligence: Why It Can Matter More Than IQ • Picard’s book: Affective Computing, 1997

16. why emotions? • “Shallow” improvement of subjective experience • Reason about emotions of others • To improve usability • Get a handle on another aspect of the "human world" • Affective user modeling • Basis for adapting experience to users

17. name that emotion • so, we know what we’re after • but we have to assign it a name • in which we all agree upon • and means the same thing for all (most?) of us • different emotion representations • different context • different applications • different conditions/environments

18. emotion representations • most obvious: labels • people use them in everyday life • ‘happy’, ‘sad’, ‘ironic’, etc. • may be extended to include user states, e.g. ‘tired’, which are not emotions • CS people like them • good match for classification algorithms

19. labels • but… • we have to agree on a finite set • if we don’t, we’ll have to change the structure of our neural nets with each new label • labels don’t work well with measurements • is ‘joy’ << ‘exhilaration’ and in what scale? • do scales mean the same to the expresser and all perceivers?

20. labels • Ekman’s set is the most popular • ‘anger’, ‘disgust’, ‘fear’, ‘joy’, ‘sadness’, and ‘surprise’ • added ‘contempt’ in the process • Main difference to other sets of labels: • universally recognizable across cultures • when confronted with a smile, all people will recognize ‘joy’

21. from labels to machine learning • when reading the claim that ‘there are six facial expressions recognized universally across cultures’… • …CS people misunderstood, causing a whole lot of issues that still dominate the field

22. strike #1 • ‘we can only recognize these six expressions’ • as a result, all video databases used to contain images of sad, angry, happy or fearful people • a while later, the same authors discussed ‘contempt’ as a possible universal, but CS people weren’t listening

23. strike #2 • ‘only these six expressions exist in human expressivity’ • as a result, more sad, angry, happy or fearful people, even when data involved HCI • can you really be afraid when using your computer?

24. strike #3 • ‘we can only recognize extreme emotions’ • now, happy people grin, sad people cry or are scared to death when afraid • however, extreme emotions are scarce in everyday life • so, subtle emotions and additional labels were out of the picture

25. labels are good, but… • don’t cover subtle emotions and natural expressivity • more emotions are available in everyday life and usually masked • hence the need for alternative emotion representations • can’t approach dynamics • can’t approach magnitude • extreme joy is not defined

26. other sets of labels • Plutchik • Acceptance, anger, anticipation, disgust, joy, fear, sadness, surprise • Relation to adaptive biological processes • Frijda • Desire, happiness, interest, surprise, wonder, sorrow • Forms of action readiness • Izard • Anger, contempt, disgust, distress, fear, guilt, interest, joy, shame, surprise

27. going 2D • vertical: activation (active/passive) • horiz.: evaluation (negative/positive)

28. going 2D • emotions correspond to points in 2D space • evidence that some vector operations are valid, e.g. ‘fear’ + ‘sadness’ = ‘despair’

29. going 2D • quadrants useful in some applications • e.g. detect extreme emotions during team comms in FPSs

30. going 3D • Plutchik adds another dimension • vertical  intensity, circle  degrees ofsimilarity • four pairs of opposites

31. going 3D • Mehrabian considers pleasure, arousal and dominance • Again, emotions are points in space

32. what about interaction? • these models describe the emotional state of the user • no insight as to what happened, why the user reacted and how the user will react • action selection • OCC (Ortony, Clore, Collins) • Scherer’s appraisal checks

33. OCC (Ortony, Clore, Collins) • each event, agent and object has properties • used to predict the final outcome/expressed emotion/action

34. summary on emotion • perceived emotions are usually short-lasting events across modalities • labels and dimensions are used to annotate perceived emotions • pros and cons for each • additional requirements for interactive applications

35. a bit of theory the quest for Flow (and fun)

36. a theory of fun • Raph Koster • lead designer of Ultima Online • creative director of Star Wars Galaxies • http://www.theoryoffun.com/theoryoffun.pdf • http://www.raphkoster.com

41. a theory of fun

42. a theory of fun • ‘We talk so much about emergent gameplay, non-linear storytelling, or about player-entered content. They’re all ways of increasing the possibility space, making self-refreshing puzzles’ • So, what is it that makes a game ‘fun’? • ever-lasting challenge • How can we keep challenges coming?

43. the concept of Flow • a state of concentration or complete absorption with the activity at hand and the situation. It is a state in which people are so involved in an activity that nothing else seems to matter (Csikszentmihalyi,1990) • “Being completely involved in an activity for its own sake. The ego falls away. Time flies. Every action, movement, and thought follows inevitably from the previous one, like playing jazz. Your whole being is involved, and you're using your skills to the utmost”

45. flow revisited • the ‘holy grail’ of game design • just the right amount of challenge • making a game very hardgamers quit • making a game very easygamers bored

46. flow revisited • it’s not about the graphics • or the controller • or the franchise (e.g. sports games) • just ask Rovio • makers of Angry Birds • $80M/yr, 600M dl’s

47. flow revisited • ‘smart’ games adapt to player skill and engagement • keeping them coming back for more • at the end of the day…

48. that’s flow!

49. back to the drawing board • what can we model? • and how? • definition of ‘affective computing’ • ‘affective computing is computing that relates to, arises from, or deliberately influences emotion or other affective phenomena’ -- Roz Picard, 1995 • ‘a set of observable manifestations of a subjectively experienced emotion’ -- Merriam-Webster’s dictionary

50. observable manifestations