Affective computing and the role of emotions in computer science and engineering

1. Affective computing and the role of emotions in computer science and engineering Antonio Camurri Corso di Progettazione e Produzione Multimediale

2. Goals • Make computers able to - recognize emotions - express emotions - respond intelligently to human emotion - regulate and utilize human emotions • Enormous potential for • progress in science (understanding of cognitive and emotional phenomena) • novel computer applications • novel (anthropomorphic) interfaces to existing applications. • Main “schools”: • (USA) “Affective Computing”, Rosalind W. Picard, 1997, MIT Press. • (Japan) “KANSEI Information Processing”, Intl. Workshop, 1997. • (EU) Emotion research (Scherer) , Empathy and entrainment

3. Applications • Applications where computers have a social role (“instructor”, “tutor”,“helper”, “companion”): to be able to recognize users’ emotions may enhance their functionality. • Rehabilitation and therapy software: e.g. for autistic children, for monitoring motor and cognitive tasks in patients affected by Parkinson or Alzheimer. • Affective robots (anthopomorphic or not): emotional behavior to reduce stress, fear, and increase trust in users. For example: support to independent living in elder people (home activities, intelligent prosthesis), • ECAs (Embodied Conversational Agents): model relationships between emotions and behaviour can help to synthesize emotion-oriented ECAs • To improve planning and decision systems: emotions influence decision processes: e.g., in game theory: emotion models to boost performance of existing techniques and algorithms. • Human Computer Interaction (HCI): To measure cognitive and motoric performance in users of computer applications • To monitor users’ stress level • Social networked media, games, and entertainment • Artistic applications • Edutainment, children educational software and affective robots • Security systems: behavior monitoring of individuals and crowd in public spaces (e.g. airports)

4. Emotion theories • Emotions are -cognitive: mental component of emotion is emphasized; appraisal theories - physical: bodily component is emphasized; physiological responses • Difficulties to define and ambiguous: “emotion” can mean different things, hard to define scientifically: • “I feel happy” (feeling an emotion) • “I want to make understand to others happiness” (cognitive, the tactor viewpoint in a theatre play) • “I feel engaged with someone” (entrainment, empathy) • “music or artistic emotions” (emotions in audience, between musicians playing together)

5. Emotion theories Computational models and related affective software systems are grounded on results from other disciplines: • From psychology and cognitive sciences: • discrete emotion theories (basic emotions: e.g. joy, sadness, fear, anger) • dimensional theories, continuous models (e.g. valence/arousal spaces) • From neurosciences: • Novel mind and brain theories: embodiment (enaction, active perception, mirror neurons, etc.) • From physics and complex dynamic system theories: • empathy and entrainment: based on theories of synchronisation.

6. Open problem: how to model an emotional process? • Psychological models • Physiological models • Computational models of emotions: • analysis of emotions • synthesis of emotional behavior (in robots, software agents). • Important for successful approaches: emotions do not refer (only) to states, it is fundamental to take into account the dynamics of an emotional process

7. The Component-Process Model (from psychology) (1) • Developed by Klaus Scherer (1984, 2000, 2001; Geneva Emotion Research Group, Dept. Psychology, Univ. Geneve) • Emotions as constantly changing phenomena integrating more components • Five subsystems - the cognitive system (appraisal) - the autonomic nervous system (arousal) - the motor system (expression) - the motivational system (action tendencies) - the experiential system (feeling) • Synchronization of all the processes

8. Component-Process Model (2) • Emotional processes arise from: - Appraisal process - Memory - Empathy

9. Component-Process Model (3) • Corpora for studying and analysing emotions: • How to validate algorithms for emotion analysis and recognition? • The GEMEP Archive: an annotated collection of short video recordings of emotions, portraied by professional theatre actors, following the Scherer’s emotion model • (video demo)

10. Building affective and emotional computing systems • Affective signals and systems • Recognizing and expressing affect: from analysis to synthesis • Validating affective computing systems • Emotion synthesis

11. Affective signals and systems

12. The physiology of emotions • The limbic system (amygdala) is involved in emotional processes - Through its direct connections with the hypothalamus, it modulates the autonomic nervous system activity - It receives information from the thalamus - Low and high roads to amygdala (LeDoux, 1996) Sensory cortex High road Sensory thalamus Amygdala Low road Emotional responses Emotional stimulus

13. Properties of affective signals (Picard, 1997) • Response decay An emotional response is of relatively short duration, and will fall below a level of perceptibility unless it is re-activated

14. Properties of affective signals • Repeated strikes Rapid repeated activation of an emotion causes its perceived intensity to increase

15. Properties of affective signals • Temperament and personality influences A person’s temperament and personality influence emotion activation and response

16. Properties of affective signals • Non-linearity The human emotional system is non-linear, but may be approximated as a linear system for a certain range of inputs and outputs

17. Properties of affective signals • Time-invariance The human emotional system can be modeled as independent of time for certain durations

18. Properties of affective signals • Activation Not all inputs can activate an emotion; they have to be of sufficient intensity. This intensity is not a fixed value, but depends on factors such as mood, temperament, cognitive expectation, context

19. Properties of affective signals • Saturation No matter how frequently an emotion is activated, at some point the system will saturate and the response of the person will no longer increase. Similarly, the response cannot be reduced below a “zero” level

20. A typical representation for emotion signals • The input of an emotional system is a complex function of cognitive and physical events • It is often approximated by a simple nonlinear function applied to the inputs to the emotional system: • X = input stimuli (originating inside and/or outside the person); • Y = the sigmoid function; • S = controls the steepness of the slope (depends on personality); • x0 = shifts the sigmoid left or right; • g = controls the gain applied by the sigmoid; • y0 = shifts the sigmoid up or down • Different activation and saturation levels

21. Properties of affective signals • Cognitive and physical feedback Inputs to the system can be initiated by internal cognitive or physical processes. The physiological expression of an emotion can provide a feedback which acts as another input to the system, generating another emotional response

22. Properties of affective signals • Background mood All inputs contribute to a background mood, whether or not they are below the activation level for emotions. The most recent inputs have the greatest influence on the present mood

23. Where affective signals come from? • Speech: prosody, syntactic analysis (which word thesaurus and how concepts are expressed) • Facial expressions (FACS: Ekman 1977) • Body movement and gesture (Accelerometers, videocameras, other movement sensors) • Physiological signals: EMG, EEG, EGC, … • Social parameters (from analysis of social media)

24. From inputs to Emotion models • Affective inputs need to be processed in order to determine higher level parameters, which define the coordinate of emotion spaces. • Dimensional emotion spaces in the literature: • Russel valence-arousal (2D) • Wundt, Scherer valence-arousal-power (3D)

25. Valence-Arousal theories • Use emotional signals to model valence and arousal components • For both analysis and synthesis purposes

26. Russel’s valence-arousal model arousal high activity angry enthusiasm fear happy negative neutral positive valence sad serene low activity In Russel’s model, Basic emotions are located in a circle centered on the neutral

27. Recognizing and expressing affect (from analysis to synthesis)

28. Multimodal emotion dialogue

29. Multimodal emotion recognition • Which cues do reveal emotions, preferences, and attentional states of a person? • How integrated audio, visual, and possibly physiological inputs can enable more accurate or efficient emotion recognition with respect to using a single modality? • How to individuate and collect multimodal data responsible for emotional expressions, and how should it be used in an emotional software? • Are we measuring spontaneous or posed data?

30. Multimodal emotion recognition • Labelled or continuous emotion? The labeling of emotions into different states led researchers toward pattern recognition approaches to recognizea nd alcssify emotions, using different modalities as inputs to the emotion recognition models • Face: optical flow estimation to recognize facial expressions • Voice: speech rate, pitch average, intensity, prosody cues, etc. • Physiological patterns: heart rate, breath rate, skin conductance, ... • Posture and postural attitude • Expressive gesture: fluidity, quantity of motion, repetitivity, contraction/expansion, symmetry, …

31. Multimodal approach to emotion recognition • Affect recognition is more accurate when it combines multiple modalities, information about the user’s context, situation, goal, and preferences • Probabilistic models:

32. Collecting multimodal data for emotion recognition Five factors influence the affective data collection (Picard et al.) • Spontaneous vs posed • Lab setting vs real world • Expression vs feeling • Open recording vs hidden recording • Emotion-purpose vs other purpose

33. From analysis to synthesis • Manual annotation of affective videos • Automatic extraction of features in affective videos

34. Emotion Synthesis

35. The Ortony Clore Collins (OCC) Cognitive Model • Aim: cognitive emotions synthesis • Model of cognitive appraisal for emotions • Idea: representing emotions not by using sets of basic emotions or dimensioned space, but by grouping emotions according to cognitive eliciting conditions

36. The Ortony Clore Collins (OCC) Cognitive Model • Emotions arise from valenced (positive or negative) reactions to situations consisting of events, agents and objects • Specifications for 22 emotion types • Disadvantages: low-level details of implementation are not present in the model (e.g., how emotions interact, mix and change their intensity with time, etc.)

37. D(p, e, t) = desireability e = event p = person t = time Ig (p, e, t) = global intensity variables, expectedness, proximity, reality Pj(p, e, t) = potential for generating a state of joy j Tj = threshold value for joy Ij = intensity of joy IF D(p, e, t) > 0 THEN set Pj(p, e, t) = fj(D(p, e, t), Ig (p, e, t)) where fj() is a function specific to joy j IF Pj(p, e, t) > Tj(p,t) THEN set Ij(p, e, t) = Pj(p, e, t) – Tj(p, t) ELSE set Ij(p, e, t) = 0 OCC-rule example Synthesis of Joy = returns positive value if event is expected to have beneficial consequences and viceversa

38. A computational model for the synthesis of artificial emotions:an application to interactive multimodal environments (MEs)(Camurri, Ferrentino, Dapelo; 1997)

39. Interactive MEs • We call interactive ME an active space populated by agents observing and interacting with users, allowing them to communicate by means of full-body movement, singing, playing • Users get feedback from an ME in real time in term of sound, music, visual media and actuators • ME agents should be capable of changing their character • Development of a model of artificial emotions to implement the emotional component of such a ME agents architecture

40. Example of a possible scenario • Two different agents involved during artistic performances • An agent capable to extract from humans some gesture and movement features and to control the generation of sound and music • Another agent based on a robotic platform on wheels that manages the movement of a robot • Agents communicate to each other • Social interaction between the robotic agent and the performers

41. Machine gesture • ME communicates to humans by means of the movement and gesture of its physical (or robotic) agents • Robot as multimodal system: integration of movements, audio, music and animated artificial face • Robot emotional behaviour changes according to the emotional characterization of the stimuli • Robot emotional behaviour in terms of movement behaviour and sound and music outputs

42. The model of artificial emotions (1) • It consists of a two-dimensional space which can be navigated • A point in such space represents the character of the agent • The two axes represent the degree of affection of the agent towards itself and towards others, respectively • We call these two axis “EGO” and “NOS” • E.g., a point placed in the positive x (EGO) axis represents an agent whose character is in a good disposition towards itself • The emotion space is partitioned into thirteen regions

43. The model of artificial emotions (2) • Each region is labeled by the kind of character the agent simulates • A point in a region represents the character of the agent and an area around such point represents its mood or disposition • Each region is defined by a maximum and minimum threshold for the quantities of the EGO and NOS character value • For kinds of stimuli: positive-EGO, positive-NOS, negative-EGO, negative-NOS • For each region, the behaviour is defined by the four kinds of stimuli: character moves in response to these four stimuli • For each region there are four styles of movements to define

44. Applications: a music example • Generation of a musical output reflecting the emotional state of an agent-robot navigating on stage • Aim: obtaining a performance presenting the flow of emotions in the agent by means of the integration of music and movement of the agent-robot