Affective Computing: Machines with Emotional Intelligence

1. Affective Computing: Machines with Emotional Intelligence Hyung-il Ahn MIT Media Laboratory

2. Skills of EmotionalIntelligence: • Expressing emotions • Recognizing emotions • Handling another’s emotions • Regulating emotions \ • Utilizing emotions / (Salovey and Mayer 90, Goleman 95) if “have emotion”

3. We have pioneered new technologies to recognize human affective information:Sensors, pattern recognition and common sense reasoning to infer emotion from physiology, voice, face, posture & movement, mouse pressureMind-Read: Recognizing complex cognitive-affective states from joint face and head movements

4. Future “teacher for every learner”

5. Sit upright Lean Forward Slump Back Side Lean Can we teach a chair to recognize behaviors indicative of interest and boredom? (Mota and Picard)

6. What can the sensor chair contribute toward inferring the student’s state: Bored vs. interested? Results (on children not in training data,Mota and Picard, 2003): 9-state Posture Recognition: 89-97% accurate High Interest, Low interest, Taking a Break: 69-83% accurate

7. Detecting, tracking, and recognizing facial expressions from video (IBM BlueEyes camerawith MIT algorithms)

8. Affective-Cognitive Mental StatesBaron-Cohen et al. AUTISM RESEARCH CENTRE, CAMBRIDGE Assertive Committed Persuaded Sure Agreeing Absorbed Concentrating Vigilant Concentrating Complex Mental States (subset) Disapproving Discouraging Disinclined Disagreeing Asking Curious Impressed Interested Interested Brooding Choosing Thinking Thoughtful Thinking Baffled Confused Undecided Unsure Unsure

9. Technology that understands and responds to human experience like a caring, respectful person would, for example: Knows when a person/customer is: • Concentrating, and does not interrupt unless very important • Thinking, and can pause to let you think • Unsure, and can offer to explain differently • (Not) interested in what it says • (Dis)agreeing, and can adjust response respectfully

10. Technology with people sense will perceive cognitive-affective states, e.g., before interrupting hmm … Roz looks busy. Its probably not a good time to bring this up Inference and reasoning about mental states Modify one’s actions Persuade others Analysis of nonverbal cues

11. Inferring Cognitive-Affective State from Facial+Head movements (el Kaliouby, 2005) • Experimental Evaluation • Conclusions Facial feature extraction Head & facial action unit recognition Head & facial display recognition Mental state inference Head pose estimation Hmm … Let me think about this Feature point tracking Other examples: Agree Disagree

12. Robotic Computer (RoCo) : World’s first physically animated computer 75% sit in front of computers (static) Back pain/injury = #2 cause of missed work Physical movement helps prevent/reduce back pain Goals : • Fostering healthy posture • Building social rapport • Improved task performance (Affect-Congruent behavior) Animated Desktop Monitor: RoCo = Robotic Computer

13. RoCo Behavior

14. When should RoCo move? (Future work & not topic of this paper, but important to mention) • NOT when: you’re concentrating, interested, in the middle of an engaging task, or otherwise attentive/focused on the monitor’s content. • Might make a micro-movement when you’re looking away or blinking in the middle of a task. • Might make a larger movement to attract a new user, bow to welcome, or when user shifts tasks and hasn’t shifted posture (etc.)

15. RoCo’s postures congruous to the user affect “Stoop to Conquer” : Posture and affect interact to influence computer users’ comfort and persistence in problem solving tasks People tend to be more persistent and feel more comfortable when RoCo’s posture is congruous to their affective state N=(17)

16. “Stoop to Conquer”: Posture congruent with emotion improves persistence (# tracing attempts, two different experiments)

17. We are creating new computational models to measure human affective experience and to predict human decision-making & preference A multi-modal affective-cognitive measures for product evaluation with computational models of predicting customer decisions Predicting customer product preferences by combining information about emotion and cognition

18. Background findings to inform new research: The brain uses both emotion (affect) and cognition in decision making -> model should combine both affect and cognition A person in an experiment is likely to cognitively bias their self-report of what they like. -> method should not rely on only self-report When a person is cognitively loaded they are more likely to use emotion in decision-making. -> method should slightly load person cognitively

19. Background findings to inform new method: Multiple measures of affect provide most robust assessment: -> method can measure affective physiology (face, skin conductance) as well as behavior and self-report Sweeter beverages are preferred on the first sip; long-term accumulation of something mildly bad is required before it is “bad enough to notice” -> method should require lots of sips of every beverage

20. More complete understanding of consumer desire Multi-Dimensional Response Facial Expression AFFECTIVE LIKING Emotions Skin Conductance ANTICIPITORY FEELING Arousal Physical NUMBER OF SIPSAmount Consumed Self Report COGNITIVE LIKING Purchase intent Liking Expectation

21. Videos of Testing • Here is a sneak preview of my project. Make sure to look for consumers emotions that may not be captured in self reported questions.

22. Test Products Products chosen with clear performance differences • Stronger Performer – • Pepsi Vanilla • Performed in top 25%, green region, in Directions HUT • Weaker Performer – • Pepsi Summer mix • Performed in lower 40%, lower yellow region, in Directions HUT

23. Affective Computing • Two techniques performed simultaneously • Facial Imaging and Head Positioning Tracking face muscle movements to interpret emotions • Galvanic Skin Response (GSR) Measures Arousal, used as an intensity measure for emotions

24. Affective-Cognitive Mental States Facial HeadExpression Position + = Interpretation • Concentrating • Thinking • Confused • Interested • Agreeing • Disagreeing GSR Shows Intensity

25. Method: Choice Technique • Choice technique - respondent selected one of two vending machines • Process is repeated 30 times • Eventually respondents realized each machine favors a different product and will select the vending machine hoping to receive their favored product • 70/30 probability of either product coming out of either machine

26. Method - General Set-Up Machine 1 Machine 2 135 135 246 246 Two cups on each side of the computer: Pepsi Vanilla and Pepsi Summer Mix Use of straws avoided blocking facial reaction

27. Machine Selection Sip on Resulted Beverage Answer Questions Experimental Set Up

28. Method - Step 1 RANDOMLY CHOOSE A VENDING MACHINE • Each vending machine directed you to sip a beverage

29. Method- Step 2 RESPONDENTS SIP RESULTED BEVERAGE

30. Method – Step 3 • Answer Questionnaire used in standard CLT • Overall Liking (beverage and machine) • Purchase Intent, Comparison to Expectation

31. Method – Step 4 • Reselect a machine • 30 machine selections were made

32. Data collection timeline Data collected throughout experiment Choice 1 70% Vanilla 30% Mix Choice 2 70% Mix 30% Vanilla Start vanilla or mix How much do you like the sip? Select Outcome Sip Question Measuring ANTICIPITORY FEELING (hope/dread) Skin conductance Evaluate Start (Next trial) Measuring AFFECTIVE LIKING (initial reaction) Facial expression Skin conductance Measuring COGNITIVE LIKING (post reaction) Self-report

33. Videos of Testing

34. Videos of Testing

35. Videos of Testing

36. Videos of Testing

37. Discussion Analysis • Our hypothesis is that joining quantitative and qualitative methodologies will help provide understanding of consumers’ real product evaluations