270 likes | 888 Views
11/15/2011 11:40 AM. 2. Overview. MotivationPsychological View of Nonverbal BehaviorEncoding Vs. DecodingNonverbal BehaviorExpression of Emotion through Nonverbal BehaviorArtistic View of Emotional ExpressionRealizing Emotional Expression. 11/15/2011 11:40 AM. 3. Motivation. What is interesting about the display of emotion?The development of Virtual characters BelievableCapable of open-ended interactionEngaging For this, a character must be able to signal its internal state through i9459
E N D
1. 11/15/2011 7:40 PM 1 The Display of Emotion Brent Lance
CS 543 Lecture Hi, my name is Brent Lance. I’d like to take the opportunity to thank you all for coming today. I’ll be presenting my thesis proposal on generating expressive gaze manner.Hi, my name is Brent Lance. I’d like to take the opportunity to thank you all for coming today. I’ll be presenting my thesis proposal on generating expressive gaze manner.
2. 11/15/2011 7:40 PM 2 Overview Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
3. 11/15/2011 7:40 PM 3 Motivation What is interesting about the display of emotion?
The development of Virtual characters
Believable
Capable of open-ended interaction
Engaging
For this, a character must be able to signal its internal state through its external behavior So, what’s the problem? We’re interested in creating virtual characters. These characters, whatever domain they’re in, need to be believable and engaging. We believe that for this to be true, a character must be able to signal its internal state through its external behavior. For example, here’s a very expressive character here clearly displaying his internal emotional state.So, what’s the problem? We’re interested in creating virtual characters. These characters, whatever domain they’re in, need to be believable and engaging. We believe that for this to be true, a character must be able to signal its internal state through its external behavior. For example, here’s a very expressive character here clearly displaying his internal emotional state.
4. 11/15/2011 7:40 PM 4 Virtual Characters Keyframe Animation
Classic technique from 2D cell animation
Skilled animator develops key frames
Interpolation between them creates movement
Slow, iterative process
Produces most realistic behavior
Takes very long time to develop
Animations are not reusable
5. 11/15/2011 7:40 PM 5 KeyFrame Animation Here’s a video of emotional state being expressed in a large part through gaze. Go through it, and act it out.
Here’s a video of emotional state being expressed in a large part through gaze. Go through it, and act it out.
6. 11/15/2011 7:40 PM 6 Motion Capture Motion Capture Animation
Cameras record movement of sensors placed on an actor
Software reconstructs movement and approximation of character’s anatomy
Allows for quicker generation of movement
Requires expensive hardware and analysis tools
Captured sequences not reusable
Can be difficult to retarget capture motion to 3D model being animated
7. 11/15/2011 7:40 PM 7 Motion Capture
8. 11/15/2011 7:40 PM 8 Procedural Animation Generation of movement without human oversight
Virtual Agents
Video Game Characters
Can use libraries of keyframe or motion capture animation
Allow for dynamic or interactive environments
Do not have the quality of previous two methods
9. 11/15/2011 7:40 PM 9 Procedural Animation I’m now going to show you a video from one of my favorite games, Oblivion by Bethesda Studios. There’s a lot of character, and I’d like you to look at how they are expressing emotionI’m now going to show you a video from one of my favorite games, Oblivion by Bethesda Studios. There’s a lot of character, and I’d like you to look at how they are expressing emotion
10. 11/15/2011 7:40 PM 10 Motivation - Revisited Hopefully, knowledge of how emotion is expressed through nonverbal behavior can improve interactive virtual agents to the point where they are as engaging as traditionally animated characters
11. 11/15/2011 7:40 PM 11 Overview Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
12. 11/15/2011 7:40 PM 12 Nonverbal Behavior Encoding
The display of information through external behavior
Decoding
The comprehension of information through observation of external behavior
13. 11/15/2011 7:40 PM 13 Encoding Types of encoding
[Ekman & Friesen, 1969] define three types of coding
Arbitrary
No relation between coding and meaning
Iconic
Coding is representation of meaning
Intrinsic
Coding is meaning
14. 11/15/2011 7:40 PM 14 Complications Encoding may not be deliberate
Encoding may be used to mask actual information
Encoding may be idiosyncratic
Men and women encode & decode differently
Different cultures have different coding rules
15. 11/15/2011 7:40 PM 15 Realistic vs. Believable Decoding research is easier than Encoding research
Generating Believable expressive behavior is easier than generating Realistic expressive behavior
“Realistic” -> behavior as a human would have performed it
“Believable” -> behavior understandable by an observer
16. 11/15/2011 7:40 PM 16 Overview Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
17. 11/15/2011 7:40 PM 17 Nonverbal Communication Technically, communication without words
Extremely broad field [Knapp & Hall, 1997]
Includes:
Environment
Appearance
Dress, grooming
Height, weight
Race & gender
None of these really express emotion
However, they may affect appraisal ratings
18. 11/15/2011 7:40 PM 18 Overview Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
19. 11/15/2011 7:40 PM 19 Expression of Emotion What nonverbal behaviors display emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
Expression of Emotion in Man and Animals [Darwin, 1872].
20. 11/15/2011 7:40 PM 20 Posture - Demonstration
21. 11/15/2011 7:40 PM 21 Posture - Demonstration
22. 11/15/2011 7:40 PM 22 Posture - Demonstration
23. 11/15/2011 7:40 PM 23 Torso Posture Torso Posture is affected less by emphasis and other speech-related behaviors [Bull & Connelly, 1985]
Positive and Negative emotions are both easily distinguished both from static body postures [Coulson, 2004], [Schouwstra & Hoogstraten, 1995], and from dynamic trunk movement [de Meijer, 1989].
24. 11/15/2011 7:40 PM 24 Expression of Emotion What nonverbal behaviors display emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
25. 11/15/2011 7:40 PM 25 Head Position
26. 11/15/2011 7:40 PM 26 Head Position Movement of the head is tightly connected to speech [Hadar et al., 1985], [McClave, 1999].
Static posture of the head is still useful for the display of emotions
Head which is tilted upwards is viewed as more dominant, and displaying a more positive than one tilted downwards [Kappas et al., 1994], [Mignault & Chaudhuri, 2003].
27. 11/15/2011 7:40 PM 27 Expression of Emotion What nonverbal behaviors display emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
28. 11/15/2011 7:40 PM 28 Facial Expression Facial expression is one of the stronger channels for emotional expression
Paul Ekman
Performed many of early experiments on facial expression
Found that facial expression was highly cross cultural
Developed the Facial Action Coding System (FACS)
29. 11/15/2011 7:40 PM 29 Happiness
30. 11/15/2011 7:40 PM 30 Anger
31. 11/15/2011 7:40 PM 31 Sadness
32. 11/15/2011 7:40 PM 32 Fear
33. 11/15/2011 7:40 PM 33 Disgust
34. 11/15/2011 7:40 PM 34 Facial Expression Other researchers have shown that
Children who are Deaf/Blind from birth share many of the same facial expressions
As do very young infants
As well as monkeys
[Knapp & Hall, 1997]
35. 11/15/2011 7:40 PM 35 Expression of Emotion What nonverbal behaviors display emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
36. 11/15/2011 7:40 PM 36 Hand Gestures Much more research has been done on the association of hand gestures with the communication of concepts
However, research has demonstrated that level of arousal, and to a lesser extent the valence of emotion can be determined through arm movement [Pollick et al., 2001]
37. 11/15/2011 7:40 PM 37 Expression of Emotion What nonverbal behaviors display emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
38. 11/15/2011 7:40 PM 38 Gaze Behavior What do we mean by gaze behaviors?
Eye shape and eyebrow position are part of facial expression
Length of gaze & mutual gaze
Gaze attraction vs. aversion
39. 11/15/2011 7:40 PM 39 Non-Emotional Gaze Gaze is closely linked to speech-related behaviors
Patterns of gaze for individuals speaking are often different from patterns of gaze for listeners [Kleinke, 1986].
Attention and Saliency also control eye movement [Argyle & Cook, 1973].
40. 11/15/2011 7:40 PM 40 Gaze and Dominance Gaze is very important in signaling dominance [Exline, 1971]
However this relationship is not fully understood
Seen in the way monkeys perform dominance displays when looked at by experimenters
Highly dominant individuals look less while listening, and more while speaking than do individuals of low dominance [Knapp, 1997].
41. 11/15/2011 7:40 PM 41 Gaze and Arousal Gaze is also closely related to physiological arousal [Argyle & Cook, 1973]
Receiving gaze and engaging in mutual gaze increases arousal
Gaze is used as a regulation for physiological arousal
Increased blinks, and increased pupil dilation are also strong signals of arousal
42. 11/15/2011 7:40 PM 42 Gaze and Preference Gaze also has a drastic effect on liking
Individuals who gaze very little at people they interact with are rated lower on scales of preference, credibility, attractiveness, and are less likely to be hired [Burgoon et al., 1985], [Argyle & Cook, 1973].
Individuals are also more likely to look at people that they like [Argyle & Cook, 1973]
43. 11/15/2011 7:40 PM 43 Expression of Emotion What nonverbal behaviors display emotion?
Torso Posture
Head Position
Facial Expression
Hand Gestures
Gaze Behavior
Movement
44. 11/15/2011 7:40 PM 44 Movement & Locomotion Emotion can be recognized through how people walk
But further, emotion can be recognized through the movement of simple geometric shapes [Rime et al., 1985]
45. 11/15/2011 7:40 PM 45 Movement
46. 11/15/2011 7:40 PM 46 Locomotion
47. 11/15/2011 7:40 PM 47 Synchrony The interactions of all of these behaviors is as important as any single behavior
If not, emotional signals lack coherency
The ability of users to recognize emotion decreases
Believability and engagement decrease
48. 11/15/2011 7:40 PM 48 Conclusion – Nonverbal While there are many different ways emotion can be signaled through nonverbal behavior
Only facial expression (and possibly prosody) can clearly differentiate between individual emotional categories
Much nonverbal behavior research done using dimensional models of emotion that can be applied to factor analyses of nonverbal behavior results
49. 11/15/2011 7:40 PM 49 Overview Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
50. 11/15/2011 7:40 PM 50 Art & Emotion Looking at three artistic views of emotional expression
Delsarte’s Structural Acting System
Laban Movement Analysis, a system derived from choreography
Emotion in 2D Disney Animation
51. 11/15/2011 7:40 PM 51 Structural Acting System Francois Delsarte (1811-1871)
French opera singer who damaged his voice
Turned to the expression of emotion through acting
Compiled extremely extensive library of movements, and the meanings of those movements
However, this library has not been empirically verified
52. 11/15/2011 7:40 PM 52 Delsarte Recently, [Marsella et al., 2006] performed an evaluation of some of Delsarte’s work on hand gestures
“Delsarte’s cube”
Small imaginary cube directly in front of chest
Placement of hands on cube sends different signals
i.e. hands placed on inside surface of near face of cube to reveal possession
53. 11/15/2011 7:40 PM 53 Laban Movement Analysis Rudolph Laban (1879-1958)
Hungarian Dancer and Choreographer
Developed Labanotation, a notation for recording dance
Rendered mostly obsolete through widespread distribution of video recording technology
Developed, in conjunction with several of his students, Laban Movement Analysis, a taxonomy for describing movement
54. 11/15/2011 7:40 PM 54 Laban Movement Analysis Qualitative, hierarchical taxonomy
Effort
Space
Indirect/Direct
Time
Sustained/Sudden
Weight
Light/Strong
Flow
Free/Bound
Shape
55. 11/15/2011 7:40 PM 55 EMOTE [Chi et al., 2000] used Laban parameters Effort and Shape to animate emotion
Theory was that instead of specifying emotional state, would instead specify LMA parameters to animate a character
Extended by [Zhao & Badler, 2005] which learned LMA parameters from motion capture or video recording
56. 11/15/2011 7:40 PM 56 Disney Animation Principles of Disney animation laid out in [Thomas & Johnston, 1981], extended to 3D animation in [Lasseter, 1994].
57. 11/15/2011 7:40 PM 57 Keyframe Animation Skilled animators drew key frames
2nd Tier drew the “in-betweens”
Developed principles such as “Squash & Strech”, “Anticipation”, and “Follow Through”
58. 11/15/2011 7:40 PM 58 2-D Animation Extremely slow process
Highly iterative
Highly intuitive
Very difficult to clearly explain
Requires great deal of training, and a certain amount of talent
Results are excellent
59. 11/15/2011 7:40 PM 59 Analyzing 2D Animation [Lance et al., 2004], and [Chafai et al., 2006] were attempts to analyze 2D animation for information on nonverbal behavior for gaze and gesture, respectively
Moderately successful
Extremely labor intensive
60. 11/15/2011 7:40 PM 60 Overview Motivation
Psychological View of Nonverbal Behavior
Encoding Vs. Decoding
Nonverbal Behavior
Expression of Emotion through Nonverbal Behavior
Artistic View of Emotional Expression
Realizing Emotional Expression
61. 11/15/2011 7:40 PM 61 Realizing Behavior
62. 11/15/2011 7:40 PM 62 Animation Curves
63. 11/15/2011 7:40 PM 63 Animation of Behavior Qualitative descriptions of nonverbal behavior need to be transformed into quantitative changes in animation
Two Choices:
Directly manipulate the animation curves yourself
Optimization
Develop and utilize an abstraction, such as EMOTE or the Body Markup Language (BML)
64. 11/15/2011 7:40 PM 64 Conclusion Many sources of information for the expression of emotion through human behavior
Lots of them are qualitative
Or not experimentally verified
Or describe more subtle relationships than direct emotional state -> behavior
Picture is still very incomplete
Still lots of good information for a starting point!
65. 11/15/2011 7:40 PM 65 References Argyle, M., Cook, M. Gaze and Mutual Gaze. Cambridge University Press. 1976.
Bull, P., Connelly, G. Body Movement and Emphasis in Speech. Journal of Nonverbal Behavior, 9(3). Fall 1985.
Burgoon, J., Manusov, V., Mineo, P., Hale, J. Effects of Gaze on Hiring, Credibility, Attraction and Relational Message Interpretation. Journal of Nonverbal Behavior. 9(3). Fall 1985.
Chafai, N., Pelachaud, C., Pele, D., Breton, G. Gesture Expressivity Modulations in an ECA Application. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference. 2006.
Coulson, M. Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence. Journal of Nonverbal Behavior. 28(2). Summer 2004.
Chi, D., Costa, M., Zhao, L., Badler, N. The EMOTE Model for Effort and Shape. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing, New York, NY. 2000.
Darwin, C., The Expression of Emotions in Man and Animals. 1872.
Ekman P., Friesen, W.V., The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding. Semiotica. 1, 49-98. 1969.
Exline, R. Visual Interaction: The Glances of Power and Preference. In Weitz, S. (ed), Nonverbal Communication: Readings with Commentary. Oxford University Press, 1974.
Hadar, U., Steiner, T. J., Rose, F. C. Head Movement During Listening Turns in Conversation. Journal of Nonverbal Behavior, 9(4). Winter 1985.
Kappas, A., Hess, U., Barr, C., Kleck, R. Angle of Regard: The Effect of Vertical Viewing Angle on the Perception of Facial Expressions. Journal of Nonverbal Behavior. 18(4). Winter 1994.
Kleinke, C. Gaze and Eye Contact: A Research Review. Psychological Bulletin. v. 100, n. 1. 1986.
66. 11/15/2011 7:40 PM 66 References (Cont.) Knapp, M., Hall, J. Nonverbal Communication in Human Interaction. Fourth Edition. Harcourt Brace Publishers. 1997.
Lance, B., Marsella, S., Koizumi, D. Towards Expressive Gaze Manner in Embodied Virtual Agents. AAMAS Workshop on Empathic Agents. 2004.
Lasseter, J. Principles of Traditional Animation Applied to 3D Computer Animation. Computer Graphics. v. 21, n 4. July 1987.
Marsella, S., Carnicke, S. M., Gratch, J., Okhmatovskaia, A., Rizzo, A. An Exploration of Delsarte’s Structural Acting System. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference. 2006.
McClave, E. Linguistic Functions of Head Movements in the Context of Speech. Journal of Pragmatics. v. 32, 2000.
de Meijer, M. The Contribution of General Features of Body Movement to the Attribution of Emotions. Journal of Nonverbal Behavior. 13(4). Winter 1989.
Mignault, A., Chaudhuri, A. The Many Faces of a Neutral Face: Head Tilt and Perception of Dominance and Emotion. Journal of Nonverbal Behavior. 27(2). Summer 2003.
Pollick, F., Paterson, H., Bruderlin, A., Sanford, A. Perceiving Affect from Arm Movement. Cognition. i 82, B51-B61. 2001.
Rime, B., Boulanger, B., Laubin, P., Richir, M., Stroobantst, K. The Perception of Interpersonal Emotions Originated by Patterns of Movement. Motivation and Emotion. v. 9, n. 3. 1985.
Schouwstra, S., Hoogstraten, J. Head Position and Spinal Position as Determinants of Perceived Emotional State. Perceptual and Motor Skills. v. 81, 1995.
Thomas, F., Johnston, O. The Illusion of Life: Disney Animation. Walt Disney Productions. 1981.
Zhao, L., Badler, N. Acquiring and Validating Motion Qualities from Live Limb Gestures. Graphical Models. v. 67 i. 1. January, 2005.