270 likes | 377 Views
Towards a virtual agent using similarity-based laughter production. Jérôme Urbain, Stéphane Dupont, Thierry Dutoit, Radoslaw Niewiadomski, Catherine Pelachaud TCTS Lab, Faculté Polytechnique de Mons CNRS - LTCI UMR 5141, Institut TELECOM - TELECOM ParisTech. Context.
E N D
Towards a virtual agent using similarity-based laughterproduction Jérôme Urbain, Stéphane Dupont, Thierry Dutoit, Radoslaw Niewiadomski, Catherine Pelachaud TCTS Lab, Faculté Polytechnique de Mons CNRS - LTCI UMR 5141, Institut TELECOM - TELECOM ParisTech
Context • Desire to communicate with machines as we do with humans • Lack of naturalness, expressivity, perception of emotions • Laughter is a very important signal: • Conveys emotional contents • Communicative • ... <footer>
Objectives • Improve the expressivity of an Embodied Conversational Agent (ECA) by enabling it to laugh: • acoustic laughter production • synchronisation with virtual agent expressions • ability to instantaneously answer to a laughter <footer>
Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming projects <footer>
Multimodal expressions of emotions • Go beyond « static images »/linear expressions (e.g. beyond basic emotionspredictions) • Complete data obtainedfromtheory and literature • Multimodal expressions of emotions • across modalities • many emotions are expressed by sequences (or combination) of multimodal signals rather than monomodal signals (eg static facial expressions) D. Keltner, B. N. Buswell Embarrassment: Its Distinct Form and AppeasementFunctions
Multimodal expressions of emotions • Difficulty: lack of relevant research and video-corpora • new video-corpus of examples of spontaneous multimodal behaviors (based on TV broadcasts) • annotation in Anvil (Kipp 2001) • Specification of Behaviors Sets and Constraints
Corpus of emotional displays • Annotation of audio-visual recordings from reality shows, hidden camera recordings, Belfast Naturalistic database, EmoTV corpus. • The observed people are non actors in emotional situations: natural and not stereotyped multimodal behaviour is displayed • 20 video clips (3 to 14 seconds each): • relief (2), tension (6), joy (2), sadness (2), anger (3), despair (1), fear (4)
Multimodal annotation scheme • Annotation on Anvil v4.7.6 (Kipp, 2001) with 5 tracks: • Emotion (inferred from the situation) • Facial expression (FACS coding) • Head movement • Gaze movement • Gesture
Multimodal Joy annotation Video annotations with the Anvil software: multimodal display of joy from the Belfast Naturalistic Emotional Database (Cowie et al., 2003)
Multimodal Joy annotation • Joy: • arm movements, • head/torso movements towards front and backwards, • tilts, and micro tilts, • movements to the side, • Great arm movements (like playing on a drum) • Facial expressions like smile and raise eyebrow. Humaine video-corpus
Multimodal expressions of emotions • formalization of multimodal expressions of emotions by constraints FML: Emotion label Constraints: - time constraints - ordering - simultaneity - proba of occurrence Emotion Behavior Sets: (set of signals of different modalities) Annotation of corpus variety ofmultimodal expressions of emotions
Multimodal expressions of emotions • literature constraints <multimodal emotion="embarrassment"> <signals> <signal id="1" name="head=head_down_strong" repetitivity="0" min_duration="2" ... <signal id="2" name="head=head_left_strong" repetitivity="0" min_duration="5" ... <signal id="3" name="gaze=look_down" repetitivity="0" min_duration="2" ... <signal id="4" name="gaze=look_right_strong" repetitivity="0" min_duration="1" ... <signal id="5" name="gaze=look_left_strong" repetitivity="0" min_duration="1" ... <signal id="6" name="affect=smile" repetitivity="1" min_duration="2" ... ... </signals> <cons> <con type="minus"> <arg id="6" type="start"/> <arg id="2" type="start"/> <lessthan value="0"/> </con> <con type="minus"> <arg id="7" type="start"/> <arg id="2" type="start"/> <lessthan value="0"/> </con> .... D. Keltner, B. N. Buswell Embarrassment: Its Distinct Form and Appeasement Functions annotation
Multimodal expressions of emotions • implementation of the algorithm <emotion id="e1" type=“joy" start="1.0" end="14" /> BML A set of behaviours FML animation FMLRealizer BMLRealizer
Example Embarrassment Embarrassment Joy
Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming projects <footer>
Audio Cycle • A prototype application for browsing through musical loop libraries. AudioCycle provides the user with a graphical view where the audio extracts are visualized and organized according to their similarity in terms of musical properties, such as timbre, harmony, and rhythm. The user is able to navigate in this visual representation, and listen to individual audio extracts, searching for those of interest. • Richer in features than other similar concepts we have seen.
Technologies & Architecture Audio, Musical Loops 3D Audio Rendering Audio Analysis Visualization 3D Visual Rendering Meta-Data & Features User Input
Extractedfeatures • Timbre: Mel-Frequency Cepstral Coefficients • Harmony and Melody: Chromas (information about the notes played) • Rhythm: periodicity, « Beats per Minute »
Visualization: Clustering Loop Cluster Centroïd Reference Loop
Adaptation to laughter • Firsts tests: Audio Cycle as it is: • some grouping of classes (whisper-like, « retained » laughters, melodious laughters, …) • some grouping of classes • laughing audience
Adaptation to laughter • New feature set (« laughter »): • mean pitch • rate of voiced frames • mean energy • maximum aplitude • duration • Automatic laughter bursts segmentation
Adaptation to laughter • First laughter synthesis using concatenation of bursts
Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming projects <footer>
Future Work • Adapt Audio Cycle to Laughter Laughter Cycle: • improve features • model for concatenation of bursts • how to combine laughters
Future Work • Audovisual laughing machine: eNTERFACE’09 • synchronisation between audio and ECA expressions, using emotional behavior descriptors • automatic answer to an input laughter
Future Work • Numediart Research Program: http://www.numediart.org • eNTERFACE’09: http://www.infomus.org/enterface09/ • CALLAS project: http://www.callas-newmedia.eu/