1 / 27

Towards a virtual agent using similarity-based laughter production

Towards a virtual agent using similarity-based laughter production. Jérôme Urbain, Stéphane Dupont, Thierry Dutoit TCTS Lab, Faculty of Engineering, Mons Radoslaw Niewiadomski, Catherine Pelachaud CNRS - LTCI UMR 5141, Institut TELECOM - TELECOM ParisTech. Context.

luz
Download Presentation

Towards a virtual agent using similarity-based laughter production

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Towards a virtual agent using similarity-based laughterproduction Jérôme Urbain, Stéphane Dupont, Thierry Dutoit TCTS Lab, Faculty of Engineering, Mons Radoslaw Niewiadomski, Catherine Pelachaud CNRS - LTCI UMR 5141, Institut TELECOM - TELECOM ParisTech

  2. Context • Desire to communicate with machines as we do with humans • Lack of naturalness, expressivity, emotional behaviors • CALLAS European project: integrate emotions in new media • Laughter is a very important signal: • Conveys emotional contents • Contagious • ...

  3. Objectives • Enrich the expressivity of an Embodied Conversational Agent (ECA) by enabling it to laugh: • acoustic laughter production • synchronisation with virtual agent expressions • ability to instantaneously answer to an incoming laughter

  4. Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming developments

  5. Multimodal expressions of emotions • Go beyond « static images »/linear expressions (e.g. beyond basic emotions predictions) • Complete data obtained from theory and literature • Multimodal expressions of emotions • across modalities • many emotions are expressed by sequences (or combination) of multimodal signals rather than monomodal signals (eg static facial expressions) D. Keltner, B. N. Buswell Embarrassment: Its Distinct Form and AppeasementFunctions

  6. Multimodal expressions of emotions • Difficulty: lack of relevant research and video-corpora • new video-corpus of examples of spontaneous multimodal behaviors (based on TV broadcasts) • annotation in Anvil (Kipp 2001) • Specification of Behaviors Sets and Constraints

  7. Corpus of emotional displays • Annotation of audio-visual recordings from reality shows, hidden camera recordings, Belfast Naturalistic database, EmoTV corpus. • The observed people are non actors in emotional situations: natural and not stereotyped multimodal behaviour is displayed • 20 video clips (3 to 14 seconds each): • relief (2), tension (6), joy (2), sadness (2), anger (3), despair (1), fear (4)

  8. Multimodal annotation scheme • Annotation on Anvil v4.7.6 (Kipp, 2001) with 5 tracks: • Emotion (inferred from the situation) • Facial expression (FACS coding) • Head movement • Gaze movement • Gesture

  9. Video

  10. Multimodal Joy annotation Video annotations with the Anvil software: multimodal display of joy from the Belfast Naturalistic Emotional Database (Cowie et al., 2003)

  11. Multimodal Joy annotation • Joy: • arm movements, • head/torso movements towards front and backwards, • tilts, and micro tilts, • movements to the side, • Great arm movements (like playing on a drum) • Facial expressions like smile and raise eyebrow. Humaine video-corpus

  12. Multimodal expressions of emotions • formalization of multimodal expressions of emotions by constraints FML: Emotion label Constraints: - time constraints - ordering - simultaneity - proba of occurrence Emotion Behavior Sets: (set of signals of different modalities) Annotation of corpus variety ofmultimodal expressions of emotions

  13. Multimodal expressions of emotions • literature constraints <multimodal emotion="embarrassment"> <signals> <signal id="1" name="head=head_down_strong" repetitivity="0" min_duration="2" ... <signal id="2" name="head=head_left_strong" repetitivity="0" min_duration="5" ... <signal id="3" name="gaze=look_down" repetitivity="0" min_duration="2" ... <signal id="4" name="gaze=look_right_strong" repetitivity="0" min_duration="1" ... <signal id="5" name="gaze=look_left_strong" repetitivity="0" min_duration="1" ... <signal id="6" name="affect=smile" repetitivity="1" min_duration="2" ... ... </signals> <cons> <con type="minus"> <arg id="6" type="start"/> <arg id="2" type="start"/> <lessthan value="0"/> </con> <con type="minus"> <arg id="7" type="start"/> <arg id="2" type="start"/> <lessthan value="0"/> </con> .... D. Keltner, B. N. Buswell Embarrassment: Its Distinct Form and Appeasement Functions annotation

  14. Multimodal expressions of emotions • implementation of the algorithm <emotion id="e1" type=“joy" start="1.0" end="14" /> BML A set of behaviours FML animation FMLRealizer BMLRealizer

  15. Example Embarrassment Embarrassment Joy

  16. Video

  17. Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming developments

  18. Laughter production • Laughter synthesis is a very hard task • Our approach: • Enable the user to efficently browse through a large laughter database to find a desired utterance - Create new laughters by combining units from the database

  19. Numediart: Audio Cycle • A prototype application for browsing through musical loop libraries. AudioCycle provides the user with a graphical view where the audio extracts are visualized and organized according to their similarity in terms of musical properties, such as timbre, harmony, and rhythm. The user is able to navigate in this visual representation, and listen to individual audio extracts, searching for those of interest. • Richer in features than other similar concepts we have seen.

  20. Adaptation to laughter • Firsts tests: Audio Cycle as it is: • some grouping of classes (whisper-like, « retained » laughters, melodious laughters, …) • some grouping of classes • laughing audience

  21. Adaptation to laughter • New feature set (« laughter »): • mean pitch • ratio of voiced frames • mean energy • maximum amplitude • duration • to be continued…

  22. Laughter synthesis • Automatic laughter bursts segmentation

  23. Laughter synthesis • Concatenation of bursts Real Synthesizeded • Most similar burst in database (4400 bursts)

  24. Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming developments

  25. Upcoming developments • Acoustic side: Adapt Audio Cycle to Laughter Numediart project « Laughter Cycle » (april – june): • improve features • model for concatenation of bursts

  26. Future Work • Audovisual laughing machine: eNTERFACE’09 project « AVLaughterCycle » (July – August, Genova, Italy): • synchronisation between audio and ECA expressions, using emotional behavior descriptors • automatic answer to an input laughter

  27. Future Work • Numediart Research Program: http://www.numediart.org • eNTERFACE’09: http://www.infomus.org/enterface09/ • CALLAS project: http://www.callas-newmedia.eu/

More Related