1 / 99

zsofi@cs.utwente.nl

Interacting with Virtual Humans Zsófia Ruttkay Associate Professor Human Media Interaction, Dept. of CS, University of Twente, NL http://hmi.ewi.utwente.nl/~zsofi/ SSIP 2009 Debrecen. zsofi@cs.utwente.nl. Overview of the talk. Intro About Virtual Humans.

akiko
Download Presentation

zsofi@cs.utwente.nl

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Interacting with Virtual HumansZsófia RuttkayAssociate ProfessorHuman Media Interaction, Dept. of CS, University of Twente, NLhttp://hmi.ewi.utwente.nl/~zsofi/SSIP 2009 Debrecen zsofi@cs.utwente.nl

  2. Overview of the talk Intro About Virtual Humans FA Analysis of facial expressions of humans and cartoons Non-photorealistic facial animation - CharToon Emotion Disc Evaluation of personality cues Full body Control with markup – GESTYLE Current work - conductor Challenges for IP

  3. About Virtual Humans I • More or less human-like synthetic characters to interact with users • In roles as:- information provider- tutor- sales assistant- ‘translator’ (sign language)- avatar in telepresence and VR- entertainment, games- also medium for experiments • Research since 15 years, recently own events:IVA, Gesture WS, AAMAS workshop series • Book by J. Cassell et al. (2000, MIT Press)

  4. Examples Rea from MIT, Cassell et al. Steve from USC J. Rickel, L. Johnson et al.

  5. Examples OLGA – KTH

  6. Examples Max – Univ. of Bielefeld

  7. Examples Mission Exercise, USC ISI + ICT

  8. Examples Carmen’s Bright Ideas from USC ICT, Marsella et al.

  9. Examples USC – ISI, Adele, Shaw et al.

  10. Examples Let’s Face It! –M. Bartlett et al. Univ. San Diego, 2008.

  11. Virtual conductor Virtual Conductor HMI, Univ. of Twente, The Netherlands

  12. Virtual scientific institute Meta Institute for Computational Astrophysics Piet Hut http://www.ugotrade.com/2008/07/19/astrophysics-in-virtual-worlds-implementing-n-body-simulations-in-opensim/

  13. Robots Hiroshi Ishiguro ATR Intelligent Robotics and Communications Laboratories http://www.irc.atr.jp/Geminoid/

  14. Geminoids Geminoid by I. Ishiguro at ATR, Japan http://www.irc.atr.jp/Geminoid/

  15. About VHs - general architecture nl processing (meta) speech understanding face interpretation gesture recognition environment monitoring (object, resources, situation) perception (input) decision (what to do) action (output) multimodal output related to environment user model affect model discourse model domain knowledge goals … real-time suspension of disbelief ‘good’ for …

  16. About VHs – Design parameters • Embodiment: head – torzo - full • ‘Look’: gender, age, profession, personality • Design: realistic/not, 2d/3d • Communication channels & directionsspeech, facial expressions, gestures, body language • What is to be expressed: emotional, cognitive, phys. state references to changing surrounding (incl. user) • Role: monologue – dialogue, relation to user: gender, cluture, socal status

  17. About VHs – Hierarchy of behavior independent behavior goal-oriented behavior personality, emotions high-level control bidirectional mm comm. multimodal output single modal output rendering, TTS, ...

  18. About VHs - development cycle realtime adoptable mind body computational model implementation of VA evaluation of VA analysis of H-H interaction What? How? non-realism: animation, art microlevel in application

  19. About VHs – Some issues • Mind and body issue • Why to use VHs?- ease of use: broader user group, less load- more effect- more fun • Moral and legal issues: clones, manipulation • To mimic humans, or less is more? • Interdisciplinary research

  20. About VHs – Do people take them? • Illusion – do people believe, react to as if theywere real? • YES Clifford Nass: CASA paradigm Computers Are Social Actors • Series of experiments on effect of micro and macro design • User’s characteristics matter

  21. Overview of the talk Intro About Virtual Humans FA Analysis of facial expressions of humans and cartoons Non-photorealistic facial animation - CharToon Emotion Disc Evaluation of personality cues Full body Control with markup – GESTYLE Current work - conductor Challenges for IP

  22. Motivations for FA research • Need for: expressive, animated synthetic faces • Functions of facial (and other) nonverbal signals: - emotions - cognitive state- speech punctuation- dialog control- additional/redundant info (size, location, …)- personality and culture of speaker • All contribute to: comprehension, memory, effectivity … motivation, joy, engagement

  23. Questions concerning FA • Which expressions to consider? • What are those expressions like? - static and dynamical characteristics - generic and individual characteristics • What facial model to use? - (non-)photorealism - beyond realism/simplification of motions? • Tool to make synthetic face models and animate them

  24. Facial anatomy • about 44 pairs of muscles • bundle • sheet • circular • physiological data missing • coarticulation • many parameters • nonlinear effects

  25. Psychology of facial expressions • “meaningful” – physically possible - impossible • biological need/content/emotional state/discourse regulator/… • muscle activity – emotion sensing in brain • difference between real and ‘on demand’ expressions data collection must be done with care:- spontaneous/stimulated expression - natural setting (e.g. in news)- real people vs. actor?

  26. History …. • 1882: Darwin: The Expression of the Emotions in Man and Animals • 1862: Duchenne: photos of expressions induced by electric signals

  27. History: coding of facial expressions • 1970 - ...: Paul Ekman (U. of California) • 1989: 6 general emotional expressions:happiness, surprise, sadness, fear, disgust, anger • FACS [1978] coding system – still preferred inUS • discrete • visual effectof (groups of) muscles

  28. Coding of facial expressions: MPEG4 • 1999: ISO standard • widely used • 68normalised parameters (FAPs) • continuous • black points x, y, (z) displ • min, max, neutral value

  29. Emberi arckifejezések leirása: MPEG4 • # FAP name FAP description Unit • 31 raise_l_i_eyebrow Vertical displacement of left inner eyebrow ENS • 32 raise_r_i_eyebrow Vertical displacement of right inner eyebrow ENS • 33 raise_l_m_eyebrowVertical displacement of left middle eyebrow ENS • 34 raise_r_m_eyebrow Vertical displacement of right middle eyebrow ENS • 35 raise_l_o_eyebrow Vertical displacement of left outer eyebrow ENS • 36 raise_r_o_eyebrow Vertical displacement of right outer eyebrow ENS • 37 squeeze_l_eyebrow Horizontal displacement of left eyebrow ES

  30. Analysis of facial expressions – own investigation 2002 On demand expressions of untrained subjects Track 15 FAPs marked on face

  31. The E-Space: PCA

  32. The E-Space: 2 components’ graph (tracked) Hor: eyebrows Vert: mouth corners

  33. The E-Space: CharToon data

  34. The E-Space: CharToon data • Experienced animator • 59 expression stills: • Anger: Be_Careful_What_You_Say • Smile: Absolute_Joy, Adoration, Smile • Other: Doubt_suspicion

  35. The E-Space: Graph (cartoon) • Hor: eyebrows • Vert: stretch mouth • 3rd: smile/squeeze (all a-symmetric)

  36. The E-Space: Conclusion (stills) • Tracker FAPs not always sufficient to distinguish (negative) emotions • More symmetry in tracked expressions • Animator uses more of the expression space • smiling shape of the mouth and raising of the eyebrows are most significant

  37. Time curves of FAPs

  38. Actuation of muscles during smile and anger (I. Essa, PhD at MIT, 1995)

  39. The E-Space: Time curves

  40. Co-actuation of muscles during smile (I. Essa, PhD at MIT, 1995)

  41. Facial expression analysis - dynamism • Time IS characteristic and important • Trapezoid activation function, variants (camel, …) • Still open questions: shapes, constraints, … • No data(base) published (MS either)

  42. STAR on facial expression tracking Near real-time tracking of faps Other than basic 6 expressions Mixture, masking, cultural variants Dynamism of articulation Spontaneous expressions • Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek,C., Fasel, I., Movellan, J.R. (2006). Automatic Recognition of Facial Actions in Spontaneous Expressions. Journal of Multimedia 1(6) p. 22-35. • Cohn, J. F. & Ekman, P. Methods for measuring facial actions. In J.A. Harrigan, R. Rosenthal, & K. Scherer (Eds.), Handbook of nonverbal behavior research methods in the affective sciences. NY: Oxford. • Tian, Y.L., Kanade, T., & Cohn, J.F. Facial expression analysis. In S.Z. Li & A.K. Jain (Eds.), Handbook of face recognition. NY: Springer. • KTH, Sweeden – cheap and light hw for facial tracking

  43. Analysis of Mona Lisa result of computer facial expression analysis UvA + Univ. of Illionis, Dec. 2005 smile: 83% disgust: 9% fear: 6% anger: 2% What do the numbers mean? How were they obtained? Would it mean the same in the 1500s?

  44. Overview of the talk Intro About Virtual Humans FA Analysis of facial expressions of humans and cartoons Non-photorealistic facial animation - CharToon Emotion Disc Evaluation of personality cues Full body Control with markup – GESTYLE Current work - conductor Challenges for IP

  45. Facial animation – model based Basic principle: • Deformation principles – 2d skeleton, 3d real/virtual muscle, … • Key positions: values of controllers • Inbetweens by interpolation Issues: • Reusable, parametrised repertoire • Blending • Superposition (talking while smiling) • Visual speech - coarticulation

  46. Facial animation • how to interpolate: linear, C1, C2 • how to choose ti (times for keyframes) to t1 t2 t3 t4 t5

  47. 3D model

  48. 3d facial modelling (CWI)

  49. 3d facial animation

More Related