1 / 97

A Demonstration of Continuous Interaction with Elckerlyc

A Demonstration of Continuous Interaction with Elckerlyc. Herwin van Welbergen, Dennis Reidsma Job Zwiers. Temporal coordination: from turn based interaction to continuous interaction. Current interaction with virtual humans Using speech, facial expression and gestures

talia
Download Presentation

A Demonstration of Continuous Interaction with Elckerlyc

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Demonstration of Continuous Interaction with Elckerlyc Herwin van Welbergen, Dennis Reidsma Job Zwiers

  2. Temporal coordination: from turn based interaction to continuous interaction • Current interaction with virtual humans • Using speech, facial expression and gestures • Turn-based, ‘push-to-talk’ interaction paradigm

  3. Our goals • Allow continuous interaction • All partners express themselves continuously • In parallel • Needs prediction and anticipation

  4. Continuous interaction Temporal coordination in human behavior • Timing of backchannel • Allowing room for (intrusive) backchannel from partner • Overlap during non-intrusive backchannel • … In this talk • Focus on specifying continuous interaction • Using minor timing and parameter value adjustments in ongoing behavior

  5. Elckerlyc • Temporal coordination in interaction with virtual humans • Elckerlyc: open source platform for behavior generation designed for such temporally coordinated behavior (… and other things) • Executes behavior specified in the behavior markup language (BML) • Implements SAIBA Framework

  6. Why BML/SAIBA? > 10 years of research on ECAs Increasingly sophisticated models Directed to different aspects of behavior Building a state-of-the-art ECA entails re-implementing all these models Allow reuse of models

  7. SAIBA framework Three-level setup of many existing ECAs Separation between stages Each stage is a black box Clear-cut definition of interfaces between stages FML BML Intent Planning Behavior Planning Behavior Realization (Smartbody, Greta, ACE, Elckerlyc) Feedback Feedback

  8. BML Example

  9. BML Example

  10. BML Behaviors Gesture Head Gaze Speech Locomotion Posture Facial expression eyes Coordination torso legs

  11. Phases and sync-points

  12. Synchronizing behaviors

  13. Example: gaze shift to moving target • <bml id="bml1"> <speech id="bluespeech" start="gaze1:ready"> <text>I'm staring at the blue cube.</text> </speech> <speech id="greenspeech" start="gaze2:start"> <text>Look! A green sphere.</text> </speech> <speech id="bluespeech2" start="gaze2:end"> <text>Looking at the blue cube again.</text> </speech> <gaze id="gaze1" start="1" ready="2" type="AT" modality="NECK" target="bluebox"/> <gaze id="gaze2" start="4" ready="5" relax="6" end="7" type="AT" modality="NECK" dynamic="true" target="greensphere"/> • </bml>

  14. Continuous interaction using SAIBA

  15. Continuous Interaction Scenarios • Virtual Trainer

  16. Continuous Interaction Scenarios • Virtual Trainer <bml id="bml1"> <bmlt:procanimation id="exercise1" name="squat"/> <constraint id="c1"> <synchronize ref="exercise1:beat1"> <sync ref="excerciseAnticipator:beat1-0.5"/> </synchronize> <synchronize ref="exercise1:beat2"> <sync ref="excerciseAnticipator:beat2-0.5"/> </synchronize> ... </constraint> </bml>

  17. Demo

  18. Continuous Interaction Scenarios • Taking the turn (as soon as possible) <bml id="bml1"> <speech id="speech1" start="speechStopAnticipator:stop+x"> <text>Bla bla</text> </speech> </bml>

  19. Demo • Uses pre-planning • Easily allows incremental updates of speech end prediction

  20. Continuous Interaction Scenarios • Keeping the turn

  21. Demo

  22. Conclusion • Elckerlyc provides mechanisms for the specification and execution of behaviors synchronized to predicted human behaviour • Flexible pre-planning • Incremental update of user behavior prediction

  23. Open issues • Specifying parameter changes • <bmlt:setparameter id=“pchange1" start="10" end="speech1:end” target="speech1" parameter="volume" curve="linear" startvalue="25" endvalue="100"/> • Send a parameter specification message • Allow easy experimentation with parameter change • Synchronization to behaviors • Parameter value curve (e.g. linear, ease-in ease-out, …) • Using BML? • Conceptually doesn’t match very well with other behaviors • Should not modify synchronization constraints • Requires specialized mechanisms to refer to behaviors in previous blocks

  24. Open Issues • Creating anticipators • Currently we use placeholders • Real-time music tempo tracking and prediction (Pieter Bos et al) • Predicting listener backchannel relevant moments (Louis-Philippe Morency, Iwan de Kok, ...) • Fitness excercise tempo tracking and prediction (Eike Dehling) • Real-time turn-giving prediction (Jonsdottir et al 2008)

  25. Enterface 2010 • Backchannel aware guide • Predicts and anticipates completions, acknowledgement and continuers of a user • Response handling strategies • Modulated by politeness • Ignore (and speak louder) • Acknowledge response • Wait for acknowledgement before continuing • Change route-giving plan • Etc • Requires • Adaption of timing of ongoing behavior • Adaptation of parameterization of ongoing behavior • Graceful interruption

  26. Thank you for your attention Acknowledgments Stefan Kopp’s group Bart van Straalen Ronald Paul Mark ter Maat Zsofia Ruttkay Greta developers SmartBody developers • More information/demo • http://hmi.ewi.utwente.nl/showcase/Elckerlyc • Contact Dennis, Herwin: {dennisr,welberge}@ewi.utwente.nl

  27. END The End.

  28. Enterface 2010 • Route recap: dealing with acknowledgements • A1: So, we went to the square with the obelisk… • A2: We crossed the river using the suspension bridge… • A3: Crossed the square with the arch monument… • A4: And took the first street to the right. • Do not deal with it: simply elicit the feedback, pause a bit, then continue speaking, whatever the subject does • Deal with it: elicit feedback, pause a bit, if feedback is there, wait till it is finished before acknowledging (smile, nod?) and continuing • Deal with it, the “continuous interaction way”: elicit feedback, wait, but if feedback comes and it is non-intrusive, immediately acknowledge the feedback (smile, nod?) and continue speaking even while the user is still finishing the feedback • Evaluate responsiveness, attentiveness, politeness, etc

  29. Enterface 2010 • Eliciting of completion • A: Do you know the way to the square with the… euhm… [makes thinking gestures]… euhm… that thing [makes iconic gesture for the arch monument]… euhm… [looks at the listener inviting a response]… euhm… how do you call it?... etc. • [correct completion:] • U: arch monument? • A: Yes, the arch monument. • [Incorrect completion:] • U: Obelisk? • A: No, I mean… euhm... the square with the arch monument.

  30. Some boring technical facts Written in Java ≈80,000 lines of code (Greta: 45,000; SmartBody: 35,000) including ≈ 30,000 lines code for Collada loading and custom graphics ≈ 200 (J)Unit test cases, ≈ 1100 tests Takes advantage of multiple processors/cores Physical simulation (OdeJava) Rendering (Jogl) Text To Speech (MS Speech API, Mary TTS, Text-based)

  31. Licensing Pre-release under the GPL v3 license No source repository yet, contact Herwin for code ‘Official’ versioned release soon Inverse dynamics code is available under MIT license http://www.herwinvanwelbergen.nl/index.php?selected=phd

  32. Elckerlyc design concerns • Continuous interaction • Allow fast and flexible planning and re-planning • Allow specification of synchronization to predictions • Flexibility • Easily exchange internal modules, add new modalities, … • Motion generation • Not so much focus on designing (procedural) animation • But on combining motion from all kinds of paradigms • In sequence and parallel • Physical simulation, mocap/keyframe animation, procedural motion, biomechanical models, … • New: mixed dynamics

  33. Elckerlyc Architecture • Highlights • Scheduling and playing stages • Scheduler can easily be exchanged • Scheduler resolves timing of a sync point per constraint • Sync points can slightly adjusted

  34. Animation plan • Animation plan • Result of planned BML • Specification of motion to be executed by the player • Contains timed motion units • Timing linked to time pegs

  35. Organization of motion • Motion Units • (≈LMPs, gestures, controllers, …) • Parameters, linked to BML parameters • Can be executed, typically rotate joints • At canonical time (0..1) • Contain phases, key times can be aligned to time pegs • Key times are assigned canonical time values • Physical, procedural, mocap/keyframe, specialized

  36. Procedural motion units • Procedural motion units • Function of time (0..1) and parameter vector • of end effector position and/or joint rotation • Custom functions for perlin noise, splines, … • Example: • <EndEffector local="false" target="r_wrist” • translation="0;(1-t)*starty+t*endy;0.3"/> • Key frames • Mocap • Can be imported from Greta gestures

  37. Physical motion units Physical controllers Input: desired state Minimize discrepancy between current and desired state Output: joint torques that guides the VH closer to this state Can cope with external perturbation Video from Abe et al. 2007

  38. Physical simulation for gesture • Advantages • Models force transference between body parts • Models momentum effects • Overshoot • Models pendulum like motion easily • Realistic balance • Desired state specification fits some motions well • Loosely hanging arm (desired state: slightly damp arm motion) • Balance (desired state: projected CoM in support polygon) • Spine (desired state: spine pose)

  39. Physical simulation for gesture • Disadvantages • Precise timing and limb placement is an open issue • It is unknown if and when a controller achieves a desired limb position • These aspects are crucial for (speech accompanying) gesture

  40. Mixed dynamics • Mixed dynamics • Use kinematic motion on body parts that require precise timing/placement • Use physical simulation the remaining part of the body • Calculated the forces the kinematic parts apply on the physical part • Using inverse dynamics • Allow switching between different kinematic/physical representations • Depending on the needs of the moment

  41. Comparison with mocap

  42. Executing a Greta gesture

  43. Executing a SmartBody gesture

  44. Physical spine (experimental)

  45. Switching

  46. Transition motion units • Define a transition animation from physics to kinematics • Or kinematics to kinematics • Flexible start state • Predicted end state • Only define type, joints and start and end time • <bmlt:transition id="trans1" class="HermiteSplinePhysicalTransition" start="armhang:end" end="7"> • <bmlt:parameter name="joints" value="l_shoulder,l_elbow,l_wrist"/> • </bmlt:transition>

  47. Prediction • Animation predictor • Can predict a joint configuration at a certain time • Uses a copy of the animation plan with predictable timed motion units • Predictable timed motion units • Deterministically define pose at each time • Procedural motion unit • Determinism during certain motion phases • Stroke of gesture • Gaze at static target

  48. Custom kinematic motion units • Adhere to the motion unit interface • Move joints on the basis of time (0..1) and parameter values (can be changing during motion) • Provide canonical key positions for synchronization • Motion should be smooth (C0 continuous) • Can make use of predicted motion • Can use a flexible start position • Currently: • Gaze, pointing • Planned: MURML motion units

  49. Animation player • Animation player • Executes an animation plan • Timing is linked to time pegs • Time warping: converts ‘real’ time to canonical time • Combines physical simulation and kinematic motion automatically

  50. Scheduling • Scheduling • Determine synchronization constraints between behaviors • Distribute behavior over planners • BML => plans • Assign preliminary times to pegs

More Related