A Demonstration of Continuous Interaction with Elckerlyc

1. A Demonstration of Continuous Interaction with Elckerlyc Herwin van Welbergen, Dennis Reidsma Job Zwiers

2. Temporal coordination: from turn based interaction to continuous interaction • Current interaction with virtual humans • Using speech, facial expression and gestures • Turn-based, ‘push-to-talk’ interaction paradigm

3. Our goals • Allow continuous interaction • All partners express themselves continuously • In parallel • Needs prediction and anticipation

4. Continuous interaction Temporal coordination in human behavior • Timing of backchannel • Allowing room for (intrusive) backchannel from partner • Overlap during non-intrusive backchannel • … In this talk • Focus on specifying continuous interaction • Using minor timing and parameter value adjustments in ongoing behavior

5. Elckerlyc • Temporal coordination in interaction with virtual humans • Elckerlyc: open source platform for behavior generation designed for such temporally coordinated behavior (… and other things) • Executes behavior specified in the behavior markup language (BML) • Implements SAIBA Framework

6. Why BML/SAIBA? > 10 years of research on ECAs Increasingly sophisticated models Directed to different aspects of behavior Building a state-of-the-art ECA entails re-implementing all these models Allow reuse of models

7. SAIBA framework Three-level setup of many existing ECAs Separation between stages Each stage is a black box Clear-cut definition of interfaces between stages FML BML Intent Planning Behavior Planning Behavior Realization (Smartbody, Greta, ACE, Elckerlyc) Feedback Feedback

8. BML Example

9. BML Example

10. BML Behaviors Gesture Head Gaze Speech Locomotion Posture Facial expression eyes Coordination torso legs

11. Phases and sync-points

12. Synchronizing behaviors

13. Example: gaze shift to moving target • <bml id="bml1"> <speech id="bluespeech" start="gaze1:ready"> <text>I'm staring at the blue cube.</text> </speech> <speech id="greenspeech" start="gaze2:start"> <text>Look! A green sphere.</text> </speech> <speech id="bluespeech2" start="gaze2:end"> <text>Looking at the blue cube again.</text> </speech> <gaze id="gaze1" start="1" ready="2" type="AT" modality="NECK" target="bluebox"/> <gaze id="gaze2" start="4" ready="5" relax="6" end="7" type="AT" modality="NECK" dynamic="true" target="greensphere"/> • </bml>

14. Continuous interaction using SAIBA

15. Continuous Interaction Scenarios • Virtual Trainer

16. Continuous Interaction Scenarios • Virtual Trainer <bml id="bml1"> <bmlt:procanimation id="exercise1" name="squat"/> <constraint id="c1"> <synchronize ref="exercise1:beat1"> <sync ref="excerciseAnticipator:beat1-0.5"/> </synchronize> <synchronize ref="exercise1:beat2"> <sync ref="excerciseAnticipator:beat2-0.5"/> </synchronize> ... </constraint> </bml>

17. Demo

18. Continuous Interaction Scenarios • Taking the turn (as soon as possible) <bml id="bml1"> <speech id="speech1" start="speechStopAnticipator:stop+x"> <text>Bla bla</text> </speech> </bml>

19. Demo • Uses pre-planning • Easily allows incremental updates of speech end prediction

20. Continuous Interaction Scenarios • Keeping the turn

21. Demo

22. Conclusion • Elckerlyc provides mechanisms for the specification and execution of behaviors synchronized to predicted human behaviour • Flexible pre-planning • Incremental update of user behavior prediction

23. Open issues • Specifying parameter changes • <bmlt:setparameter id=“pchange1" start="10" end="speech1:end” target="speech1" parameter="volume" curve="linear" startvalue="25" endvalue="100"/> • Send a parameter specification message • Allow easy experimentation with parameter change • Synchronization to behaviors • Parameter value curve (e.g. linear, ease-in ease-out, …) • Using BML? • Conceptually doesn’t match very well with other behaviors • Should not modify synchronization constraints • Requires specialized mechanisms to refer to behaviors in previous blocks

24. Open Issues • Creating anticipators • Currently we use placeholders • Real-time music tempo tracking and prediction (Pieter Bos et al) • Predicting listener backchannel relevant moments (Louis-Philippe Morency, Iwan de Kok, ...) • Fitness excercise tempo tracking and prediction (Eike Dehling) • Real-time turn-giving prediction (Jonsdottir et al 2008)

25. Enterface 2010 • Backchannel aware guide • Predicts and anticipates completions, acknowledgement and continuers of a user • Response handling strategies • Modulated by politeness • Ignore (and speak louder) • Acknowledge response • Wait for acknowledgement before continuing • Change route-giving plan • Etc • Requires • Adaption of timing of ongoing behavior • Adaptation of parameterization of ongoing behavior • Graceful interruption

26. Thank you for your attention Acknowledgments Stefan Kopp’s group Bart van Straalen Ronald Paul Mark ter Maat Zsofia Ruttkay Greta developers SmartBody developers • More information/demo • http://hmi.ewi.utwente.nl/showcase/Elckerlyc • Contact Dennis, Herwin: {dennisr,welberge}@ewi.utwente.nl

27. END The End.

28. Enterface 2010 • Route recap: dealing with acknowledgements • A1: So, we went to the square with the obelisk… • A2: We crossed the river using the suspension bridge… • A3: Crossed the square with the arch monument… • A4: And took the first street to the right. • Do not deal with it: simply elicit the feedback, pause a bit, then continue speaking, whatever the subject does • Deal with it: elicit feedback, pause a bit, if feedback is there, wait till it is finished before acknowledging (smile, nod?) and continuing • Deal with it, the “continuous interaction way”: elicit feedback, wait, but if feedback comes and it is non-intrusive, immediately acknowledge the feedback (smile, nod?) and continue speaking even while the user is still finishing the feedback • Evaluate responsiveness, attentiveness, politeness, etc

29. Enterface 2010 • Eliciting of completion • A: Do you know the way to the square with the… euhm… [makes thinking gestures]… euhm… that thing [makes iconic gesture for the arch monument]… euhm… [looks at the listener inviting a response]… euhm… how do you call it?... etc. • [correct completion:] • U: arch monument? • A: Yes, the arch monument. • [Incorrect completion:] • U: Obelisk? • A: No, I mean… euhm... the square with the arch monument.

30. Some boring technical facts Written in Java ≈80,000 lines of code (Greta: 45,000; SmartBody: 35,000) including ≈ 30,000 lines code for Collada loading and custom graphics ≈ 200 (J)Unit test cases, ≈ 1100 tests Takes advantage of multiple processors/cores Physical simulation (OdeJava) Rendering (Jogl) Text To Speech (MS Speech API, Mary TTS, Text-based)

31. Licensing Pre-release under the GPL v3 license No source repository yet, contact Herwin for code ‘Official’ versioned release soon Inverse dynamics code is available under MIT license http://www.herwinvanwelbergen.nl/index.php?selected=phd

32. Elckerlyc design concerns • Continuous interaction • Allow fast and flexible planning and re-planning • Allow specification of synchronization to predictions • Flexibility • Easily exchange internal modules, add new modalities, … • Motion generation • Not so much focus on designing (procedural) animation • But on combining motion from all kinds of paradigms • In sequence and parallel • Physical simulation, mocap/keyframe animation, procedural motion, biomechanical models, … • New: mixed dynamics

33. Elckerlyc Architecture • Highlights • Scheduling and playing stages • Scheduler can easily be exchanged • Scheduler resolves timing of a sync point per constraint • Sync points can slightly adjusted

34. Animation plan • Animation plan • Result of planned BML • Specification of motion to be executed by the player • Contains timed motion units • Timing linked to time pegs

35. Organization of motion • Motion Units • (≈LMPs, gestures, controllers, …) • Parameters, linked to BML parameters • Can be executed, typically rotate joints • At canonical time (0..1) • Contain phases, key times can be aligned to time pegs • Key times are assigned canonical time values • Physical, procedural, mocap/keyframe, specialized

36. Procedural motion units • Procedural motion units • Function of time (0..1) and parameter vector • of end effector position and/or joint rotation • Custom functions for perlin noise, splines, … • Example: • <EndEffector local="false" target="r_wrist” • translation="0;(1-t)*starty+t*endy;0.3"/> • Key frames • Mocap • Can be imported from Greta gestures

37. Physical motion units Physical controllers Input: desired state Minimize discrepancy between current and desired state Output: joint torques that guides the VH closer to this state Can cope with external perturbation Video from Abe et al. 2007

38. Physical simulation for gesture • Advantages • Models force transference between body parts • Models momentum effects • Overshoot • Models pendulum like motion easily • Realistic balance • Desired state specification fits some motions well • Loosely hanging arm (desired state: slightly damp arm motion) • Balance (desired state: projected CoM in support polygon) • Spine (desired state: spine pose)

39. Physical simulation for gesture • Disadvantages • Precise timing and limb placement is an open issue • It is unknown if and when a controller achieves a desired limb position • These aspects are crucial for (speech accompanying) gesture

40. Mixed dynamics • Mixed dynamics • Use kinematic motion on body parts that require precise timing/placement • Use physical simulation the remaining part of the body • Calculated the forces the kinematic parts apply on the physical part • Using inverse dynamics • Allow switching between different kinematic/physical representations • Depending on the needs of the moment

41. Comparison with mocap

42. Executing a Greta gesture

43. Executing a SmartBody gesture

44. Physical spine (experimental)

45. Switching

46. Transition motion units • Define a transition animation from physics to kinematics • Or kinematics to kinematics • Flexible start state • Predicted end state • Only define type, joints and start and end time • <bmlt:transition id="trans1" class="HermiteSplinePhysicalTransition" start="armhang:end" end="7"> • <bmlt:parameter name="joints" value="l_shoulder,l_elbow,l_wrist"/> • </bmlt:transition>

47. Prediction • Animation predictor • Can predict a joint configuration at a certain time • Uses a copy of the animation plan with predictable timed motion units • Predictable timed motion units • Deterministically define pose at each time • Procedural motion unit • Determinism during certain motion phases • Stroke of gesture • Gaze at static target

48. Custom kinematic motion units • Adhere to the motion unit interface • Move joints on the basis of time (0..1) and parameter values (can be changing during motion) • Provide canonical key positions for synchronization • Motion should be smooth (C0 continuous) • Can make use of predicted motion • Can use a flexible start position • Currently: • Gaze, pointing • Planned: MURML motion units

49. Animation player • Animation player • Executes an animation plan • Timing is linked to time pegs • Time warping: converts ‘real’ time to canonical time • Combines physical simulation and kinematic motion automatically

50. Scheduling • Scheduling • Determine synchronization constraints between behaviors • Distribute behavior over planners • BML => plans • Assign preliminary times to pegs