An Incremental Multimodal BML Realizer for Behavior Co-Articulation and Coordination

1. An Incremental Multimodal BML Realizer for Behavior Co-Articulation and Coordination Herwin van Welbergen Dennis Reidsma Stefan Kopp

2. Goal: Fluent interaction with virtual humans (and robots) • Beyond turn taking interaction • Continuous perception and behavior generation • Interpersonal coordination • Tight coordination of the (ongoing) behavior of the agent with other (virtual) humans

3. Fluent behavior: some examples • Acknowledge reception, understanding, attitude while listening • Respond to turn taking signals • Graceful interruption • Elicit backchannel feedback (and wait for it) • Establish joint attention • Exercise together with the interlocutor • Synchronized movement • Construct utterances incrementally, on the fly • No long upfront planning

4. Gesture co-articulation • Between the retraction of one gesture and the preparation of the next • Retraction of the first gesture is skipped • Co-articulation (or lack thereof) can have a communicative function (Kendon1980) • E.g. marking information boundaries • Has to be established on the fly

5. Behavior realizer requirements • Allow last minute changes to ongoing behavior • Top down (through BML) • Bottom up emergent behavior • But don’t break BML constraints • Real-time, incremental, fluently connect BML increments • Under-specification handling • The realizer should be smart enough to figure out unconstrained timing, generate co-articulation, .. • This should happen ‘on the fly’ • As late as possible

6. BML • A BML block specifies some constraints in shape and timing on its behaviors • BML blocks are generally underspecified • The BML Realizer has certain realization freedom • Used to achieve natural looking motor behavior

7. BML

8. Example

9. BML • Offers specification for multimodal synchronization, but: • No specification mechanisms for inter-personal synchronization • Limited specification capabilities for incrementality • No specification for co-articulation

10. BML Realizers • SmartBody (Thiebeax et al 2008) • EMBR (Heloir and Kipp 2010) • Greta (Mancini et al 2008) • RealActor (Čerecović and Pandžić 2010) • Special focus on incrementality/interactional coordination • Elckerlyc (van Welbergen et al. 2010) • ACE (Kopp and Wachsmut 2004)

11. Realizing BML the traditional way

12. Elckerlyc • Flexible behavior plan representation (Reidsma et al. 2011) • retains BML constraints • Modified in top-down fashion • Specification mechanisms for interactional coordination

13. ACE • Incremental behavior planning • Specification mechanism for chunk composition • Emergent gesture co-articulation through bottom-up plan modifications

15. Results: gesture co-articulation • Co-articulation can have a communicative function • The Behavior Planner should be able to specify whether or not co-articulation may occur

16. Specifying composition <bml id="bml2" composition="append-after(bml1)"> <speech id="speech1"> <text> At <sync id="s1"/> 6 pm you have another appointment. </text> </speech> <gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/> </bml> <bml id="bml2" composition="chunk-after(bml1)"> <speech id="speech1"> <text> At <sync id="s1"/>6 pm you have another appointment. </text> </speech> <gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/> </bml>

17. Implementing gesture co-articulation • Start a new BML block when a previous block is ‘retracting’ • All behaviors have a retraction phase • A BML block is ‘retracting’ • If all its behaviors are either finished or retracting • If needed, fluently overtake the retraction phase of a gesture in the previous block

18. No co-articulation

19. Co-articulation

20. Initial plan

21. Top down replanning step

22. Bottom-up adaptation step

23. Results: gracious interruption • Gracious interruption depends on • Behavior state (e.g. not started, retracting) • Resting posture/state • Current position of e.g. the hand • This information is available in the AnimationEngine • Smart retraction using the AnimationEngine • Replace a interrupted gesture by a new one • Co-articulation

24. Interruption

25. Usage scenario: interactional coordination • Top down specification that a pointing gesture/gaze should remain on target until an interlocutor event • Joint attention • Feedback elicitation • Bottom up emergent insertion/deletion of ‘hold’ motions

26. Summary • AsapRealizercombines state of the art features of ACE and Elckerlyc • Fusion provides new capabilities: • Interactional coordination + automatic hold phase construction • Top down interrupt with bottom-up automatic graciousness + co-articulation • Top down+bottom up adaptation capabilities are essential for fluent interaction

27. Thanks for your attention Questions? http://asap-project.org

28. SAIBA

29. Behavior/block state machine

30. Usage scenario: robotics • Internal timing predictions of one or more modalities might be unpredictable • E.g. for robot behavior • Bottom up adjustments can be used to • Adjust the behavior itself (e.g. speed up to meet a time constraint) • Move the time constraints that cannot be met, automatically adjusting the timing on other modalities linked to these constraints

31. Results: Interruption Interruption in Elckerlyc (a) Interrupting all behavior in bml1. <bml id="yieldturn"> <bmlt:interrupt id="i1" target="bml1"/> </bml> (b) Interrupt all behavior in bml1 excluding gesture1. <bml id="yieldturn"> <bmlt:interrupt id="i1" target="bml1" exclude="gesture1"/> </bml> (c) Interrupt all behavior in bml1. Insert a behavior (relaxArm) that gracefully moves the gesturing arm back to its rest position. <bml id="yieldturn"> <bmlt:interrupt id="i1" target="bml1"/> <bmlt:controller id="relaxArm" class="CompoundController" name="leftarmhang"/> </bml>

32. AsapAnimationEngine • Gesture is specified in a TimedMotionUnit • TimedMotionUnit specification: • State • Priority • Set of controlled joints • TimedMotionUnit execution: • Set a joint rotation on the skeleton • Add a physical controller • Set a RestingTimedMotionUnit

33. AsapAnimationEngine • RestingTimedMotionUnit • Manages motion related to the resting state of the virtual human • Possible implementations: • Static posture (as in ACE) • Lower body physical balance controller • … • Is always executed with the lowest priority • Can create transition TMUs to the resting state • Slerp • Drop the arm controller • …

34. Gesture Coarticulation in the AsapAnimationEngine • The priority of a TMU drops in its SUBSIDING state • AsapAnimationEngine executes TMUs in order of priority • If a TMU needs to execute motion on joints that are already in use by a higher priority TMU, drop the TMU • Automatically create a cleanup TMU that moves other joints back to the resting state

35. TimedMotionUnit bottom up adaptations • Gesture changes during execution: • start position of the hand • hand position at the start and end of the stroke • posture state at the end of the gesture • Adaptive timing and shape for gesture preparation and retraction phase • Using Fitts’ law • Only update if no other constraints act on e.g. both start and stroke-start timing

36. Elckerlyc • BML specification mechanisms for: • Interpersonal coordination • Gracious interruption [but verbose and top-down only] • Flexible plan representation • Allows adaptation of the ongoing plan • While maintaining BML constraints • So far such plan modifications were mainly managed top-down (e.g. by new BML blocks) • So far no combined shape and timing adaption mechanisms

37. ACE • MURML specification for multimodal coordination • Incrementality • Gesture co-articulation • Bottom up adaptation of ongoing behavior

38. Usage scenario: parameter change • Top-down specification of a parameter change • Increase the amplitude of an ongoing gesture • Bottom up adaptations: • Update retraction and/or preparation timing and shape accordingly • Might adjust unconstrained shape and timing parameters to retain biological plausibility

39. Gesture-speech alignment

40. AsapRealizer • Real-time multimodal behavior generation for robots and virtual humans • Incremental construction of behavior • Fluent connection of increments (e.g. behavior co-articulation) • Allowing last minute changes of ongoing behavior • Multimodal synchronization • Builds upon ACE, Elckerlyc • BML • BML 1.0 compliant (tested by RealizerTester)