1 / 28

Let’s shake hands! On the coordination of gestures of humanoids

This study explores the coordination of humanoid gestures, focusing on aligning them with external signals and other modalities. It discusses synchronization phenomena and proposes a synchronization language for real-time animation. Examples include gesture-speech coordination and virtual dancer movements to music. The research aims to define and adapt gestures for effective coordination while addressing open issues and outlining future directions in the field.

vbarry
Download Presentation

Let’s shake hands! On the coordination of gestures of humanoids

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Let’s shake hands!On the coordination of gestures of humanoids Zsofi Ruttkay Herwin van Welbergen Balázs Varga

  2. Our goals • Coordinating gesture to external signals • Coordinating gesture to other modalities • Comparison of synch phenomena of rhythmic motion and speech-accompanying gesture • Define a synchronization language • Create an adaptive real-time animation engine

  3. Content • Examples of coordination • The multimodal coordination problem • Existing solutions • Our solution • Coordinating gesture • To what? • How to adapt gesture for coordination? • How to specify coordination? • Conclusions • Open issues • Questions

  4. Coordination example:Gesture-speech coordination • Gestures and speech come from a single process of utterance formation (McNeill) => Gesture timing is not a slave of speech timing • Time alignment is achieved while we speak • The stroke of gesture precedes or ends at the phonological peak syllable of the speech • Often we need to adjust the timing of gestures or speech to make the alignment fit • Gestures can be speeded up or slowed down • Gestures can be ‘paused’ using hold phases • Speech can be stopped to finish complex gestures • Connection words (uh…) can be used to slow down speech • Etc

  5. Coordination example:The virtual dancer: moving to the music • ‘Beat moments’ in the animation should be aligned to beats in the music • Annotate all beat moments in the animation • Predict the beats in the music • Locally speed up or slow down the animation to fit to the music • There is a maximum stretch or skew in the dance motion

  6. Coordination example:The virtual trainer: tutoring exercises on music • Exercises are executed using several modalities • Body movement • Speech • Music/metronome • Sound (clap, foot tap) • Challenges • Coordination • Monitoring user => real time (re)planning • Exaggeration to point out details • Speed up / slow down • Feedback/correction • …

  7. Coordination example:Handshake: coordination between 2 humans • Handshake is used for greeting, agreeing and accepting • Complex coordination between two persons • Guided by • Social protocols • Haptic feedback • Visual feedback

  8. Generalizing: the multimodal coordination problem • ‘Behaviors’ on different modalities (speech, gesture, dance motion, music) • Synchronization between behaviors at key time moments • The timing of flexible behaviors can be adapted to achieve synchronization

  9. Coordination: related work • Classic approach in speech/gesture coordination: • Speech leads, gesture follows • MURML (Kopp et al.) • No leading modality • Planning in sequential chunks containing one piece of speech and one aligned gesture • Co-articulation at the border of chunks • BML (Kopp, Krenn, Marsella, Marshall, Pelachaud, Pirker, Thórisson, Vilhjalmsson) • No leading modality • Synchronized alignment points in behavior phases • For now, aimed mainly at speech/gesture synchronization • In development

  10. Coordination: our previous work • Virtual Dancer • Synchronization between music (beats) and dance animation • Linear time stretching/skewing • Virtual Presenter • Synchronization between speech, gesture, posture and sheet display • Leading modality can change over time • GESTYLE markup language with par/seq and wait constructs to define synchronization

  11. Our multimodal synchronization model • No leading modality, just align key moments • Every phase of a behavior has a preferred length • Stretching/skewing/skipping if necessary

  12. Coordination of hand gestures to external signals • What do we want to coordinate with? • How do synchronization constraints effect movement? • How to stretch/skew? • How can we define synchronization? • using BML scripts?

  13. Ontology of coordination signals

  14. Clapping experiment • Clapping and counting • How is the synchrony between clap and count? • How do the movement characteristics of clapping change with tempo? • Time distribution • Amplitude • Left/right hand symmetry

  15. Clapping experiment: setup • Mocap analysis of two subjects • Instructions: • Clap and count from 21 to 31 • Clap and count to the metronome

  16. Clapping experiment: results • The phonological synchrony rule was valid for counting while clapping: • Clap before phonological peak of the count • The clapping was speeded up by decreasing the path distance of the hand • A pre-stroke hold can be used to slow down • For our right-handed subjects, the right hand was moving ahead in phase compared to the left • The standard deviation of the relative phase between the left and the right hand increased with the clapping frequency

  17. Hand shake experiment • Which movement phases can be identified? • How are they coordinated? • What gaze patterns can be seen? • What movement characteristics can be identified in the different phases? • Timing, duration • Form • How is above effected by • Refusal or avoidance to shake hands • Social relations between participants

  18. Hand shake experiment: setup • Motion capture of two subjects (P1, P2) shaking hands • Annotation of gaze patterns • Variations • Basic • Triggered • P2 initiates • P2 tries to avoid shaking hands • P2 rejects

  19. Modeling coordination in BML • BML is an XML language defining multimodal synchronization • BML events can be used to synchronize with other BML scripts/world events • The BMLT observer is introduced for synchronization with (repeated) outside world events

  20. Coordination with events • BML is designed to work in event driven systems • <event> is used to fire an event message • <wait> is used to wait for an event • If the event does not occur after a set time, wait can fire a no-event message • After the event occurs, or the timeout is exceeded, the script continues

  21. Coordination with events:handshake subject 1 extend wait connect pump withdraw timeout pump withdraw extend connect subject 2

  22. Coordination with the observer • An observer observes a specific part of the world and provides timing information on that • Example: beats in music • Why observers instead of events? • Explicit outside world trigger • Multiple (repeated) trigger • Timing of observer triggers can be predicted for easy planning • Synchronization within behavior phases • <wait> does not suffice

  23. Coordination with the observerClapping

  24. Conclusions • Gesture synchronization mechanisms are also found in rhythmic motion • Adaptation of timing in gesture affects several movement characteristics • Linear speedup/slowdown does not suffice • Gesture coordination can be modeled using BMLT

  25. Open issues • What modalities do we have to stretch/skew/skip? • Can we generalize our findings from clapping/handshake? • Do the semantics of a motion change if we change its timing? • E.g. emotions, individual features

  26. Questions

  27. Easter eggs

  28. Synchronization with observer • Declaration: <observer id="beatObserver1"/> • Synchronization to beat 1<animation stroke="beatObserver1:1"> • Synchronization to closest beat/all beats <animation stroke="beatObserver1">

More Related