Control of avatar gestures

1. Control of avatar gestures Francesca Barrientos fbarr@cs.berkeley.edu Computer Science Division UC Berkeley Bridging the Gap Workshop

2. Communicating via desktop VR • Communicate verbally using speech • Communicate visually using avatars • Body pose • Body movement Bridging the Gap Workshop

3. Project goals • Build system for controlling avatar gestures • Simple input device such as a pen and tablet • Understand expression using avatar body • How do we express ourselves using avatar bodies? • How do we evaluate a user’s sense of expressiveness? Bridging the Gap Workshop

4. Control problems • Limited input and complex output • Control interface divides user’s attention Bridging the Gap Workshop

5. Nonverbal behaviors • Displays • Gesture • Facial expression • Gaze • Posture • Functions • Augment/replace speech • Regulate conversation • Indicate emotion • Cognition • Awareness • Conscious control Bridging the Gap Workshop

6. Nivi waves to Harry.) Other approaches Natural language inference Expression selection Automatic animation Bridging the Gap Workshop

7. Gesticulation • Co-occurs with speech • Free form and made up in the moment • Meaning derived from combination of words and gestures Bridging the Gap Workshop

8. Demo of Doodler Bridging the Gap Workshop

9. Doodler: things that worked • Within a small range, tracking is intuitive • Can produce free form gestures • Movement seems expressive • Control is transparent • No hunting for controls Bridging the Gap Workshop

10. Doodler: limitations • Hard to form specific shapes • Need a way to switch between different mappings • Hard to do gestures which involve different body parts serially • Lack of avatar proprioception Bridging the Gap Workshop

11. Speech independent behaviors • Emblematic gesture • Replace words • Symbolic • Arbitrary • Affective expression • Emotion • State of being Bridging the Gap Workshop

12. Demo with writing • Using alphabet Bridging the Gap Workshop

13. Writing as control • Discrete symbols + Continuously valued variables b c a • Intentional and unconscious Bridging the Gap Workshop

14. Writing as control • Elements of discrete and continuous control • Letter selects a discrete choice • Way it is written selects values for • Specify multiple variables • Explicit control? • Implicit control? • Don’t have to look Bridging the Gap Workshop

15. Summary • Simple • Continuous • Implicit • Flows • Abstract • Unwitting • Transparent • Complex • Discrete • Explicit • Conscious • Intentional • Symbolic Bridging the Gap Workshop

16. Future work • Design and record gesture vocabulary • Map variations in writing to variations in gesture animation • Figure out how to analyze expressive aspects of control Bridging the Gap Workshop

17. Facing the Gap • How to think about expression • Computer science • Psychology • Human-computer interaction • Puppetry • New culture • How to study it Bridging the Gap Workshop

18. Explore design space Bridging the Gap Workshop

19. Two control schemes • Continuous control with free form motion • Discrete control with more complex motion and multiple parameters Bridging the Gap Workshop

20. Other solutions continued • Analysis of text • ComicChat uses keywords, acronyms, punctuation, etc. • Semi-autonomous behaviors • BodyChat by Vilhjálmsson • Simple kinematic controls • Sliders and similar widgets (e.g.. Slater) • Full body motion capture Bridging the Gap Workshop

21. Other solutions for nonverbal communication • Discrete choices (menus) of expressions • Usually affective (happy, sad, angry…) • Usually facial • Usually used with chat environments • Examples: • Emotion wheel in ComicChat • Palace • Gesture/Mimic panel in Vlnet Bridging the Gap Workshop