What your avatar can reveal about your handwriting

1. What your avatar can reveal about your handwriting Avatar gesture from pen gestures Francesca A. Barrientos Computer Science Division 23 May 2002  Ph.D. Dissertation Seminar

2. Interacting in avatar worlds A wedding May 8, 1996 (Damer ‘98) Laurel’s herb farm in AlphaWorld (Damer ‘98) Ph.D. Dissertation seminar

3. Avatar nonverbal communication • In physical world, language embedded in matrix of sounds and visuals • Avatar, as a virtual body, can send nonverbal communication Ph.D. Dissertation seminar

4. Problem statement We want an interaction technique for controlling avatar gesture • Controls expressive movement • Seamless with verbal communication Ph.D. Dissertation seminar

5. Avatar gesture from pen gesture • Body gesture has symbolic and qualitative aspects • Pen gesture carries symbolic and continuous data • Pen gesture simultaneously selects avatar gesture and modulates multiple expressive qualities Ph.D. Dissertation seminar

6. Control using handwritten letter • Writing pen gesture triggers animation • Body gesture is “sweep” to side • Symbol is letter ‘s’ l • Quality being varied is size Ph.D. Dissertation seminar

7. Overview • Background on nonverbal communication • Why it’s important • Why previous control techniques are inadequate • Description of interaction technique • Description of implementation • Conclusions Ph.D. Dissertation seminar

8. Kinds of nonverbal cues • Vocalics • Appearance • Proxemics & Haptics • Affective display • Kinesics • Facial expression • Gaze • Posture • Gesture Ph.D. Dissertation seminar

9. Gesture types Deictic Iconic Metaphoric Beat Adaptors Regulators Ph.D. Dissertation seminar

10. Importance of gesture • Encodes ideas shared with speech • Clarifies meaning when speech is ambiguous • Useful cue when outside noise interferes with speech • Aids utterance generation • Smooths over social intercourse • Communicates mood/emotion Ph.D. Dissertation seminar

11. Interaction with speech • Coverbal gesture shares meaning units with spoken language • Language-like gestures fill in for a word or phrase in a sentence • Emblems have standard forms and well understood within a group Ph.D. Dissertation seminar

12. Observed virtual nonverbal displays • Presence • Appearance • Proxemics • Conversation group formation • Personal space WorldsAway from Fujitsu Avabar by Zuidema (Damer) Ph.D. Dissertation seminar

13. Designed behaviors • Static expressions • Facial • Posture • Gesture • Animated motions • Gesture • Entertaining dances • Custom animations WorldsAway from Fujitsu Ph.D. Dissertation seminar

14. Related work • Commercial worlds • Blend of text virtual communities and computer graphics • Worlds Chat – first 3D world in 1995 • Multi-user virtual environment research • Vlnet (Guye-Vuillème et al ’98) • ComicChat (Kurlander+Skelly+Salesin ‘96) • Autonomous avatars (Vilhjálmsson+Cassell’98, Cassell et al ‘94) • Acting in virtual reality (Slater et al ‘00) • Synthetic characters • Improv (Perlin+Goldberg ’96) • Alive (Blumberg+Galyean ’95, Maes et al ‘97) • Jack (Badler ’97) • Computer mediated conversation visualization • Collaboration-at-a-glance (Donath ’95) • Chat Circles (Viegas+Donath ’99) Ph.D. Dissertation seminar

15. Vlnet gesture/mimics panel • Button for each expression or gesture • Modulate speed with slider (Guye-Vuillème et al) Ph.D. Dissertation seminar

16. ComicChat • Text inference • Emoticons • Sentence Structure • Keywords • Emotion wheel (Kurlander et al) Ph.D. Dissertation seminar

17. BodyChat avatar agents • Automatic conversation regulation behaviors • Salutations • Envelope feedback • Facial expression • User specifies high-level intentions • Avatar expressions driven by chat text • Avatar software manages gestures and gaze behavior (Vilhjálmsson+ Cassell) Ph.D. Dissertation seminar

18. Limitations of current techniques • Mainly speech independent nonverbal displays • Emotional facial display and posture • Emblematic gesture • Not general • Cumbersome interface • Graphical interface requires hunt and select • Cannot scale • Lack control over multiple expressive features of movement Ph.D. Dissertation seminar

19. Why pen gestures • Natural • People doodle while talking and listening • Expressive • Reflects emotional state • Very personal • Can be intentionally manipulated • Dual nature • Analog and digital • Symbolic and qualitative • Discrete and continuous • Information and emotion Ph.D. Dissertation seminar

20. Continuous interaction Computers fragment our thinking by substituting discrete events for continuous actions. -Malcolm McCullough (Abstracting Craft, p. 53) [Gestures] can...enhance the experience of agency through kinesthetic involvement and the feeling of directness. -Brenda Laurel (Computers as Theatre, p. 158) Ph.D. Dissertation seminar

21. Interaction technique w- s- r- LibraryGenerated offline InputUser writes letter in GUI GenerationSelection and synthesis Animation Gesture performed library GUI Gesture generator Avatar animator Ph.D. Dissertation seminar

22. Interaction schematic Ph.D. Dissertation seminar

23. Design issues • Pen gesture input set • Using letters of the alphabet • Mapping frompen gesture features to avatar movement parameters • Handwriting features to extract • Gesture movement parameterization • Avatar gesture animation synthesis method Ph.D. Dissertation seminar

24. Handwriting features • Desired properties • Controllable • Computable • Most important feature types (according to handwriting analysis) • Size • Speed • Pressure Ph.D. Dissertation seminar

25. Motion parameterization • Formal systems • Shape-Effort (Labanotation) • Formal sign language systems • Physical • Size • Speed • Sustain • Emotional • Emphatic • Listless • Tentative Ph.D. Dissertation seminar

26. Physical mapping Ph.D. Dissertation seminar

27. Gesture synthesis • Input: gesture and movement parameter values • Multi-linear interpolation from set of sample gestures • Each avatar gesture comprises a set of sample motions • Each motion sample has different expression • Samples are annotated with its movement parameter values Ph.D. Dissertation seminar

28. Modulation through multilinear interpolation Gβ Speedu Iβ(0,1) Iβ(1,1) Iβ(l,k) Iβ(0,k) Iβ(1,k) k Iβ(0,0) Iβ(1,0) Sizev l • Joint trajectory I • Rotation angles over time • Gesturetypeβ • Semantic category • Set of 2n prototypes - G • n style parameters • Prototypes represent extremal trajectories • GestureinstanceIβ(u,v ) • Vector of joint trajectories • Multilinear interpolation on type produces instance Ph.D. Dissertation seminar

29. Interpolating speed • Interpolation along curves of different durations • Time dilation step • Determine duration for interpolated curve • Choose sample rate on interpolated curve • Compress slower curve - sample at proportionately slower rate • Sample faster trajectory at proportionately faster rate Angle θ t Ph.D. Dissertation seminar

30. Framework for applying pen gesture to avatar gestures • Subproblems • Expression map design • Gesture synthesis technique • Can explore other mappings and synthesis techniques • Labanotation parameterization of movement • Emotion parameterization • Customize toward particular domain Ph.D. Dissertation seminar

31. Implementation Cursive • Application for interactively controlling VRML avatars over the internet • Use to test animations • Architecture permits independence from specific avatar world software • Viewer can see animations without installing new software • Facilitate testing of shared object behaviors in virtual worlds Ph.D. Dissertation seminar

32. Avatar gesture samples • Built motion capture system • Magnetic position/orientation sensors (Flock of Birds from Ascension Corp.) • Recorded gestures • Vary size and speed • Can create very personalized gesture Ph.D. Dissertation seminar

33. Cursive screenshot Cursive window VRML browser VRML avatar Ph.D. Dissertation seminar

34. Architecture Ph.D. Dissertation seminar

35. Communication open socket request avatar Download avatar login notification VWorld server Web server sends commands Cursive communicates with any copies of user’s avatar • Driver logs into Vworld server • Other viewers receive notification • Other viewers request and download avatar copy • Avatar opens socket connection to Cursive • Cursive sends gesture commands via socket viewer host driver host Ph.D. Dissertation seminar

36. Evaluation • Very simple to control one parameter at a time • More complicated to control size and speed simultaneously • Effective usage requires practice • Require further investigation into mapping handwriting features to movement parameters • Viable technique for controlling avatar gesture Ph.D. Dissertation seminar

37. Expert use …the art of finding and executing an effective gesture is learned through the more indirect means of observation, experimentation, performance, and evaluation, and it is a skill that continues to grow over time. - Brenda Laurel (Computers as Theatre, p. 155) Ph.D. Dissertation seminar

38. Summary • Want to control richly expressive, spontaneous gesture in avatar worlds • Solution is an interaction technique employing pen gesture input • Cursive: an implementation of this interaction technique Ph.D. Dissertation seminar

39. Contributions • Novel interaction techniquethat augments the potential repertoire of avatar nonverbal communication • Possible to control spontaneous gesture • Control over expressive characteristics of gestural movement • Frameworkfor applying pen gesture to avatar gesture • Mapping handwriting features to movement parameters • Synthesis of expressive gesture • The algorithms and methods used to implement the technique • Cursive, a working application that applies the interaction technique Ph.D. Dissertation seminar

40. Future work • Explore other mappings • emotional expression in handwriting and gesture • Shape-Effort parameterization of movement • Explore other gesture synthesis methods • Reduce cost of obtaining gesture motion samples • Develop a framework for determining avatar gesture vocabulary • How many gestures • What types of gestures Ph.D. Dissertation seminar

41. Conclusion • Transmitting bodily nonverbal communication through the internet is an exciting idea. • Think of the computer as a medium for personal expression • Continuous/rich interaction • Playful behavior • Sense of engagement • Handwriting has a place in affective computing Ph.D. Dissertation seminar

42. Acknowledgements A few of the people who helped with my research and with this talk. My dissertation committee: Prof. John Canny, Prof. James Landay, Prof. John McWhorter. My gesture model: Erin Dare. My dissertation writing group: Blanca Gordo, Jeffrey Ow, Dr. Ellen Sacco-Fernandez, Lynne Horiuchi. Other talk critics: Miriam Walker, Scott Klemmer, Dan Glaser, Jeremy Risner, James Lin, Jason Hong, Dr. Eric Paulos. Ph.D. Dissertation seminar

43. Partial bibliography • Damer, B., Avatars! Exploring and Building Virtual Worlds on the Internet. Berkeley, CA: Peachpit Press, 1998. • Guye-Vuillème, A., T. Capin, I. Pandzic, N. Magnenat-Thalmann, and D. Thalmann, "Non-Verbal Communication Interface for Collaborative Virtual Environments," in Proc. CVE 98, June 1998, Manchester, 1998. • Vilhjálmsson, H. H. and J. Cassell, "BodyChat: Autonomous Communicative Behaviors in Avatars," in Proceedings of the Second International Conference on Autonomous Agents, May 9-13, 1998, Minneapolis: ACM, 1998, pp. 269-276. • Cassell, J., C. Pelachaud, N. Badler, M. Steedman, B. Achorn, T. Becket, B. Douville, S. Prevost, and M. Stone, "Animated Conversation: Rule-Based Generation of Facial Expression, Gesture and Spoken Intonation for Multiple Conversational Agents," in Proceedings of SIGGRAPH '94., 1994. • Slater, M., J. Howell, A. Steed, D. P. Pertaub, M. Gaurau, and S. Springel, "Acting in Virtual Reality," in Collaborative Virtual Environments 2000: ACM, 2000, pp. 103-110. • Perlin, K. and A. Goldberg, "Improv: A System for Scripting Interactive Actors in Virtual Worlds," in Proc. Siggraph 96, H. Rushmeier, Ed., New York: ACM Press, 1996, pp. 205-216. • Blumberg, B. M. and T. A. Galyean, "Multi-level direction of autonomous creatures for real-time virtual environments," presented at SIGGRAPH, Los Angeles, CA, 1995. • Maes, P., T. Darrell, B. Blumberg, and A. Pentland, "The ALIVE system: wireless, full-body interaction with autonomous agents," Multimedia Systems, 5, no. 2, 1997, pp. 105-12. • Badler, N., "Virtual Humans for Animation, Ergonomics, and Simulation," in IEEE Workshop on Non-Rigid and Articulated Motion, June 1997, Puerto Rico, 1997. • Donath, J. S., "The illustrated conversation," Multimedia Tools and Applications, 1, no. 1, 1995, pp. 79-88. • Viegas, F. B. and J. S. Donath, "Chat Circles," in CHI 99: ACM, 1999, pp. 6-19. • Laurel, B., Computers as Theatre. Reading, Massachusetts: Addison-Wesley, 1993. • McCullough, M., Abstracting craft : the practiced digital hand. Cambridge, Massachusetts: The MIT Press, 1996. Ph.D. Dissertation seminar