New Interface Design for Human Experience and Expression

1. New Interface Design for Human Experience and Expression Sidney Fels Human Communication Technologies Laboratory Dept. of Electrical and Computer Engineering & Media and Graphics Interdisciplinary Centre University of British Columbia Vancouver, BC, Canada ssfels@ece.ubc.ca

2. Human Communication Technologies Lab • HCT lab started in 1998 • Focus on human experience • Intimacy, Embodiment • Aesthetics of relationships • Communication • Interdisciplinary by practice and training • 8000ft2 of facilities, housed within ICICS complex • MAGIC started in 1991 • Focus on Media technologies and graphics • Research • Industrial • Cultural • Education • 2000ft2 of facilities housed in the Forest Sciences building hct.ece.ubc.ca

3. Research Directions • Three main areas of research • New interface design • Art, Music and Expression • Modeling, Tools and Graphics

4. Example Projects at HCT • New Interfaces • Interaction with Large Display Surfaces • T. Tang • Cubee: a 3D display • Ian Stavness & Florian Vogt • Affective communication interface • 2 Hearts musical system (G. McCaig) • Interactive Yoga System (xxxx)

5. Example Projects at HCT • Art, Music and Technology • Sound • Tooka • Tongue’n Groove • D’Groove • Interactive Art • Iamascope • Swimming Across the Pacific • Plesiophone • 1,001,001 Faces • Sound Room & Sound Weave • 1 Kingsway: Flow • Performance • Waking Dream • Forklift Ballet • Laika Space Program

6. Example Projects at HCT • Modeling, Tools, Graphics • Biomechanical Modeling for Articulatory Speech Synthesis • large team of people • Parallel Distributed Camera Array & Local Positioning System • large team of people

7. New Interfaces, Art & Technology

8. Iamascope • Interactive multimedia artwork • Participants are put inside a large kaleidoscope • Participants movements also map to music

9. Music Synthesizer Iamascope System 170” Video Projector Kaleidoscopic Image speakers Active Video Region Video Camera Video Image Vision-to-music subsytem Texture Memory Texture Mapping Music Production Image Processing Kaleidscope subsytem

10. Iamascope Video Overview

11. Swimming Across the Pacific • Swimming Across the Atlantic • Queen ElizabethII • Misheff, 1982 • Southampton to New York • Contemporary vision • Swimming in an airplane • Los Angeles to Tokyo • Created swimming apparatus • Exhibition • Audience participation • Team of 8 people

13. Tooka: A Two Person Flute • Create musical instrument that needs two people to play • Explore human intimacy

14. How Tooka Works Pitch Bend Octave Vibrato Capture/Sustain Volume Pitch

16. Cubee: 3D display

17. Hardware

18. Cubee Scenes • Static object glued to box • e.g. teapot display case • Static scene fixed in world • e.g. volumetric cut-plane • Dynamic objects inside box • e.g. ball rolling through 3D maze

19. Glove-TalkII • Translates hand gestures to speech • like a musical instrument • mapping partially learned • speech is • intelligible • expressive • slower than normal

20. Von Kempelen (1790) Fels & Hinton (1994) Dudley et al. (1939) Bell & Bell (1880) Artificial Vocal Tract Phoneme Generator Finger Spelling Syllable Generator Word Generator 10-30 100 130 200 500 Spectrum of Gesture-to-Speech Mappings Fels & Hinton (1990) Kramer & Leifer (1989) (msec) approximate time/gesture for connected speech

21. Glove-TalkII Vocabulary • Loosely based on articulatory model of speech • Vowels • open configuration of hand represents open vocal tract • X,Y position determines vowel sounds (like tongue) • Consonants • constrictions in hand represent constriction in vocal tract • Stop consonants produced with ContactGlove • Volume controlled with foot pedal (air pressure) • Pitch controlled by Z position of hand (vocal cord tension)

22. X Glove-TalkII System • 3 neural nets • Output: Parallel Formant Speech Synthesizer • ALF, F1, A1, F2, A2, F3, A3, AHF, V, F0 • 100 Hz, 6 bit quantization [0, 63] Right Hand Data x,y,z, roll, pitch, yaw (60 Hz) 10 flex angles 4 abduction angles thumb and pinkie rotation wrist pitch and yaw (100Hz) Fixed Pitch Mapping Speech Output Synthesizer V/C Decision Network Preprocessor Contact Switches Vowel Network Combining Function Consonant Network Foot Pedal Fixed Stop Mapping

23. Glove-TalkII VIDEO Vocal Sounds Alphabet Numbers Sam I Am

24. Intimacy and Embodiment • Want interfaces that feel “good” to use • Humans and machines intimately linked • degree of intimacy supported may determine success • Types of relationships: • human to human • human to machine

25. Intimacy • Intimacy is a measure of match between the behaviour of an object and the control of that object. • extension of “control intimacy” from electronic musical instruments analysis (Moore, 1997) • High intimacy implies: • object feels like an extension of self • satisfaction derives from interacting with object • emotional expression flows • requires cognitive effort to prevent • Relationship dynamics • Aesthetics of interaction

26. self object Case 2: Self embodies Object self object Case 3: Self disembodied from Object object self Case 4: Object embodies Self object self Intimacy: Embodiment vs. Disembodiment Aesthetics of Interaction response Case 1: Object disembodied from Self control reflection belonging

27. Directions • How to achieve intimate experiences? • Two user-centric techniques: • mirrors • Eliza (Weizenbaum, 1966), Iamascope (Fels & Mase, 1999), Wooden Mirror (Rozin, 1999), bots • masks • many other complex techniques

28. Summary: HCT Activities • Modeling, Graphics and Tools • ArtiSynth for articulatory speech synthesis • OpenVl for providing image processing infrastructure • New Interface Design • approach to design using intimacy • Art, Music and Technology • explore expressive directions of technology

29. Questions: Contact Information: http://hct.ece.ubc.ca/ Sidney Fels ssfels@ece.ubc.ca