Jun Rekimoto Interaction Laboratory SONY Computer Science Laboratories, Inc.

1. SmartSkin： An Infrastructure for Freehand Manipulation on Interactive Surfaces Jun Rekimoto Interaction Laboratory SONY Computer Science Laboratories, Inc.

2. Background • Various types of “realworld” surfaces become interactive media • Ex:DigitalDesk[Wellner] • Support collaborative work • Fusion of virtual & real • Camera-based input technologies • Sensitive to lighting conditions, occlusion • Camera cannot be integrated in the surface • Possibility of other principle？

4. Lev Theremin1896-1993

5. Lev Theremin1896-1993

6. SmartSkin: sensing principle transmitter transmitter receiver receiver 2πfCV

7. SmartSkin: sensing principle Conductive & Grounded Object (Human Body) transmitter receiver transmitter receiver

8. To host PC wave signal reference signal Analog-Digital converter receiver receiver receiver receiver receiver receiver reference signal analog switch low-pass filter signal from electrode amp to ADC amp

9. Prototype I:Interactive Table • An wooden table covered by 8×9 electrodes. • 30fps

10. Prototype I:Interactive Table • An wooden table covered by 8×9 electrodes. • 30fps

13. Multiuser, Multihand interactions

14. Distance measurement

15. Input by hand(arm) shape

16. Peak position estimation a potential field created by bicubic interpolation sensor values peak position

17. Potential Driven Movement

18. PrototypeII: GesturePad 24 (transmitters)× 32 (receivers) (1cm pitch)

21. Multiple finger detection Various image processing algorithm can be applied. • Connected region labeling • Pattern matching • Object tracking • Etc.

23. Interactions by Multi-fingers

24. Interaction by physical objects As a platform of tangible interfaces. electrodes

25. SmartSkin • Capacitive sensing (c.f., Thermin) • Mesh-shaped electrodes • Multiple object tracking • 2.5D (2D position, distance)＋shape • Fingertip detection by interpolating • Can be embedded in various objects • Low-cost, scalable

26. Discussions： analogy between biological “Skin” and artificial surfaces • form • Sensors on the surface • Input（outside→skin） • Touch, pressure, temperature • Output（skin→outside） • image (e.g., Chameleon) • Tactile • communication • Artificial surface -> Skin

27. Future Directions • Interaction techniques featuring multi hands/fingers, shapes • Non planar surface • Integration with a flat-panel display • Data communication • Ex: table becomes a communication medium • Special Thanks to … • Takaaki Ishizawa、Asako Toda（Keio University） • Kentaro Fukuchi (Tokyo Institute of Technology)

28. Smart Keyboard • Two hand input →Text Input • Single Hand →Command

29. fabric shield layer receiver transmitter body Interactive Clothing

32. DiamondTouch [MERL2001] • Identifies individual users • Requires external electrodes (chairs)

33. Near-Field data transmission

34. Hand tracking without camera

35. 1-10μA 100-500KHz (5Vp-p) 動作原理（2） amp I-V amp A/D Converter LPF analog switch Transmitter (AVR)

36. Wireless .11, BT etc. ID

37. 電極 物体を介した操作

38. Application Domains • Where to install? • Furniture, Floor, Wall • Wearables -- Clothing, Bag, Shoes • Tablet, Keyboard, Mouse, Gamepad • Mobile Devices • Robot • How to use • as a primary input device • to augment existing interaction

39. Multi-Hand/Finger Input

40. 複数の指による入力

41. 距離の判定

43. The Concept • Beyond The GUI dogma (Mouse+Display) • Inspired by analogies between biological skin & artificial surfaces • As An Input Device • Touch, Proximity, Digital Data (Electrostatic/magnetic coupling), energy • As An Output Device • Pixel Spatial Pattern, Pixel Temporal Pattern (beacon), Digital Data, energy, tactile, sound? • As A Communication Device • Information, Motion, Gesture, Emotion, Intimacy

44. なぜSkin? • 生物の「皮膚」の機能 • 保護、形態保持 • 入力（触覚） • 出力（表示）