Interaction Techniques with Mobile Devices

1. Interaction Techniques with Mobile Devices Jingtao Wang jingtaow@cs.berkeley.edu March 6th, 2006 Guest Lecture for CS160

2. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

3. Why Mobile Devices Matters • 6.5 billion people in the world • 1.5 billion cell phones worldwide • 500 million PCs (?) • 46 million PDAs • 1 million TabletPCs • Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned

4. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

5. Mark Weiser (1952 – 1999) Introduced the idea of “ubiquitous computing”

6. Weiser’s Vision “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it”

7. Weiser’s Vision: Pervasive • Mainframe • many people share 1 computer • PC • 1 person with 1 computer • Ubicomp • many computers server each person

8. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

9. Key Challenges in Making Mobile Applications • Limited Physical Resources • CPU, Memory, Screen Size, Input Devices, Battery Life etc • Diversified Context of Use • Different Activities • Limited Attention

10. Limited Physical Resources • A mobile device usually has 1/100 CPU power, 1/30 Screen resources, 1/20 Memory, and extremely limited input devices when compared with desktops in the same era. • Small Screen Geography is different b. Small Screen a. Large Screen

11. Diversified Context of Use

12. Different Activities • People use small-screen devices for different activities than desktops; don’t assume you understand these activities already

13. Limited Attention • Don’t assume your applications have people’s full attention; they’re doing something else while using your device.

14. Context, Activity, Attention • There is more opportunity for purpose-specific or context-specific devices than for general-purpose solutions that try to work for everyone in any situation.

15. One Sentence Summary • There is no silver bullet in designing mobile applications, but there is one sentence you should remember - Mobilize, Don’t Miniaturize !

16. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices

17. Input Techniques for Mobile Devices • Pointing • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

18. Common Pointing/Navigation Techniques JogDial iPod Dialpad TrackPoint Touch Screen Four-directional keypad

19. TinyMotion – Camera Phone Based Pointing • Detecting the movements of cell phones in real time by analyzing image sequences captured by the built-in camera. • Typical movements include - horizontal and vertical movements, rotational movements and tilt movements.

20. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

21. (Virtual) Keyboard Input

22. How to Make QWERTY Keyboards Portable ? • Break

23. Making QWERTY Keyboards Portable • Reducing the Absolute Size • Reducing the Number of Keys • Making Keyboards Foldable • Projective Keyboard

24. Projective Keyboard From http://www.vkb-tech.com

25. Projective Keyboard – Working Mechanisms • Template creation • Reference plane illumination • Map reflection coordinates • Interpretation and communication

26. Can We Perform Better Than QWERTY? • Originally QWERTY layout is manually optimized for two handed, alternative typing to minimize mechanical jam OPTI ATOMIK FITALY OPTI II

27. Theories Behind Quantitative Keyboard Layout Optimization • Fitt’s Law • Digraph Distribution Model in a Language

28. Can We Use Less Buttons than a Full QWERTY? 12-button Keypad 15-button Keyboard Half Keyboard

29. Disambiguation Methods for Reduced Size Keyboard • The QWERTY keyboard itself is ambiguous! ( A vs. a, 3 vs. #) • Pressing Several Keys together (shift key) • Multiple Key Press • Multi-Tap (22.5 wpm*) • Two-Key Input (25.0wpm*) • Dictionary/Statistics Based Disambiguation Methods • T9/T15 (45.7 wpm*) • LetterWise *Performance Upper Bound Estimation from Silfverberg 2000

30. FingerSense – Button Disambiguation by Fingertip Identification • Differentiating a pressing action by identifying the actual finger involved • Can be Faster than Regular Tapping When the Adjacent Tapping Involves Different Fingers and Different Buttons (59% on a phone keypad)

31. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

32. Handwriting Input • 1938 George Hansel, U.S. Patent 2,143,875, machine recognition of handwriting • 1957 T. L. Dimond's stylator - the first on-line handwriting recognizer prototype • Newton, Palm Pilot, PocketPC, CrossPad, TabletPC

33. Handwriting Recognition - Terminology • Printed Character Recognition (OCR) • Relatively mature these days, key challenges – layout analysis, fonts recovery, robust recognition for low quality, low resolution input • Major Usage – Digital Library • Handwritten Character Recognition • Online HWR (With Temporal info) • Character, Word, Sentence Level • Offline HWR (Using raster image as input, no temporal info) • Major Usage : Bank Check Recognition, Postal Automation

34. Word/Sentence Level Recognizer • Build on Top of Character Recognizer • General Strategy • Over Segmentation • Call Character/Component Recognizer, Get a List of Candidates with Scores • Apply Geometry Spatial Information ( size, component gap ) Language Information (Dictionary, Language Model etc) to Each Sub Path • Use Hypnosis Search (Dynamic Programming, A* etc) to Determine the Best Possible Path

35. Challenges in Online HandwritingRecognition • Character Set/Dictionary Size (Especially Asian Languages!) • Cursive Writing Styles/Broken Strokes/Duplicate Strokes/Omitted Components • Stroke Order Variations • Limited memory and CPU Power in Small Devices

36. Some Prototype Recognizers from IBM

37. New Form Factors - Anoto Pen • Commercial Product is Available In the U.S. Market (Logitech IO Pen) • Uses A Camera Mounted Beside the Tip of the Pen and Preprinted Dot Patterns to Detect Pen Movment

38. SHARK – Shorthand Writing on Stylus Keyboard • A Combination of Virtual Keyboard and Handwriting Recognition • Writing Shape of a Word (Shorthand) is Defined By the on Screen Location of Characters in the Word

39. EdgeWrite Input • An EdgeWrite user enters text by traversing the edges and diagonals of a square hole imposed over the usual text input area • Faster and More Reliable Than Regular Graffiti • Especial Useful for People with Motor and Muscle Disabilities

40. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

41. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

42. Emerging Marker Based Interactions on Camera Phones

43. Towards More Sensitive Mobile Devices

44. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

45. Peephole Displays (With Demo)

46. Zoomable Interface on Mobile Devices • ZoneZoom By Microsoft • Take advantage of spatial memory VS.

47. Halo - A Virtual Periphery for Mobile Devices Provding Visual Cue for Objects Located Out of the Small Screen

48. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

49. Using Mobile Devices with Desktop Computers • Pebbles Project at CMU • Using a PDA as additional keypad, touch pad, scroll wheel and controller of PointPoint slides for desktop Applications • http://www.pebbles.hcii.cmu.edu/

50. Using Mobile Devices with Laptops Wang 2002