Input Research at Microsoft Sensors, Mice, and Keyboards, Oh My!

1. Ken Hinckley Microsoft Research March 1, 2002 Input Research at Microsoft Sensors, Mice, and Keyboards, Oh My!

2. Develop interaction techniques & software to leverage new HW capabilities Contextual sensing of the user & physical environment (“passive input”) Bridge the physical and virtual worlds. Enhance the degree & directness of input, improve performance of devices / sw Theory: models & taxonomies of input, Experimental studies My Research Approaches

3. Overview • A few comments on Keyboards &Design of the MS Office Keyboard • What’s so Great about the Mouse Anyway? & Overview of mouse innovations • Fun with Sensors on PDA’s & Cell Phones • Document Navigation Research

4. Why Are We Still Using Keyboards? Sholes typewriter, 1868 • They leverage human skills • Procedural Memory (“muscle memory”) • Power law of practice: T = a P b • Minimal attention / cognitive resources required • Touch-typing: you can focus on what you’re composing

5. Why Have Keyboards Changed So Little? • Qwerty Layout (1878) ? • Dvorak probably only 5-15% faster. (retrain on qwerty?) • Chording keyboards VERY fast but slow to learn • E.g. stenographer 250-300 wpm max, 3 years to learn • Why Haven’t Alternatives Replaced It? • Natural handwriting is slow (15-20 wpm) • Voice Dictation Input • English keyboard-mouse text entry about twice as fast (cwpm) as voice dictation (Karat et al. 1999) • Speaking words can interfere with STM (Karl 1993)

6. Some Trends in Keyboard Design • Ergonomic design • Some Ergo designs (e.g. split) seem to have benefits • Integrated Pointing Devices • Mobile applications: • Small foldable keyboards • telephone keypad text entry • Soft (virtual) keyboards • very demanding of visual attention • Wireless, New hotkeys on top, colorful ID, etc.

7. MS Office KeyboardHugh McLoone, Ken Hinckley, Ken Fry, Steve Bathiche • Commercial kbd designed for bimanual control • Product makes a bet that users get two-handed input, and that it will sell

8. Principles of Bimanual Action • Yves Guiard, 1987. For right-handers: • Right-to-left reference: Action of the right hand occurs within the frame-of-reference defined by the left. • Scale Asymmetry: Movements of the right hand occur at higher spatial and temporal frequencies than the left • Left-hand Precedence: Action starts with the left hand.

9. Principles of (Bimanual) Scrolling • Scrolling is a background task that should be assigned to the nonpreferred hand. • Right-to-left reference: Movement of mouse cursor is within current document view. • Scale Asymmetry: Scrolling is a coarse task, cursor movement & selection are high-precision • Left-hand Precedence: Scrolling precedes detailed activity in the document. (MacKenzie 1998)

10. No switching between pointing & scrolling Overlapped action of the 2 hands Maintain visual focus & concentration on work Buxton & Myers 1986bimanual scrolling ~25% faster than scroll bar Bimanual Scrolling

11. Bimanual Controlon Office Kbd • Navigation controls on left • Scrolling [wide wheel] • Web [Forward / Back] • Windows [AppToggle] • Cut, Copy, Paste also well suited to left side • Compound selection [or placement of IP] + articulation of command

12. Left Pod design explorations • Rows allow fingers to fall naturally on buttons • AppSwitch below C/C/P; scroll function at top

13. Empathy & Forgiveness • Many users perceive keyboards as unforgiving, cryptic, dangerous to touch • F1-F12 meaningless to most persons • No clearly labeled way to Undo mistakes • Insert, NumLock easy to hit accidentally & trigger modal behavior • Yet keyboards with less functionality do not sell • Must incorporate legacy keys for back-compatibility

14. What kbd functions do people actually use?

15. Efficient one-step commands • Clearly labeled keys do not exist for many of the most commonly used sw functions • Print, Save, Cut / Copy / Paste, … • Traditional hotkeys require key chords: • Novice users do not know or use them • Awkward (unnatural postures, static load) • error-prone (e.g. Ctrl+V instead of Ctrl+C) • Slow to articulate (e.g. Alt +  for Back) • Menus (pull-down, right-click, marking menus) require pointing device acq when hands on kbd

16. Examples of clear, efficient, less error-prone design • F1-12 buried (F Lock); labelled fn’s • Insert moved under =Delete enlarged • NumLk, ScrollLk, SysRq, etc. buried • Undo / Redo

17. Office Kbd: Future Work • Much potential for enhancements, new features, for left pod / bimanual control • Experimental studies of user performance • Validate current design elements • Quantify opportunities for improvements • How to innovate / improve / add functionality to kbds in general? (Comfort, efficiency, …)

18. 1968 Douglas Englebart & colleagues Stanford Research Institute Separate rollers for x, y Why has it endured for so long? The First Mouse

19. Some Desirable Properties of the Mouse • Approx. as good as pointing with hand itself (Card) • from Balakrishnan et al. CHI ’97: • Flexibility of grip – both power & precision grasp • Stability – mouse stays put, fast to acquire • vs. stylus; slightly better for high-precision tasks • Relative mode: limits footprint, no nulling problem • Buttons integrated & orthogonal to sensed plane • Dragging works well, minimal motion on click • Familiar, entrenched device; +10% not enough

20. What has been explored for desktop mice? What strategies are there to improve user performance? What has succeeded in the marketplace? What are some interesting areas for the future? What are some really silly ideas? A Quick Illustrated Tour: Some Axes of Innovation in Computer Mice

21. Image Corrleation using CMOS camera 2KHz + sampling rate, almost any surface No moving parts, cost still dropping, 100% of market? Wireless {phone, network, device, …} = horseless carriage Gyros, …? Microsoft IntelliEye Logitech Cordless MouseMan Wheel New Implementation Technologies GyroPoint

22. Scrolling (1) save time to grab scrollbar (2) maintain visual focus on primary task Wheel: MS IntelliMouse, Mouse Systems Isometric Joystick: IBM ScrollPoint Touchpad: Fujitsu ScrollPad Mouse … Multi-Channel Inputfor Secondary Tasks

23. Rotation Two-ball mouse, MacKenzie also Wacom tablet “4D Mouse” Tilting (4DOF) Rockin’Mouse, Balakrishnan VideoMouse (6DoF) Hand Proximity in X, Y, Z FieldMouse, (Josh Smith, MIT) Additional Sensed Degrees-of-Freedom

24. Fingerprint scan for user ID / login uMatch biolink mouse biolinkusa.com GSR, heart rate, temperature, … EmotionMouse (IBM BlueEyes project) Biometric Sensing

25. Project: Touch-Sensing Devices • Sense hand contact via capacitance • Input modality for awareness • New events: Touch, Release • For Palm area • for button, wheel, thumb, … • Goal: use sensors to enhance & simplify UI • Passive sensing(zero activation force) • Might be possible to do user ID via capacitive coupling

26. Fade UI widgets in / out depending on devices held Cursor sonar on touch Esp. Large displays, multi monitors Hide cursor on release Examples: Adapt UI Based on What the User Is Doing

27. Hand Sensor for Wireless Mouse • Capacitance detector in mousesenses presence of your hand • When you release it, the mouse powers down to save the battery • Conductive foil on inside of mouse shell,1 pin on mouse’s microprocessor • Costs virtually nothing

28. Force, Tactile feedback 10-20% faster pointing speed? But, errors, multiple targets,dependent on visual feedback Imposes constraints on device (esp. force) Still no real proof it has benefit for typical desktop tasks vs. software constraints? Tactile nib in mouse (MacKenzie) Vibrotactile / Force Feedback Tractile (Cambell)

29. A Bit of Fun Dumb Input Trends at the Dawn of the 21st Century Ken Hinckley Microsoft Research

30. Dumb Input Trends • Nail together 2 random things • TeleMouse

31. Dumb Input Trends • Nail together 2 random things • TeleMouse (Altra Technology Inc.) #1 All-Time Dumbest Input Device

32. Dumb Input Trends • Nail together 2 random things • TeleMouse (Altra Technology Inc.) • #2: Universal remote / corkscrew / bottle opener(x-10.com)

33. Dumb Input Trends • Nail together 2 random things • Transparent Plastic • “Interfaces should be transparent to the user”

34. Dumb Input Trends • Nail together 2 random things • Transparent Plastic • Edible Input Devices • Candy Mouse (Amurol Confections Company)

35. Dumb Input Trends • Nail together 2 random things • Transparent Plastic • Edible Input Devices • Less is More?

36. no-button mouse (0% error rate)

37. And Now for Something Completely Different

38. Mobile Devices vs. My Cat: Which is smarter? Mr. Cleo • Aware of sounds, objects • Knows if I walk in the room • My Mobile Device • Unaware of environment • Oblivious to my presence • Selfish & Inconsiderate

39. Can Sensors Enhance Mobile Device UI? • Sensors: • Tilt • Touch • Proximity • ~$3.00 parts cost • What new features do inexpensive, ubiquitous sensors enable? • VIDEO

40. A few comments on video • Auto Power-On: Device understands being “picked up”; This is a valid “command” !!! • P/L switching: Get the most out of a small display & hold how it best suits content / controls • TiltScroll: Touch screen bezel to start • Users see benefits in One-handed operation, Natural, Simpler movement (vs. D-Pad)

41. The “Polite”Cell Phone • Ken Hinckley, Eric Horvitz • Quiet Ringing: • Ring volume softens as soon as you pick it up • Notices when you look at caller ID • Mutes ring if you choose to ignore call • Listen to phone to answer incoming call • No wrestling with TALK button / flip top • Backlight activation when look at screen • Sensors to choose target device for notifications • Prefers vibration alert when you are touching it

42. Speakerphone& Private Use • Ericsson R520 with “proximity switch” (?) • We use proximity range sensor • Private Mode: • Hold to ear, Lowers volume • Speakerphone Mode • Remove from ear, e.g. on table: Maximum vol. • Simple, works surprisingly well • Can cup hand over phone for an “aside” • Objects near phone may trigger private mode

43. Other Fun Stuff • Games • Kirby Tilt’n’Tumble $29.99 • Detect activity, e.g. Walking • Shaking • Shake to erase :-)

44. Sensors Wrap-up • A few simple, low-cost sensors enable: • Automatic Power-On • Portrait-Landscape detection • “Tilt Scroll” • Mobile Phone features • Games, Activity Detection, etc. • Same commands can work across many sizes of device: Pager  PocketPC  TabletPC • Inexpensive, ubiquitous sensors are coming… What are they good for? Are you ready for them?

45. Ken Hinckley (MSR) Ed Cutrell (MSR) Steve Bathiche (R+D PM, Hardware) Tim Muss (Usability Lead) Document Navigation Research • Can performance of the scrolling wheel be improved? • How does it compare to competitors? • How to experimentally evaluate scrolling performance?

46. Accelerated Scrolling Wheel • How do users roll the wheel? • (1) trying to get somewhere fast, or • (2) reading • Roll faster  move further! • No loss of control! • The user does nothave to learn anything new! • For Δt < 0.1, • Δy = K1(1 + K2Δt)α

47. Experimental Study: How do you Evaluate Scrolling? • Previous work: Distance & precision uncontrolled factors… OOPS! • Which technique is the fastest?

48. D W Fitts’ Law: Cough Syrup for Pointing Devices • Widely used to study rapid, aimed movements (Fitts 1954) • Used in pointing device studies since 1978 • Task: Point at a target, W wide at distance D • The Law: • MT = a + b log2(D/W + 1) • a, b fit by linear regression using observed MT • Never applied to scrolling

49. Frame Target line Scrolling Experimentfounded in Fitts’ Law • Scroll back & forth between 2 lines in a doc • Ex: comparing paragraphs

50. Representative Tasks for Scrolling • We experimented with several tasks: • Scrolling while proofreading text for misspellings • Searching for highlighted line in document • [and following the link – Zhai] • Searching for highlighted target word in document, in presence of highlighted distracter words • Fitts’ task sensitive to subtle device diffs • Cognitive & visual search issues ignored