User Interface Research at Microsoft Mary Czerwinski

1. User Interface Researchat Microsoft Mary Czerwinski

2. Research Strategy Leverage Human Capabilities Exploit Technology Discontinuities Compelling Task: Information Access

3. Engaging Human Abilities • understand complexity • new classes of tasks • less effort communication motor perceptual cognitive Helps User

4. Data Mountain “Strongest cue ... relative size” Subject Layout of 100 Pages

5. Motivation: Task • Large information spaces are hard to navigate • Users develop Personal Information Spaces • Familiarity and customization • WWW  Favorites/Bookmarks • File System  Desktop icons • “Average” user

6. Motivation: Technology • Apply Information Visualization and 3D User Interface techniques • Aim to leverage human abilities • Spatial memory • 3D Perception • Pattern recognition • Spatial Memory is the focus • Does it work in Desktop 3D?

7. 3D UI Hypotheses • Advantages: • Easier to evoke spatial metaphor • Easier to engage spatial memory • Display more information • Disadvantages: • Need for camera manipulation • People get lost • Need for high-DOF input device?

8. A Sweet Spot? • Use 3D Visuals • Apply 3D cues • Occlusion, Relative Size, Shadows • Spatialized Audio • 2D Interaction technique • No egocentric navigation • Objects come to the user

9. Video

10. What’s New Here? • Space is topologically 2D, allowing 2D Interaction technique • Direct manipulation placement to tap spatial memory? • Deeper Processing • “Physical Objects”

11. Usability Study: Design • DM Pilot with 10 users • 22 users in the test • All experienced with IE4 • Half used IE, half the Data Mountain • 100 web pages from Top50 & roulette • Phases: Store, Re-organize, Retrieve • Incorrect retrievals not discouraged • Four cueing conditions

12. Cueing conditions

13. Pilot Results • Pilot essential for design iteration • Video showed post-pilot Data Mountain • Fixes: • Prevent page loss due to occlusion • Provide visual correspondence between title pop-up and thumbnail • Spatialize audio

14. Usability Study 1: Results • Organizing times comparable • Reliably faster (26%) retrieval with Data Mountain, averaged across all cues • DM as fast as IE or faster in all conditions • DM users had fewer incorrect retrievals • IE users did better with Title than All • DM users leveraged all information

19. Informal Observations • “Oh, that’s over there” • Layouts are very personal • Users invent & adjust categ’s as they go • Categories used very similar across users • Low cost of interrogation is important • Titles, Thumbnails, “Open Hierarchy” • Landmarks and 3D cues could be better

20. Grouping mechanism • Users did fine • without a grouping mechanism • without category labels • without multi-level hierarchy • Users grouped anyway • Some sub-grouping possible & seen • Some labeled with salient thumbnails

21. Open Questions • Relative contribution of Visual Cues vs. Spatial Memory? • Other ways to leverage deeper processing • Scalability • Manual placement means incremental use • Light-weight deeper hierarchy? • Multiple Data Mountains  mental overlay?

22. Study #2 • Better determine the contribution of spatial location memory • Examine longevity of memory for cued web pages in DM • Track user satisfaction

23. “Thumbless” Data Mountain

24. Study 2 Methods • Brought back old users after 6 months • Reran through retrieval phase of original study • Every other block of 10 trials, turned off the thumbnail images during retrieval • DVs: RT, incorrects, failures, subjective ratings, and cue rankings

29. Cue Ranking Results • On average, thumbnail images ranked most helpful (avg. = 1.8) • Mouse-over text (avg. = 2.0) next, then... • Spatial location of the web page (avg. = 2.2) • Spatialized audio feedback ranked least helpful cue (avg. = 4.0). • These differences were significant as determined by a Kruskall-Wallis nonparametric test, chi-square(3) = 24.02, p < .001.

30. Subjective Measures

31. Study 3: Visualizing Implicit Queries • 2 studies examined Implicit Queries’ effect: • on info management • on info retrieval • Visualized with Data Mountain • 3D, spatial layout of web pages • Pages similar to currently selected page highlighted

32. Implicit Queries (IQ): Performance Benefits • IQ users (IQ1 & 2) • built more categories • took longer organizing • faster finding pages • Retrieval far more frequent implies IQ faster overall IQ1,2 use different similarity metrics

33. Future Work • Multi-item dragging • Help users keep layout tidy • Web pages “live” • Continue researching “deeper processing” • Computer generated spatial layouts • Landscaped terrain--better landmarks

34. 3D UI Conclusions • Exploration of Desktop 3D “sweet spot” • Prototype already outperforms existing GUI in effectiveness • Usability test informs design evolution • System low on policy lets users choose an organization • Spatial memory does work in Desktop 3D

35. Camera-Based Head-Motion Parallax • Motion parallax is one of strongest 3D depth cues

36. Glances • Novel 3D navigation technique • Lightweight, ephemeral • Two-handed • Non-dominant hand used for glance gesture • Low-cost awareness of surrounding area

37. Toolspaces • Objects attached to virtual body • Glances used to get to toolspaces • Multiple uses • 3D widgets • Information store • Object transport

38. Video

39. 3D UI Timetable • 1/99: ~ 85% of new PC’s ship with 3D • 6/99: expect 100% will ship with 3D • Turn over installed base: • 5 years if only driven by games • 3 years if we have compelling non-game apps • Target: 2002 MS shipping 3D capable apps • 3D Shell + 3D Visualizations

40. Microsounds • Experimentally verified heuristics for creating sounds for human computer interaction • If you don’t have much to say, don’t take too long to say it. • How short can a sound be and still be distinguishable and meaningful? • The answer is <50ms • Sounds were distinguishable one from another • Coded intuitive negative-neutral-positive distinctions

41. Microsounds • Useful for: • Increasing user awareness of interface events and state without disrupting the user’s attention. • Visually impaired. • Next steps • Experimentally test coding power of pitch relationships in <80ms Microsounds • Use Microsound techniques to sonify the state of Web transactions

42. Audio UI Timetable (?) • Necessary hardware available now • Virtualize audio devices + system level mixing: 6 mo. to 1 yr • OS events available for sonification: 1 yr • Consistent use of audio in MS apps: 1 yr • Authoring for 3rd party: 2 yr

43. The TouchMouse • Sense hand contact via capacitance • Input modality for awareness • New events: Touch, Release • For Palm area • for button, wheel, thumb, … • Goal: use sensors to enhance & simplify UI • Zero activation force: Passive sensing • Flexible form factor: It’s paint. Sense touch on curved surfaces, tight spaces, moving parts, ...

44. Example: UI always up vs. maximum doc real estate Most widgets only useful if you’re holding mouse So fade in / out portions of display via touch Just use mouse the same way you always have Test users loved it: Easy & just does the right thing What’s a TouchMouse Good For?

45. Touch-sensing PsPC w/ touchpad Help overcome limited screen real estate Lightweight actions for efficient access to cmds Make better use of both hands Past successes: Sharing data, handedness, tilt Sensing enables new UI’s with better awareness of context and thus can potentially both simplify & enhance the user’s computing experience. Sensors on WinCE Palm-sized PC’s

46. Summary • Identify and engage human abilities • Exploit technology discontinuities • Result: easier access to more complexity • Result: dramatic increase in user base