1 / 40

Informal Tools for Multimodal, Context-based User Interface Design

This article discusses the challenges of designing user interfaces for universal computing devices and proposes an approach using multimodal interaction and context-based design. It also highlights the importance of prototyping tools for designing effective user interfaces.

rashidap
Download Presentation

Informal Tools for Multimodal, Context-based User Interface Design

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. http://guir.berkeley.edu Informal Tools for Multimodal, Context-based User Interface Design James A. Landay July 7, 1999 HCC Retreat

  2. Natural Tides of Innovation ?? Innovation Universal Computing Integration Personal Computer Workstation Server Log R Minicomputer Mainframe 7/99 Time

  3. Universal 1 : including or covering all or a whole collectively or distributively without limit or exception 2 a : present or occurring everywhere b : existent or operative everywhere or under all conditions <universal cultural patterns> 3 a : embracing a major part or the greatest portion (as of mankind) <a universal state> <universal practices> b : comprehensively broad and versatile <a universal genius> 4 a : affirming or denying something of all members of a class or of all values of a variable b : denoting every member of a class <a universal term> 5 : adapted or adjustable to meet varied requirements (as of use, shape, or size)

  4. Specialized Devices Away From the “Average Device” • Powerful, personal capabilities from specialized devices • small, highly mobile or embedded in environment • “Intelligence” + immense storage and processing in the infrastructure • Everything connected Laptops, Desktops

  5. HCI Challenges • Universal computing devices will not have the same UI as “dad’s PC” • a wide range of devices • often w/ small or no screens & alternative I/O • e.g., pens, speech, vision • special purpose to particular applications • “information appliances” • lots of devices per user • all working in concert

  6. HCI Challenges (cont.) • Design of good appliances will be hard • single device design is easy • hard to design the same “application” in a consistent manner across many devices • e.g., calendar app.: one speech based & one GUI based • hard to make different devices work together • which device is used when? • multiple UIs & modes/device, which to “display”? • building awareness of context of use into design is the key to some of these issues • multimodal input is assumed, but little design support for creating multimodal interfaces

  7. Our Approach • Build • novel applications on existing appliances • e.g., NotePals on the Palm PDA & CrossPad • new information appliances • Evaluate appliances in realistic settings • Iterate • use the resulting experience to build • more interesting appliances • better design tools & analysis techniques

  8. Outline • HCI Challenges for Universal Computing • Multimodal interaction • Why is building MM UIs hard? • Best practices for designing GUIs • Proposed approach to building MM UIs • Using context to our advantage

  9. Multimodal Interaction • When communicating with people we use more than one mode at a time! • gesture & speak • sketch & speak • etc. • Computers would be easier to use and more useful if they also worked like this

  10. Benefits of Multimodal (MM) Interaction on Heterogeneous Devices • Computers could be used in • more situations • when hands are full or vision used for something else • more places • e.g., walking, car, factory floor, etc. • Interfaces would be easier to use • use innate perceptual, motor, & cognitive skills • More people would be able to use computers • including users w/ vision or motor impairments • MM UIs likely to become predominant

  11. Why is building MM UIs hard? • Often require “recognition” technology • speech, handwriting, sketches, gesture, etc. • Recognition technology is immature • finally “just good enough” • single mode toolkits just appearing now • no prototyping tools • Hard to combine recognition technologies • still requires experts to build systems • few toolkits or prototyping tools! • This was the state of GUIs in 1980

  12. Design Prototype Evaluate Best Practices for Designing GUIs • Iterative design • Prototyping tools are key to this success

  13. Early Stage UI Design • Brainstorming • put designs in a tangible form • consider different ideas rapidly • Incomplete designs • do not need to cover all cases • illustrate important examples • Present several designs to client or design team • No need at this stage for “coding”

  14. Prototyping Tools for Multimodal UI Design Should Support • Iterative design methodology • Informal techniques designers currently use in the early stage of UI design • sketching

  15. Prototyping Tools for Multimodal UI Design Should Support • Iterative design methodology • Informal techniques designers currently use in the early stage of UI design • sketching • storyboarding

  16. Landay Dictation Machine Prototyping Tools for Multimodal UI Design Should Support • Iterative design methodology • Informal techniques designers currently use in the early stage of UI design • sketching • storyboarding • “Wizard of Oz” |Um…. I’ll see you in the morning.

  17. Our Approach: Sketches, Models & Context-aware Design Tools • Infer models from design “sketches”or other informal representations • model is an abstraction of apps UI design • model for representing contexts & UI implications • Use models to • semi-automatically generate UIs on diverse platforms • dynamically adapt a particular appliance UI to changing context of use

  18. How to Specify Events • We have a good idea how to for visual UIs • visually! • But how about speech or gestures?

  19. Specifying Non-Visual Events • How do designers do this now? • speech • scripts or grammars (advanced designers only) • flowcharts on the whiteboard • “Wizard of Oz” -> fake it! • gestures • give an example & then tell programmer what it does • We can do the same by demonstration (PBD) • demonstrate example of act (e.g., speech) • demonstrate result • system infers program • just a prototype, so doesn’t have to be too general

  20. result of demonstrating a pen gesture for delete Specifying Non-Visual Events

  21. Combining the Visual & Non-Visual • How do you see what the system inferred? • necessary for editing • generate a visual representation • flowchart seems like a start (common in speech UIs) • appropriate? what should it look like?

  22. Type: User Stimulus Mode: Speech ARR_CITY Type: System Response Mode: Speech “Please name the departure city” Type: User Stimulus Mode: Speech DEP_CITY Type: Computation Lookup ARRIVAL_TIME in “flight times” using DEP_CITY, ARR_CITY A flowchart representing inferences made from the demonstration of a flight arrival time application Type: System Response Mode: Speech “The flight from DEP_CITY to ARR_CITY will arrive at ARRIVAL_TIME” Type: System Response Mode: Speech “DEP_CITY departures arriving to which city?” Specifying Non-Visual Events

  23. Combining the Visual & Non-Visual • How do you see what the system inferred? • necessary for editing • generate a visual representation • flowchart seems like a start (common in speech UIs) • appropriate? what should it look like? • Combining visual & non-visual events • e.g., end-user dragging truck while saying “fast” • use a visual language that combines visual storyboard of GUI w/ flowchart for non-visual • VL better be simple...

  24. Supporting Heterogeneous Devices • Consider sketches as an abstraction • Infer a “model” from the sketches • Use methods from model-based UI tools to • generate UIs for multiple devices • generate alternative modes for a single spec. on one device • Hard problems • how to abstract? • how do you generate a “good” UI? • keep the designer in the loop

  25. Take Advantage of Context: Monitor Environment & Actions to Improve Interaction • Which devices are present & available? • there is a wall display -> use it for my wearable • device discovery

  26. Take Advantage of Context: Monitor Environment & Actions to Improve Interaction • Which devices are present & available? • there is a wall display -> use it for my wearable • device discovery • What is occurring in the environment? • people are talking -> don’t rely on speech I/O • speech sensing • What is the state of the user? • hands using tools -> use speech I/O & visual Out • tangible tools or vision processing • Solution: UI design tools that understand context as well as multiple devices & modalities

  27. Design Goals • Let designer’s rapidly produce “rough cuts” • doesn’t need to handle all cases • Allow end user testing & fast modification • Generate code that can help start UI for multiple devices • designer adds more detail & improves interaction • programmers add necessary code

  28. What We’ve Accomplished So Far • Informal tools for UI design • sketch-based tools for GUI / Web design • built & tested 1st generation, building next gen. now

  29. What We’ve Accomplished So Far • Informal tools for UI design • sketch-based tools for GUI / Web design • built & tested 1st generation, building next gen. now • informal tool for speech UI design • designed & implementation in progress

  30. What We’ve Accomplished So Far • Informal tools for UI design • sketch-based tools for GUI / Web design • built & tested 1st generation, building next gen. now • informal tool for speech UI design • designed & implementation in progress • Automatic generation of simple control UIs • First cut designs for multimodal • UI design tool & appliance (SpeechCorder w/ ICSI) • Experience w/ appliances & simple context • NotePals

  31. Take Home Ideas • Universal Computing is about supporting people • Success will require the design & evaluation of new appliances (device + app + UI) that • take advantage of natural modes of input • especially multimodal input! • take advantage of context • are used in realistic settings • Experience, new architectures, and new tools will make this design problem easier

  32. http://guir.berkeley.edu Informal Tools for Multimodal, Context-based User Interface Design James A. Landay July 7, 1999 HCC Retreat

  33. Research Plan • Finish implementation of informal tools • study usage (especially of speech UI design) • use results to design multimodal design tool • Develop algorithms for extracting app model • Build context-aware applications w/o tools • two testbeds to create & study • wirelessly networked PDAs in classroom/learning • extraction of tacit context using social networking • build taxonomy of contexts • how they should effect UI?

  34. Research Plan (Cont.) • Implement tool for multimodal UI design • extract model & generates UI for 2 diverse platforms • uses simple context ques • Develop algorithms for capturing context • Evaluate usage (apps & tools) in target settings • Extend multimodal UI design tool • generate multi-platform UIs that dynamically adapt • allow context to be fully integrated in decisions

  35. HCI Goals of Universal Computing • Some of the roots of Universal Computing are in the ideas of Mark Weiser • Ubiquitous/Pervasive/Invisible/Calm Computing • “... does not live on a personal device of any sort [PDA or dynabook], but is in the woodwork everywhere.” • “you don’t want personal technology, you want personal relationships”

  36. HCI Goals of Universal Computing • Some of the roots of Universal Computing are in the ideas of Mark Weiser • Ubiquitous/Pervasive/Invisible/Calm Computing • “... does not live on a personal device of any sort [PDA or dynabook], but is in the woodwork everywhere.” • “you don’t want personal technology, you want personal relationships” • Universal Computing is about • supporting people’s tasks • most often includes working with other humans • making people’s lives easier • just creating ubiquitous technology does not solve this

  37. E-mail Reports Design ideas Presentations Computers Support Human-Human Communication (HHC)

  38. Traditional Software Interfaces • Force translations to formal representations • sometimes we want this (e.g., conference slides) • sometimes we don’t (e.g., creative tasks)

  39. “Put me in a room with a pad & a pencil and set me up against a hundred people with a hundred computers -- I’ll outcreate every goddamn sonofabitch in the room.” -- Ray Bradbury, Wired 6.10 Traditional Representations • Rigid and unambiguous • hard to mix (e.g., few tools support rough sketches) • warp perceptions of the viewer and user • Increase time • encourage precision • Inhibit creativity • “tunnel vision”

  40. Informal Communications Styles • Speaking • Writing • Gesturing • Sketching Informal UIs do not immediately translate natural input, allowing users to work more naturally

More Related