1 / 47

quill: A UI Design Case Study

quill: A UI Design Case Study. A. Chris Long Postdoctoral Research Fellow HCI Institute. Administrivia. Turn in Homework #1 Homework #2 assignment on course website. Outline. Gestures quill Design quill Architecture & Implementation Object-Oriented Design

braden
Download Presentation

quill: A UI Design Case Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. quill: A UI Design Case Study A. Chris Long Postdoctoral Research Fellow HCI Institute

  2. Administrivia • Turn in Homework #1 • Homework #2 assignment on course website

  3. Outline • Gestures • quill Design • quill Architecture & Implementation • Object-Oriented Design • Long-running Tasks and Interfaces • Recognition-based UIs • How Gesture Recognition Works • More quill Implementation

  4. Gestures • Mark that invokes a command

  5. Gestures • Mark that invokes a command delete me

  6. Gestures • Mark that invokes a command • Benefits • Fast • Save screen space • Avoid “aerobic interface” • Specify operation and operand(s) simultaneously delete me

  7. Problems With Gestures • For users • Computer often misrecognizes gestures • Difficult to learn and remember gestures • For designers • Gestures are hard to design

  8. Who: Designer of pen-based UI When: During design phase What: Designing gestures Why: To design better gestures quill’s Place in the World Design Test Prototype

  9. User Interface Application “back end” OS Pen Input Display Another View of quill

  10. Designing quill • Design low-fi prototype • Pilot test low-fi • Redesign • Implement Java version

  11. Low-fi Prototype

  12. Current Version • Tree view • Expanded view • Suggestion for improvement

  13. Recognition Problem

  14. Gestures, Packages, and Groups, Oh My! Gesturepackage MyApplication Gesturegroup Edit View … Gesturecategory copy … cut Gesture …

  15. JDK Core Gesture Objects Gesture Gesture Category … Recognizer Gesture Analyses Widgets quill Architecture

  16. quill Architecture

  17. Java/Swing 93 classes, 26,000 lines Clipboard Desktop windows Tree display Main window Notice management Implementation Overview

  18. Gesture Objects Gesture GestureCategory GestureGroup GesturePackage Gesture Display Widgets GestureDisplay GestureInteractor GestureCategoryDisplay GestureGroupDisplay GesturePackageDisplay GestureCategoryThumbnailDisplay Object Oriented Design

  19. OO Design GestureObject GestureContainer DefaultGestureObject AbstractGestureContainer GestureCategory GestureGroup GesturePackage

  20. Analyzing Gestures • Problem: Analysis may take a long time • Solutions?

  21. Analyzing Gestures • Problem: Analysis may take a long time • Strategies for analyzing gestures • Run analysis in foreground, disable all actions • Analyze in background • Disable all conflicting actions • Allow anything, cancel analyses • Allow anything

  22. Multithreading • Benefits • ? • Drawbacks • ?

  23. Multithreading • Benefits • Computation in background • Can do very complex tasks • User can perform any action, any time • Drawbacks • Must synchronize data across threads • Deadlock • Race conditions

  24. Multiple Threads in Java • Thread class • Start separate thread that runs any method • synchronized keyword • Prevents a method from being called or object from being accessed by more than one thread at once • SwingWorker class • Easy to run background thread that produces a result

  25. Recognition-Based UIs • Examples • Pen gesture • Handwriting • Speech • Advantages • ? • Disadvantages • ?

  26. Recognition-Based UIs • Advantages • Natural input • Informal • Multiple interpretations • Disadvantages • No clear separation of data and commands • Ambiguous commands • Individual differences • Multiple interpretations

  27. Dealing with Recognition • Confirm dangerous operations • Multiple modes may help disambiguate • Speech + pen, for example • Provide undo/redo

  28. Training Measure geometric features of gestures Compute average feature values for each type of gesture Compute how important each feature is Recognition (a.k.a. Classification) Measure geometric features of unknown gesture Compare with features of known gestures Pick closest known gesture How Does Recognition Work?

  29. Rubine’s Feature Set • Initial angle • Length of bounding box diagonal • Angle of bounding box diagonal • Distance between first and last points • Angle between first and last points • Length • Total angle traversed

  30. Rubine’s Feature Set (cont.) • Sum of absolute value of angle at each point • Sum of squared angle at each point (sharpness) • Square of maximum speed • Duration

  31. Weakness of Feature-based Recognition • Feature set has to be right

  32. Feature-based Training • Mean feature vector • Per-class covariance matrix

  33. Training (cont.) • Average covariance matrix • Weights estimation

  34. Feature-based Classification • For all gesture classes c, compute • Return class with max vc

  35. quill Implementation Issues (cont’d) • Suggestions (cont’d) • When do they disappear? • User says to ignore • No longer apply • When are two notices the same? • At time A, gestures X and Y are too similar • At time B, something happens and they aren’t similar • At time C, they are similar again • Display notice again?

  36. Implementation Issues (cont’d) • Annotations • Special purpose vs. generic • How to compose? • Gesture display • Desktop windows • Tiled • Standard MDI • Display factory

  37. Implementation issues (cont’d) • Generic properties void GestureObject.setProperty(String name, Object value) Object GestureObject.getProperty(String name) • What happens to properties like “isMisrecognized” when copy & paste? • Persistent vs. transient

  38. Implementation issues (cont’d) • Generic properties void GestureObject.setProperty(String name, Object value) Object GestureObject.getProperty(String name) • What happens to properties like “isMisrecognized” when copy & paste? • Persistent vs. transient

  39. Implementation Issues (cont’d) • Menus: enable/disable items, and dispatching commands • Context-dependent menu items • Context-dependent gesture drawing • Solution: “Chain of command” pattern that follows selection

  40. Questions? • Other topics • Grad school • Berkeley (city or university) • Speech UIs • More on pen-based UIs

  41. The End Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides http://www.cs.cmu.edu/~chrisl/ http://guir.berkeley.edu/projects/quill/

  42. Extras

  43. Criticisms • Recognition is due to poor recognizers, which will eventually be perfect • Not for a long time, if ever • Learning and memory predictions still useful • Recognition will never be good enough • Already good enough for many users • Is improving over time • Typing is faster and more accurate than writing • Pens won’t replace keyboards, but are better in many situations

  44. Criticisms (cont.) • Gestures will always be too hard to learn and remember, and will never be useful enough • Gestures very often used on paper in many domains • Design tool doesn’t guarantee good gestures • Will provide useful advice • Will shorten design cycle • There are only a small number of common gestures • Application/domain-specific gestures • Personal shorthand

  45. Criticisms (cont.) • Marking menus are better than free-form gestures • Can’t specify additional information with marking-menus • Shapes are arbitrary • All straight lines and 45/90º angles

  46. Criticisms (cont.) • What’s wrong with traditional GUI interaction techniques? • Pen  mouse • Pen • Finer control • Has other dimensions (e.g., pressure, tilt) • Mouse • More buttons (sometimes)

  47. Pen vs. Mouse • Kato, et al • Pen significantly faster & more accurate than mouse for precise dragging • Pen significantly faster for pointing • Pen has directional dependencies; mouse doesn’t • Mackenzie, et al • Pen slightly faster for pointing • Mouse slightly faster for dragging

More Related