quill: A UI Design Case Study

1. quill: A UI Design Case Study A. Chris Long Postdoctoral Research Fellow HCI Institute

2. Administrivia • Turn in Homework #1 • Homework #2 assignment on course website

3. Outline • Gestures • quill Design • quill Architecture & Implementation • Object-Oriented Design • Long-running Tasks and Interfaces • Recognition-based UIs • How Gesture Recognition Works • More quill Implementation

4. Gestures • Mark that invokes a command

5. Gestures • Mark that invokes a command delete me

6. Gestures • Mark that invokes a command • Benefits • Fast • Save screen space • Avoid “aerobic interface” • Specify operation and operand(s) simultaneously delete me

7. Problems With Gestures • For users • Computer often misrecognizes gestures • Difficult to learn and remember gestures • For designers • Gestures are hard to design

8. Who: Designer of pen-based UI When: During design phase What: Designing gestures Why: To design better gestures quill’s Place in the World Design Test Prototype

9. User Interface Application “back end” OS Pen Input Display Another View of quill

10. Designing quill • Design low-fi prototype • Pilot test low-fi • Redesign • Implement Java version

11. Low-fi Prototype

12. Current Version • Tree view • Expanded view • Suggestion for improvement

13. Recognition Problem

14. Gestures, Packages, and Groups, Oh My! Gesturepackage MyApplication Gesturegroup Edit View … Gesturecategory copy … cut Gesture …

15. JDK Core Gesture Objects Gesture Gesture Category … Recognizer Gesture Analyses Widgets quill Architecture

16. quill Architecture

17. Java/Swing 93 classes, 26,000 lines Clipboard Desktop windows Tree display Main window Notice management Implementation Overview

18. Gesture Objects Gesture GestureCategory GestureGroup GesturePackage Gesture Display Widgets GestureDisplay GestureInteractor GestureCategoryDisplay GestureGroupDisplay GesturePackageDisplay GestureCategoryThumbnailDisplay Object Oriented Design

19. OO Design GestureObject GestureContainer DefaultGestureObject AbstractGestureContainer GestureCategory GestureGroup GesturePackage

20. Analyzing Gestures • Problem: Analysis may take a long time • Solutions?

21. Analyzing Gestures • Problem: Analysis may take a long time • Strategies for analyzing gestures • Run analysis in foreground, disable all actions • Analyze in background • Disable all conflicting actions • Allow anything, cancel analyses • Allow anything

22. Multithreading • Benefits • ? • Drawbacks • ?

23. Multithreading • Benefits • Computation in background • Can do very complex tasks • User can perform any action, any time • Drawbacks • Must synchronize data across threads • Deadlock • Race conditions

24. Multiple Threads in Java • Thread class • Start separate thread that runs any method • synchronized keyword • Prevents a method from being called or object from being accessed by more than one thread at once • SwingWorker class • Easy to run background thread that produces a result

25. Recognition-Based UIs • Examples • Pen gesture • Handwriting • Speech • Advantages • ? • Disadvantages • ?

26. Recognition-Based UIs • Advantages • Natural input • Informal • Multiple interpretations • Disadvantages • No clear separation of data and commands • Ambiguous commands • Individual differences • Multiple interpretations

27. Dealing with Recognition • Confirm dangerous operations • Multiple modes may help disambiguate • Speech + pen, for example • Provide undo/redo

28. Training Measure geometric features of gestures Compute average feature values for each type of gesture Compute how important each feature is Recognition (a.k.a. Classification) Measure geometric features of unknown gesture Compare with features of known gestures Pick closest known gesture How Does Recognition Work?

29. Rubine’s Feature Set • Initial angle • Length of bounding box diagonal • Angle of bounding box diagonal • Distance between first and last points • Angle between first and last points • Length • Total angle traversed

30. Rubine’s Feature Set (cont.) • Sum of absolute value of angle at each point • Sum of squared angle at each point (sharpness) • Square of maximum speed • Duration

31. Weakness of Feature-based Recognition • Feature set has to be right

32. Feature-based Training • Mean feature vector • Per-class covariance matrix

33. Training (cont.) • Average covariance matrix • Weights estimation

34. Feature-based Classification • For all gesture classes c, compute • Return class with max vc

35. quill Implementation Issues (cont’d) • Suggestions (cont’d) • When do they disappear? • User says to ignore • No longer apply • When are two notices the same? • At time A, gestures X and Y are too similar • At time B, something happens and they aren’t similar • At time C, they are similar again • Display notice again?

36. Implementation Issues (cont’d) • Annotations • Special purpose vs. generic • How to compose? • Gesture display • Desktop windows • Tiled • Standard MDI • Display factory

37. Implementation issues (cont’d) • Generic properties void GestureObject.setProperty(String name, Object value) Object GestureObject.getProperty(String name) • What happens to properties like “isMisrecognized” when copy & paste? • Persistent vs. transient

38. Implementation issues (cont’d) • Generic properties void GestureObject.setProperty(String name, Object value) Object GestureObject.getProperty(String name) • What happens to properties like “isMisrecognized” when copy & paste? • Persistent vs. transient

39. Implementation Issues (cont’d) • Menus: enable/disable items, and dispatching commands • Context-dependent menu items • Context-dependent gesture drawing • Solution: “Chain of command” pattern that follows selection

40. Questions? • Other topics • Grad school • Berkeley (city or university) • Speech UIs • More on pen-based UIs

41. The End Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides http://www.cs.cmu.edu/~chrisl/ http://guir.berkeley.edu/projects/quill/

42. Extras

43. Criticisms • Recognition is due to poor recognizers, which will eventually be perfect • Not for a long time, if ever • Learning and memory predictions still useful • Recognition will never be good enough • Already good enough for many users • Is improving over time • Typing is faster and more accurate than writing • Pens won’t replace keyboards, but are better in many situations

44. Criticisms (cont.) • Gestures will always be too hard to learn and remember, and will never be useful enough • Gestures very often used on paper in many domains • Design tool doesn’t guarantee good gestures • Will provide useful advice • Will shorten design cycle • There are only a small number of common gestures • Application/domain-specific gestures • Personal shorthand

45. Criticisms (cont.) • Marking menus are better than free-form gestures • Can’t specify additional information with marking-menus • Shapes are arbitrary • All straight lines and 45/90º angles

46. Criticisms (cont.) • What’s wrong with traditional GUI interaction techniques? • Pen  mouse • Pen • Finer control • Has other dimensions (e.g., pressure, tilt) • Mouse • More buttons (sometimes)

47. Pen vs. Mouse • Kato, et al • Pen significantly faster & more accurate than mouse for precise dragging • Pen significantly faster for pointing • Pen has directional dependencies; mouse doesn’t • Mackenzie, et al • Pen slightly faster for pointing • Mouse slightly faster for dragging