1 / 48

Jason Hong and James Landay University of California Berkeley Group for User Interface Research

Jason Hong and James Landay University of California Berkeley Group for User Interface Research. Motivation. Sketching is useful in many settings. Survey of Informal Ink Apps. Whiteboard Tivoli Flatland. Creative / Design SILK Music Notepad Knight Elec. Cocktail Napkin Teddy

aviva
Download Presentation

Jason Hong and James Landay University of California Berkeley Group for User Interface Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Jason Hong and James Landay University of California Berkeley Group for User Interface Research

  2. Motivation • Sketching is useful in many settings

  3. Survey of Informal Ink Apps • Whiteboard • Tivoli • Flatland • Creative / Design • SILK • Music Notepad • Knight • Elec. Cocktail Napkin • Teddy • PatchWork • Capture • ZenPad • NotePals • Dynomite • FiloChat • Others • Pegasus • XLibris • PerSketch • Kramer's Patches

  4. Example - Elec. Cocktail Napkin Gross and Do (UIST96)

  5. Example - Flatland Igarashi, Edwards, LaMarca, and Mynatt (AVI2000)

  6. Common Features • Pen input as ink or gesture • Stroke is pen input from dragging • Ink is a stroke that stays on-screen • Gesture is a stroke that activates a command • Recognize ink as objects • Transformation / clean-up of ink • Immediate / deferred processing of ink erase me

  7. Common Features • Contain objects in addition to ink • Selecting and moving objects • Grouping of objects • Layering of objects

  8. Related Work • Commercial Software Support • Newton OS • PenPoint • Microsoft Windows for Pen Computing • Windows CE • Palm OS • Problems • Form-based and handwriting interfaces • Not easily extensible for building informal ink apps

  9. Related Work • Research Software Support • ArtKit (Henry et. al. 1990) • Garnet Pen Int. (Landay et. al. 1993) • Patches (Kramer, 1994) • OOPS (Mankoff et. al. 2000) • Problems • Need more reusable libraries for handling and processing strokes • Need more extensibility so lots of kinds of informal ink apps can be built

  10. Problems • Software infrastructure support for informal ink apps rudimentary • Pen-based interaction techniques difficult to implement • Similar to GUI development in 1980s • Lots of applications + interaction techniques • Need cohesive frameworks + toolkits

  11. Goals of SATIN • Toolkit support for informal ink apps • Provide reusable mechanisms for handling + processing strokes • Separate mechanism from policy • Be extensible for new kinds of apps • Reusable widgets for pens • Distribute this toolkit for use

  12. Overview • Motivation • Survey of Informal Ink Applications • Two Applications Built with SATIN • SATIN Architecture Overview • Conclusions

  13. DENIM

  14. SketchySPICE

  15. SketchySPICE

  16. SketchySPICE

  17. Some Metrics

  18. Overview • Motivation • Survey of Informal Ink Applications • Two Applications Built with SATIN • SATIN Architecture Overview • Conclusions

  19. Architecture Overview Application SATIN relies on Java2D and Swing Applications use SATIN, can use Java2D and Swing SATIN Swing Java2D Java Core Classes Java Virtual Machine

  20. Architecture Overview SATIN can be broken into twelve concepts Recognizers Views Transitions Interpreters Scenegraph Rendering Stroke Libraries Events Notification Widgets Command Clipboard

  21. Recognizers 0.86 0.65 0.63 Cut Copy Undo • Problem - many recognition techniques • Recognizers let new recognition technologies be plugged into SATIN • Given a stroke, return n-best list • No actions taken • SATIN comes with Rubine's Recognizer (implemented by Chris Long) Recognizer

  22. Interpreters erase me • Problem – many ways of handling strokes • Interpreters let new ways of handling strokes be plugged into SATIN • e.g. straighten a line • e.g. issue a command Ink Interpreter Gesture Interpreter

  23. Interpreters and Recognizers 0.86 0.65 0.63 Cut Copy Undo • Decoupled recognition (Recognizers) from actions (Interpreters) • Interpreters can use recognizers • Both are modules that can be plugged into apps built with SATIN Interpreter Recognizer Cut

  24. Multi-Interpreters • Problem – Hard to combine and extend existing interpreters • Multi-Interpreters are a collection of interpreters + dispatch policy • e.g. dispatch to chain of interpreters • e.g. disable some interpreters on context … Default Multi Interpreter Intrp A Intrp B Intrp Z … Semantic Zoom Multi Interpreter Intrp A Intrp B Intrp Z

  25. Scenegraph Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke …

  26. Stroke Dispatching • Needed an approach that enabled: • Individual objects handle strokes on own • Priority to gestures over ink • Default policy is top-down dispatch • Process stroke with its gesture interpreter • Re-dispatch stroke to one of its children • Process stroke with its ink interpreter • Handle stroke in object itself

  27. Stroke Dispatching Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

  28. Stroke Dispatching • 1. Process stroke with gesture interpreter Sheet Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke … Default Multi Interpreter Hold Select Intrp Standard Gesture Intrp

  29. Stroke Dispatching Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

  30. Stroke Dispatching • 1. Process stroke with gesture interpreter Sheet Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke … Default Multi Interpreter Hold Select Intrp Standard Gesture Intrp

  31. Stroke Dispatching • 2. Re-dispatch stroke to children Sheet Denim Label (Patch) Denim Sketch (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

  32. Stroke Dispatching • 1. Process stroke with gesture interpreter Sheet Denim Label (Patch) Denim Sketch (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

  33. Stroke Dispatching • 2. Re-dispatch stroke to children Sheet Denim Label (Patch) Denim Sketch (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

  34. Stroke Dispatching • 3. Process stroke with ink interpreter Sheet Denim Label (Patch) Denim Sketch (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke Phrase Interpreter

  35. Stroke Dispatching • 4. Handle stroke in object Sheet Denim Label (Patch) Denim Sketch (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

  36. Stroke Dispatching Sheet Denim Label (Patch) Denim Sketch (Patch) Phrase (Patch) Timed Stroke

  37. Example - Cut Shallow

  38. Example - Cut Deep

  39. Reusable Stroke Libraries Straighten Merge

  40. Reusable Stroke Libraries Split Simplify

  41. Widgets • Pie Menus • Pen PLAF • Swing "Pluggable Look And Feel" • Larger and clickable sliders • Single-click file opener

  42. Conclusions • SATIN - Java toolkit for informal ink apps • Extensible thru Recognizers, Interpreters, and Multi-Interpreters • Separates mechanism from policy for strokes • Offers reusable libraries for manipulating ink • Reusable widgets for pens

  43. Download SATIN Download SATIN at http://guir.berkeley.edu/projects/satin http://guir.berkeley.edu

  44. Extra Slides

  45. DENIM

  46. DENIM

  47. Survey of Informal Ink Apps • Elec. Cocktail Napkin • Flatland • Pegasus • FiloChat • Zenpad • PerSketch • Teddy • PatchWork • XLibris • NotePals • Dynomite • SILK • Tivoli • Kramer's Patches • Music Notepad • Knight

  48. Architecture Overview Recognizers Interpreters Scenegraph Stroke Libraries Widgets Views SATIN can be broken into twelve concepts • Rendering • Transitions • Clipboard • Notifications • Commands • Events

More Related