SATIN : S ketch A nd T ransformational IN frastructure

1. SATIN:Sketch And TransformationalINfrastructure Jason Hong UC Berkeley Group for User Interface Research Project Status Report - 07/22/1999

2. Motivation • More and more apps are using pens and some form of sketching • These apps are usually • built from scratch • not extensible or reusable • Pilot • Elec. Cocktail Napkin • Dynomite • Pegasus • FiloChat • Zenpad • XLibris • NotePals • Flatland • SILK • Tivoli • Kramer's Patches

3. Motivation • Architecture of windowing systems optimized for keyboard and mouse • Need to explore principled architectures for alternative input • Current state is analogous to windowing systems in the early 1980s

4. High-Level Research Goals • Create extensible toolkit supporting creation of pen and sketch-based apps • Explore architectures for pen-based applications • Explore new interaction techniques for using pens and for sketching • Explore new event models for other input devices

5. Overview • Motivation • Envisioned Applications • Features and Current Status • High-Level Architecture • Future Development • Feedback

6. Envisioned Applications • SILK • DENIM

7. Envisioned Applications • PosterBoard • Create posters using DigitalDesk

8. Envisioned Applications • PatchBoard • Multi-device whiteboard

9. How SATIN Fits Into the Big Picture Application The Required Berkeley CS Diagram SATIN Users may already have these parts or may have to download dynamically Swing Core Classes All versions of Java have these parts Java Virtual Machine

10. What SATIN Will Provide • Scenegraph and rendering support • GraphicalObjects • Objects with state, behavior, and view • State and behavior defined thru interpreters • Multiple views • Transformations • Per GraphicalObject, Rotate/Scale/Translate • Styles • Translucency, fill colors, line width, etc • Just set it - automatically used! • Can define styles in text files • Cascading Styles • e.g. selected objects or greyed-out objects Find

11. What SATIN Will Provide • Scenegraph and rendering support • Add, remove, group, move, etc • Pick GraphicalObject at location • Layering of objects • Automatic animation for smooth transitions • Semantic zoom

12. What SATIN Will Provide • Patches • Interpreted regions of space to help organize, process, and manipulate sketches • Step A • Step B • Step C

13. What SATIN Will Provide • Built-in support for Undo/Redo/Macros • Command pattern scheme • Automatic time-indexing of operations • Multiple streams and histories of events • Integration with Java Swing • Cut / copy / paste • Standardized Drag and Drop • Export as GIF / XML • Allow use of Java widgets • Be used as a Java widget

14. Current Status • Recognizers • Rubine's Recognizer • Interpreter • Graffiti Interpreter • Pie Menus • Performance • Minimal damage-redraw • Needs some more optimization though

15. Current Status • Basic Debugging Support • Debugging Window • Graphical Bounds • Properties file • System initializations • Sheet colors, location of data files, etc • Documentation • What you are reading • LOTS of Javadoc

16. Overview • Motivation • Envisioned Applications • Features and Current Status • Future Development • Feedback • High-Level Architecture

17. Short-term Future Plans • More Patch Operations • pull up, push down, dissolve, auto-arrange, • Multiple Views • Advanced Dispatching • "tolerant" dispatching • top-to-bottom dispatching • Streams and histories • Object histories or system histories? • Activators • If "is matrix" then load matrix interpreter

18. Feedback? • What do you need to make this work for you? • What do you like? What do you not like?

19. Overview • Motivation • Envisioned Applications • Features and Current Status • Future Development • Feedback • High-Level Architecture

20. GUIR Object Notation Vehicle Each instance of Car has four instances of Wheel The up arrow points to the parent class Semi Car Wheel 1 4 Pinto Gadget 1 n This color means classes already provided by Java JComponent Each instance of Pinto has n instances of Gadget. Each Gadget has a reference back to the Pinto it belongs to.

21. Simplified Overview of Class Architecture Interpreter Interpreter Interpreters Watchable Watcher GraphicalObject (GOb) GraphicalObject Collection Timed Stroke GraphicalObject Group JComponent Patch Sheet

22. Runtime Architecture Interpreter Interpreter Interpreter Commands Command Queue (Undo, Redo, Macros) Clipboard Selected GObs Dispatching Java Event Subsystem Stroke Assembler Sheet GraphicalObject Group Patch GraphicalObject (GOb) Command Objects Java2D Drawing Rendering Subsystem

23. GraphicalObjects in Depth Interpreter Interpreter Interpreters Cloneable Transform Serializable Style Shape GraphicalObject (GOb) Bounding Points (Polygon) Watchable Interpreter Mediator 1 n SingleStroke Listener GraphicalObject Group Progressive Stroke Listener

24. The Lifecycle of a Stroke (Current) Interpreter Interpreter Interpreter Mouse Events get forwarded here until the full stroke is completed. The Stroke Assembler then dispatches to the Sheet. Sheet onSingleStroke() Stroke Assembler 1 GraphicalObject Group Patch 2 onSingleStroke() Sheet determines if the stroke should be redispatched to another GraphicalObject 3 onSingleStroke() 4 GraphicalObject (GOb) Interpreter Mediator handleSingleStroke() The deepest object tries to handle the stroke first.

25. The Lifecycle of a Stroke (Current) Stroke Assembler GraphicalObjectGroup / Patch GraphicalObjectGroup / Patch Graphical Object Sheet mouseDragged() onSingleStroke() onSingleStroke() onSingleStroke() A onSingleStroke() A handleSingleStroke() A isHandled? B isHandled? B isHandled? A B dispatchee = getDispatchee() if (dispatchee != null) { isHandled = dispatchee.handleSingleStroke(); } if (!isHandled) { isHandled = handleSingleStroke();} return (isHandled) if (!isHandled) { isHandled = handleSingleStroke();} return (isHandled)

26. The Lifecycle of a Stroke (Future) Stroke Assembler GraphicalObjectGroup / Patch GraphicalObjectGroup / Patch Graphical Object Sheet mouseDragged() dispatch() process () dispatch () process () dispatch() process() dispatch() process() handle() handle() isHandled? • Every GraphicalObject has nine methods: • dispatchNewStroke() • processNewStroke() • handleNewStroke() • dispatchUpdateStroke() • processUpdateStroke() • handleUpdateStroke() • dispatchSingleStroke() • processSingleStroke() • handleSingleStroke() handle() isHandled? handle() isHandled?

27. GraphicalObject Coordinate Spaces • Every GraphicalObject has three coordinate spaces The top-left corner of this stroke is at (246,335) in absolute or screen coordinates. The top-left corner of this stroke is at (91,59) in relative or parent's coordinates The top-left corner of this stroke is at (0,0) in local or self coordinates In local coordinates, every GraphicalObject will have it's top-left corner at (0,0)

28. GraphicalObject Coordinate Spaces Here, we have two screen shots. The one on the left has no transform. The one on the right has a rotating transform on the Sheet. In absolute coordinates, the top-left corner of the Sheet is always (0,0). If you turn Graphical Debugging on, the coordinates displayed are absolute This is (0,0) for the Patch This is NOT (0,0) for the Patch. The transform was applied to the Sheet, changing absolute coordinates, but not anyone's local coordinates.

29. GraphicalObject Coordinate Spaces • How it works internally… Sheet Transform (txA) Absolute Transform = [txA][txB][txC] Relative Transform = [txC] Patch Transform (txB) Local Transform = [1] GraphicalObject Transform (txC) Absolute coordinate methods: getAbsoluteTransform() getAbsolutePosition() getAbsoluteBounds() getAbsoluteBoundingPoints() getAbsoluteWidth() getAbsoluteHeight() setAbsoluteBoundingPoints() Relative coordinate methods: getTransform() getPosition() getBounds() getBoundingPoints() getWidth() getHeight() setBoundingPoints() Local coordinate methods: getLocalTransform() getLocalPosition() getLocalBounds() getLocalBoundingPoints() getLocalWidth() getLocalHeight() setLocalBoundingPoints()

30. Recognizers and Interpreters in Depth Recognizers Progressive Recognizer Single-Stroke Recognizer Interpreter Rubine Recognizer Neural Net Recognizer Recognizer Recognizer Stroke Classification GraphicalObject (GOb) Stroke Classification App-Specific Code Handled? Internal Data Structures Interpreters Interactor Tree Probabilistic Tree Progressive Interpreter Single-Stroke Interpreter Graffiti Interpreter Internal Data Structure Helpers Rectilinear Interpreter Ink Clustering Algorithms (Spatial, Chronological) Recognition Data Struct(s)