The subArctic Input System and Extensions for Handling Inputs with Ambiguity

1. The subArctic Input System and Extensions for Handling Inputs with Ambiguity

2. subArctic • A Java-based GUI toolkit that I (along with Ian Smith) built and distributed in 1996-97 • Goal: highly extensible allowing support for lots of cool new interaction techniques • Emphasis on making new and strange widgets / components / interactors easy to create • “High ceiling”

3. Parties involved with a toolkit • Toolkit designer (me) • Interactor designer • Interface programmer • User

4. Parties involved with a toolkit • Toolkit designer (me) • Interactor designer • Interface programmer • User Most toolkits target support here

5. Parties involved with a toolkit By moving work up (into reusable library) • Toolkit designer (me) • Interactor designer • Interface programmer • User

6. Parties involved with a toolkit • Toolkit designer (me) • Interactor designer • Interface programmer • User But typically don’t help much here (assume a fixed library)

7. subArctic • Toolkit designer (me) • Interactor designer • Interface programmer • User SA tries to move work for many kinds of interactors into toolkit infrastructure

8. subArctic • Toolkit designer (me) • Interactor designer • Interface programmer • User Input system is a big part of that SA tries to move work for many kinds of interactors into toolkit infrastructure

9. Schema for pretty much all GUIs init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); }

10. Event Record – recording of the relevant facts about some occurrence of interest (i.e., user has manipulated an input device) Schema of a GUI init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); }

11. Schema of a GUI Send (“dispatch”) the event to the object(s) that want it and/or know how to respond to it (e.g., widget/component/interactor) init(); for (;;) { evt = wait_for_next_event(); dispatch(evt); if ( damage_exists() ) redraw(); }

12. Event dispatch • All the work happens here • Typically delegated to interactors • E.g., buttons know how to respond to press and release like buttons should • Each object keeps track of its own state • ... but which interactor gets it • Toolkit “event dispatch” process

13. Event dispatch policies • Two primary ways to decide which interactor gets an event • What are they?

14. Event dispatch policies • Two primary ways to decide which interactor gets an event • Positional dispatch • Based on where mouse is pointing • Examples… • Focus-based dispatch • Designated object always gets input • Examples…

15. Pop quiz • Should input for dragging be dispatched via positional or focus?

16. Pop quiz • Should input for dragging be dispatched via positional or focus? Answer: No! (both)

17. subArctic input policies • subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects” • Manages bookkeeping (e.g., picking) • Extensible set • Turns out there are other useful policies (e.g., for modal dialogs)

18. Move Press Release When interactors get events… • … they typically respond to them with the equivalent of a simple finite state machine

19. subArctic has lib of common FSMs • Move a lot of input handling work typically done by interactor programmer up into the toolkit • One (highly parameterized) FSM for all • Brad’s “interactor” model (awful terminology :-) • Many customized FSM (extensible set) • subArctic input model

20. FSMs moved to toolkit object • “Dispatch agent” • Translates low level input into higher level terms

21. Move Press Release Dispatch agent example: move_drag • Translated to calls in input protocol: • drag_start(); • drag_feedback(); • drag_end(); • With useful parameters (e.g. new pos)

22. Move Defined by Java interface Press Release Dispatch agent example: move_drag • Translated to calls in input protocol: • drag_start(); • drag_feedback(); • drag_end(); • With useful parameters (e.g. new pos)

23. Set of dispatch agents is extensible • E.g., can subclass for specialized kinds of drag such as “drag_within_box” or “snap_drag” • Can create custom for one interface • Once created can reuse

24. Focus Policy Positional Policy Etc… Events Text Press Etc... Move drag Click Grow drag Rollover Etc... Etc... How it all goes together

25. How does interactor indicate it wants / can handle some type of input? • “… implements input_protocol” • Where “input_protocol” is interface with calls like drag_start(), etc. • For positional that’s it! • For focus-based must also ask for focus

26. Example: Hypertext for all • User (Ken Anderson) wanted to add hyperlinks to all objects • Hold down the control key and click • His external hyperlink database would take over and map interactor id to hyperlink target • But… how do you change every interactor to do this?

27. Example: Hypertext for all • In Swing, Motif, etc. this is essentially impossible • In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down • About 15 lines of code • Works for interactors written later!

28. Questions about the SA input system?

29. Providing Toolkit Level Support for Handling Ambiguity in Recognition-Based Input See: http://doi.acm.org/10.1145/354401.354407and http://doi.acm.org/10.1145/332040.332459 Jennifer Mankoff, Gregory Abowd Georgia Institute of Technology Scott HudsonCarnegie Mellon University

30. Motivation • Recognition-based input offers the promise of naturalistic input modalities, BUT…

31. Motivation • Recognition-based input offers the promise of naturalistic input modalities, BUT… • Recognizers are imperfect • affects users • breaks current system models • New interfaces & mechanisms

32. Example Interaction • From Newton • Handwritten text

33. Example Interaction • From Newton • Handwritten text automatically replaced with best recognition result test

34. Example Interaction • Double-tap to get a correction interactor test

35. text teat ted N-Best List Keyboard Character Correction Revert to Strokes Example Interaction • Correction interactor (mediator) test

36. Example Interaction • Works well, but… • Not reusable or customizable • Hard to grow your own • Basically we don’t have toolkit support for recognition based UI

37. Motivation (cont.) • At much the same stage we were at for GUIs in 1983 • No common model for input • No re-use • Infrastructure • “widget library”

38. Goals of This Work • Robust, reusable infrastructure • Reusable library • Integrate with convent. toolkit • Don’t throw out the baby with the bathwater

39. Talk Roadmap • Requirements for handling uncertain input • Extending toolkits to handle it • Interaction techniques for ambiguity • Implementation

40. Invoking Application Actions • Action often done by callbacks • Direct procedure call to application • Hierarchical events are alternate approach • Delivered to app as well as toolkit

41. down • • • drag • • • up stroke circle User Input Corresponding Events Hierarchical Events • Low-level events contribute to production of higher-level events [Green TOG ‘86; Myers & Kosbie CHI ‘96]

42. Implicit Assumption of Certainty • Implicit in all this is the assumption that the events really happened as reported • Problems arise when this isn’t true • E.g., brittle dialogs

43. Needed to Handle Uncertainty: • Allow for (and explicitly model) multiple alternatives • alternative higher level events • in recognition context: interpretations • Detect conflicting interpretations • Mediation of conflicts

44. Needed to Handle Uncertainty: • Lexical feedback about uncertain events • split “feedback” from “action” • Library of mediators

45. How do we do this...

46. down down • • • • • • drag drag • • • • • • up up stroke stroke box circle circle Certain Input Uncertain Input Extended Event Model • Uncertainty results in multiple interpretations  interpretation graph

47. Button Recog Menu Checkbox Toolkit Extensions • Toolkit’s job is still to deliver events to objects • Now delivered to recognizers, interactors, and application objects

48. Button Recog Menu Checkbox Toolkit Extensions • Toolkit’s job is still to deliver events to objects • Objects initially only produce (reversible) feedback, no actions

49. down • • • drag • • • up Interface Appearance stroke box circle Uncertain Event Hierarchy Another Change: • Events dispatched to all who might use it

50. Details: Arranging for Mediation • Identify any conflicts • Look for a mediators • Pluggable list of them in toolkit • Mediator chosen by meta-mediator • Mediator can: “Pass”, “Pause”, “Accept”