1 / 58

Programming Support for Natural Interaction

Programming Support for Natural Interaction. Jennifer Mankoff CoC & GVU Center Georgia Tech. Acknowledgements. Gregory Abowd & Scott Hudson FCE Group & GVU NSF & IBM. Natural Interaction. Mouse + Keyboard work great for killer apps of the GUI world Problems Arise…

damian
Download Presentation

Programming Support for Natural Interaction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Programming Support for Natural Interaction Jennifer Mankoff CoC & GVU Center Georgia Tech

  2. Acknowledgements • Gregory Abowd & Scott Hudson • FCE Group & GVU • NSF & IBM

  3. Natural Interaction • Mouse + Keyboard work great for killer apps of the GUI world • Problems Arise… • Off the desktop (Ubiquitous computing) • People with disabilities • My interest: Providing general support for interaction design in this space of problems

  4. Outline of Talk • Output • Augmenting physical objects and spaces • Input • Recognition in the face of errors • Motivation and Design Space • Contribution: Toolkit-level support • Validation: 2 Toolkits, Problem areas • Low bandwidth, ambiguous input

  5. Output: Augmenting physical objects and spaces • PALplates • User study • Exploration • Domisilica • Dual augmentation • General infrastructure

  6. PALplates (CHI’97) (FX-Pal) • Based on study of paper prototype • Point of need • Dynamic UI • Identity • Location

  7. Domisilica • Dual Augmentation • Augment virtual world with information about real world state • Augment real world with virtual services, info (recipes, notes) • General Infrastructure • Example app: Cyberfridge (CVE’98)

  8. Contents: an Instant Jello Pudding:French Vanilla an Instant Jello Pudding:French Vanilla a 16oz Ferrara Capellini-Angel Hair a Hamburger Helper: Beef Noodle ...

  9. Outline of Talk • Output • Augmenting physical objects and spaces • General support for exploring new domains • Input

  10. Input • Recognition • Motivation and Design Space • Contribution: Toolkit-level support • Validation: 2 Toolkits, Problem areas • Low Bandwidth, Ambiguous Input

  11. Recognition • Recognition is becoming ubiquitous • Recognition is difficult to use • A range of interface problems result • OOPS toolkit helps solve them

  12. Research Method • Evaluators • Study individual problems/solutions • Design space of possible solutions • Builders • Facilitate solutions to sets of problems (Architectural / toolkit solutions) • Design space exploration by a larger community

  13. Input • Recognition • Motivation and Design Space • Contribution: Toolkit-level support • Validation: 2 Toolkits, Problem areas • Low Bandwidth, Ambiguous Input

  14. Definitions • Recognizer • interprets user input • creates ambiguity • Error • mistake from user’s perspective • represented with ambiguity • Mediation • dialogue between user and computer • used for resolving ambiguity

  15. SILK (Landay, 1996)

  16. Burlap

  17. Multimodal Suhm (98) McGee et al (98) Handwriting Huerst, Yang & Waibel (98) Speech Ainsworth & Pratt (92) Baber & Hone (93) IUIs Horvitz (99) Many Others… Design Space of Mediation Techniques

  18. Design Space of Mediation Techniques • Two major classes • Repetition • Choice

  19. Design Space of Mediation Techniques • Two major classes • Repetition • Choice

  20. Input • Motivation: Alternatives to Keyboard/Mouse • Recognition • Motivation and Design Space • Contribution: Toolkit-level support • Validation: 2 Toolkits, Problem areas • Low Bandwidth, Ambiguous Input

  21. OOPS Toolkit (CHI’00) • Toolkit-level support for handling ambiguity in recognition • Library of mediators • Architectural support • Based on subArctic

  22. Architectural Support • INDEPENDENT of any specific toolkit • Separation of mediators, recognizers, and application • Communication by a common internal model (ambiguous hierarchical events) • Maintains ambiguity indefinitely

  23. GUI Garnet/interactors (Myers ‘90) X toolkits (Rosenberg et al. ’90) GUI+Recognition SATIN (Hong et al ’00) Extended Amulet (Landay & Myers ’93) PPSM (Hudson&Newell ’92) Artkit (Henry et al. ’90) Context Context Toolkit (Dey et al ’00) PARCPad, etc (Schilit et al. ’94) Multimodal OAA (Cohen et al ’94/Oviatt ’99) PAC-Amodeus (Nigay& Coutaz ’95) Related Work

  24. Three key pieces • Ambiguous hierarchical events • Mediation subsystem • Changes to event dispatch

  25. down down drag drag up up • • • • • • • • • • • • stroke stroke s c Ambiguous Hierarchical Events Myers & Kosbie ‘96 s

  26. Mediation Subsystem • Ambiguity is identified automatically • Presence of multiple interactor nodes • Hierarchy is passed to mediators • Recognizers, recipients informed of accept/reject decisions • Accept/reject modifies hierarchy • Application selects mediators from library

  27. Event Dispatch • Problem: all recipients must support ambiguity • A sensed event arrives • It is dispatched to all recognizers It is mediated rec1 rec2 Input handler event rec3 rec4 recn med2 med1 med4 med3 medn

  28. Modified Event Dispatch 1. A sensed event arrives 2a. It is dispatched to ambiguity unaware components 2b. It is dispatched to all recognizers • It is mediated • Any accepted leaf nodes are dispatched to ambiguity unaware components

  29. s c stroke stroke s c Comparison 1990’s GUI OOPS recogs mediators application application recognizers meds inter interactors Input handler Input handler

  30. Input • Recognition • Motivation and Design Space • Contribution: Toolkit-level support • Validation: 2 Toolkits, Problem areas • Low Bandwidth, Ambiguous Input

  31. Validation • Doing what’s been done • Burlap (complex app) • ~15 Mediators (based on design space) • Architecture: 2 Toolkits • “direct” hookup in GUI world • Context Toolkit extensions (Dey) • Making new things possible: 4 problem areas

  32. Problem Areas (UIST ’00) • Errors & Ambiguity • rejection errors • target ambiguity • Mediation • adding alternatives • occlusion

  33. Problem: There may be multiple targets of a user action Example: clicking Kabbash (95) Worden (97) Target Ambiguity

  34. Problem: There may be multiple targets of a user action Example: Clicking Solution: Give the user a choice of all of the targets Target Ambiguity

  35. Problem: There may be multiple targets of a user action Example: Clicking Solution: Give the user a choice of all of the targets Analysis: Any interface involving mouse press/release Requires separation of concerns Works with all interactors Target Ambiguity

  36. Problem: The correct choice isn’t always present Adding alternatives

  37. Problem: The correct choice isn’t always present Example: word-prediction Adding alternatives

  38. Problem: The correct choice isn’t always present Example: word-prediction Solution: Allow the user to add choices Adding alternatives

  39. Problem: The correct choice isn’t always present Example: word-prediction Solution: Allow the user to add choices Analysis: Closely related choices (e.g. URL prediction) Requires extensible mediators Benefits from recognizer API Adding alternatives

  40. Contributions (Recognition) • Resolution of ambiguity in recognition through mediation • Architectural support (2 toolkits) • Exploration of domain of mediators

  41. Future Work (Recognition) • Testing • Implicit Input (Current) • Intelligent Mediation • “Lazy” alternative generation • Meta-intelligence • General support for varied inputs

  42. Input • Recognition • Motivation and Design Space • Contribution: Toolkit-level support • Validation: 2 Toolkits, Problem areas • Low Bandwidth, Ambiguous Input

  43. Brain-computer interface (Current) • Locked-in Syndrome • Neural Mouse Control

  44. Low-Bandwidth Input • No alternative modalities available • Need for efficient error handling • Challenge: interpret brain signal in as rich a form as possible

  45. Automatic Conversion for Low Bandwidth Input • Web Browser Example: • Automatic URLextraction • Preview Area • Generalization –Next step for natural interaction

  46. Conclusions • Interaction design for ubiquity, disability • Dual augmentation • Ambiguity in recognition • Low bandwidth or error-prone input • Exploration of new interaction techniques • Four mediation problems • Low Bandwidth input • General, re-usable solutions • Domisilica • OOPS toolkit • Extended Context Toolkit

  47. Further Information http://www.cc.gatech.edu/fce/errata/ jmankoff@cc.gatech.edu

  48. What about Alternative Explosion? • Fairly independent, discrete interactions • Some stuff stays in recognizers • Recognition API helps • Only cache ambiguous events • Work to be done…

  49. How general is “general”? • Two Toolkits • Two complex applications • Library of existing mediators • Explorations of 4 problem areas (UIST 00)

  50. Is subArctic doing the work here? • No, our minimal requirements are common in today’s toolkits: • An event-based toolkit • An input-handling module that delivers events to the appropriate places • A library of interactors/widgets • Access to source code (OOPS is not just a library!)

More Related