Interaction Techniques for Ambiguity Resolution in Recognition-based Interfaces

1. Interaction Techniques for Ambiguity Resolution in Recognition-based Interfaces Jennifer Mankoff CoC & GVU Center Georgia Tech

2. Acknowledgements • Gregory Abowd & Scott Hudson • FCE Group & GVU • NSF

3. Outline • Motivation • Definitions & Illustration • Broad Solution: OOPS • Specific Solutions • Conclusion & Future Work

4. Ambiguity • am·big·u·ous1 a: doubtful or uncertain especially from obscurity or indistinctness <eyes of an ambiguous color> b: INEXPLICABLE2: capable of being understood in two or more possible senses or ways • An anathema to computers • Normal for humans

5. Where does ambiguity arise? • Web search • user doesn’t know correct answer • multiple correct answers • Implicit input • user not involved with application • Multiple users • System states may not agree • Recognition

6. Focus: Recognition • Recognition is becoming ubiquitous • Recognition is difficult to use • A range of interface problems result • OOPS toolkit helps solve them

7. Research Methodology • Motivated by real world, non-CS problems • Evaluators • Study individual problems/solutions • Design space of possible solutions • Builders • Facilitate solutions to sets of problems (Architectural / toolkit solutions) • Design space exploration

8. Outline • Motivation • Definitions & Illustration • Broad Solution: OOPS • Specific Solutions • Conclusions & Future Work

9. Definitions • Mediation • dialogue between user and computer • used for resolving ambiguity • Recognizer • interprets user input • creates ambiguity • Error • mistake from user’s perspective • represented with ambiguity

10. SILK (Landay, 1996)

11. Burlap

12. Outline • Motivation • Definitions & Illustration • Broad Solution: OOPS • Specific Solutions • Conclusions & Future Work

13. OOPS Toolkit (CHI’00) • Toolkit-level support for handling ambiguity in recognition • Library of mediators • Architectural support • Based on subArctic

14. Library of mediators • Design space based on survey • Generic and re-usable • Three major classes • Repetition • Choice • Automatic

15. Library of mediators • Design space based on survey • Generic and re-usable • Three major classes • Repetition • Choice • Automatic

16. Library of mediators • Design space based on survey • Generic and re-usable • Three major classes • Repetition • Choice • Automatic if (result is “W.”) reject it else do nothing

17. Architectural Support • INDEPENDENT of any specific toolkit • Separation of mediators, recognizers, and application • Communication by a common internal model (ambiguous hierarchical events) • Maintains ambiguity indefinitely

18. Three key pieces • Ambiguous hierarchical events • Changes to event dispatch • Mediation subsystem

19. down down drag drag up up • • • • • • • • • • • • stroke stroke s c Ambiguous Hierarchical Events s

20. Event Dispatch • A sensed event arrives • It is dispatched to all recognizers It is mediated rec1 rec2 Input handler event rec3 rec4 recn med2 med1 med4 med3 medn

21. Mediation Subsystem • Ambiguity is identified automatically • Presence of multiple interactor leaf nodes • Hierarchy is passed to mediators • Recognizers, recipients informed of accept/reject decisions • Accept/reject modifies hierarchy • Application selects mediators from library

22. Outline • Motivation • Definitions & Illustration • Broad Solution: OOPS • Specific Solutions • Conclusions & Future Work

23. Problem Areas • Errors & Ambiguity • rejection errors • target ambiguity • Mediation • adding alternatives • occlusion

24. Problem: The user’s input is completely missed Rejection Errors

25. Problem: The user’s input is completely missed Solution: Allow the user to tell the system Rejection Errors

26. Problem: The user’s input is completely missed Solution: Allow the user to tell the system Other applications: Substitution errors Rejection Errors

27. Problem: The user’s input is completely missed Solution: Allow the user to tell the system Other applications: Substitution errors Any spatial recognition Requires extended recognizer API Rejection Errors

28. Problem: There may be multiple targets of a user action Example: clicking Target Ambiguity

29. Problem: There may be multiple targets of a user action Example: Clicking Solution: Give the user a choice of all of the targets Target Ambiguity

30. Problem: There may be multiple targets of a user action Example: Clicking Solution: Give the user a choice of all of the targets Other applications: Any interface involving mouse press/release Requires separation of concerns Works with all interactors Target Ambiguity

31. Problem: A mediator may obscure important information Occlusion

32. Problem: A mediator may obscure important information Solution: Move that information into a more visible location Occlusion

33. Problem: A mediator may obscure important information Solution: Move that information into a more visible location Other applications: Any crowded interface that uses an n-best list Requires extensible mediators Requires separation of concerns Occlusion

34. Problem: The correct choice isn’t always present Adding alternatives

35. Problem: The correct choice isn’t always present Example: word-prediction Adding alternatives

36. Problem: The correct choice isn’t always present Example: word-prediction Solution: Allow the user to add choices Adding alternatives

37. Problem: The correct choice isn’t always present Example: word-prediction Solution: Allow the user to add choices Other applications: Closely related choices (e.g. URL prediction) Requires extensible mediators Benefits from recognizer API Adding alternatives

38. Outline • Motivation • Definitions & Illustration • Broad Solution: OOPS • Specific Solutions • Conclusions & Future Work

39. Conclusions • Resolution of ambiguity in recognition through mediation • General toolkit architecture (CHI 00) • flexible, re-usable support for mediation • separates recognition, mediation, and applications • allows exploration of design space

40. Next Steps • Implicit input • Sensed information about environment • Ambiguous output • Ambient displays • Brain-computer interface • Ambiguous, limited, error-prone input

41. Locked-In Syndrome • Disease, stroke, accident survivors • Completely paraylzed • Unable to speak • Cognitively intact 500,000 people worldwide

42. Brain-computer interface • Problem: locked-in syndrome • No alternative modalities available • Need for efficient error handling • Challenge: interpret brain signal in as rich a form as possible

43. A new hope • Brain signals can be intercepted • Implanted electrodes (Schwartz, Chapin et al.) • External sensors (Junker, Wolpaw, Middendorf, Birbaumer et al., Spencer et al.) • Signals can be produced and controlled by imagined movements • Signals can be interpreted by a computer

44. A Neurotrophic Electrode • Cone electrode invented in 1987 • Animal studies showed it to be stable • FDA permission given for human implantation in 1996

45. Project Goals • High level: • Recreate movement • Restore communication • Turn disability into ability • Low level: • Intelligent Interpretation

46. Recreating movement • Haptic feedback • Muscle stimulation • Eventually re-connect nerves

47. Restore Communication • A basic need • From virtual keyboards to word-prediction

48. Turning disability into ability:Supporting Creativity • Converting a the brain signal to music • Example • Two possible experiences • Trying to play a piano with ones feet • Having an artistic voice no one else can reproduce

49. Intelligent Interpretation • Signals are difficult to control • Daily variability in signal • Patient Endurance • Adaption necessary

50. Intelligent Interpretation • Raw signal-> mouse • Logical control • Neural gestures