blindSight: Eyes-free mobile phone interaction

1. blindSight: Eyes-free mobile phone interaction Kevin Li, University of California, San Diego Patrick Baudisch, Microsoft Research Ken Hinckley, Microsoft Research

2. blindSight “How about Monday morning?” calendar “Monday 9am” preview “tic, tic, sssssh” “Yeah, looks likeI’m free after 10”

3. blindSight is an application running on Microsoft Windows Smartphone is launched when user places or receive a call. It then replaces the in-call menu unlike the in-call menu, blindSight shows nothing on the screen, but speaks to the user

4. why?

5. PCs… PC screens have the users’ undivided attention  design for the visual channel

6. eyePhone

7. environment

8. visual impairment

9. screen-less device

10. can’t seescreen

13. Lots of information is stored on mobile phones…

14. … the interfaces are visual

15. survey “I need to access as part of a phone conversation:” # of participants

16. Ok, so let’s just translate all text from visual to auditory • “Menu: Press 1 to search contacts; press 2 to add a contact; press 3 to access your calendar…” • Wait, that sounds familiar

17. Please listen carefully as ouroptions have changed…

18. related work

19. interactive voice response • User’s should be able to “dial ahead” [Perugini et al.,CHI 2007] • Zap and Zoom allows users to jump to locations using shortcuts [Hornstein, UBILAB Rep 1994] • Use visual channel to inform users about options [Yin and Zhai, CHI 2006]

20. Interactingmid-conversation • Time compress audio[Dietz and Yerazunis, UIST 2001] • Integrate speech commands into the conversation [Lyons et al., CHI 2004]

21. blindSight’s auditory feedback

22. audio is heard only by the user,not by the person at the other end

23. rationale human-human conversation contains redundancy  people can recover from audio interruptions as long as interruption is short can we use this redundancy to injectauditory feedback from the device?

24. so how do we make sure device feedback fits into these time windows of low information content?

25. rules 1. feedback only on-demand home recordvoice addcontact calendar hear voice note findcontact hearemails heartask list hear textmessage speaker phone mute

28. rules type 6 “200 hits” type 2 “12 hits” type 7 “Marion” 2. brevity find contact 2 1 3 abc def 5 4 6 jkl ghi mno 8 7 9 pqrs tuv wxyz play delete next

29. rules (what if the content is a long list,such as appointments for a day?) 3. non-speech previews of composites calendar – + whereAmIgo today week week – + previewday day day – preview3 hours 3 hours 3 hours _ block½h ½ hour ½ hour ++

32. rules (what if the content is a long list,such as appointments for a day?) 4. decomposition – + whereAmIgo today week week – + previewday day day – preview3 hours 3 hours 3 hours _ next ½ hour ½ hour

33. rules 5. interruptability user interface runs as a separate thread

34. rules modes 2 2 1 1 3 3 5 5 4 4 6 6 8 8 7 7 9 9 action action delete delete save save 6. minimizemodes pick day start time end time tue mon wed … fri thu sat sun action delete save

35. rules 6. minimizemodes (avoid wizards) – whereAmIgo today + week week – + previewday day day – preview3 hours + 3 hours 3 hours _ + block½h ½ hour ½ hour ++

36. demo video (shows fast use by an experienced user)

39. hardware

43. space

44. epoxy dots enlarged spaces

45. error Flip Ear Visual

47. . 2 1 3 5 4 6 8 7 9 * 0 #

48. error Flip Ear Visual

49. blindSight evaluation

50. interfaces vs. BlindSight (eyes-free) Smartphone 2003 (sighted)