Mobile Devices for Control

1. Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Mobile Devices for Control Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles

2. Mobile Devices • Mobile Devices for communication • As a “Personal Information Manager” • As a “Personal Digital Assistant” • As a PC replacement • “PocketPC” • For Remote Control of other devices • Remote Control of PCs • Remote Control of appliances

3. Problem • Appliances are too complex

4. Problem • Too many remotes

5. Problem • User Interfaces of Mobile Devices themselves are too hard to use

6. Problem • PC’s have only a keyboard and mouse as input devices

7. Approach • Use a personal Mobile Device as an Interface to the PC and to the Appliances

8. History • ParcTab – continuous communication • Early “organizers” – no communication • Newton – extra cost for synchronization • Palm – HotSync once a day • WindowsCE/PocketPC – ActiveSync once a day

9. Now, Handhelds will be Communicating Toshiba e740 • 802.11 (“Wi-Fi”) • BlueTooth • Cell-phone network (G3) • (Infrared) • (Wires or cradle) HP iPaq 3870 Microsoft SmartPhone

10. Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be inclose, interactive communication.” • Brad Myers, “Using Hand-Held Devices andPCs Together,” Comm. ACM,Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.

11. Pebbles Project • Performed as part of the Pebbles Project • Overall goal: Use of multiple devicesat the same time • Palm and desktop PC • Mobile Phone and “Smart Home” • Multiple handhelds in a meeting • Pocket PC and appliances • Multiple users with their devices • Single user with multiple devices

12. Pebbles is: P E B B L E S DAs for ntry of oth ytes and ocations from xternal ources. http://www.pebbles.hcii.edu/

13. Controlling a PC Remote Commander Slide Show Commander “Semantic Snarfing” Scrolling Shortcutter http://www.pebbles.hcii.edu/

14. Important Work by Others • PARCTab [Want 95], [Weiser, CACM’93] • Abowd: eClass using Handhelds [CHI’98] • Greenberg: public and private data[Personal Technologies, 3(1), 1999] • Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …] • Pick and drop;handhelds withbig displays

15. RemoteCommander • Allow PDAs to control a PC • Can be used with any application • Uses the standard (single) cursor • Don’t have to jump up andgrab mouse • All mouse and keyboard functions • Use PDA like touchpad • Graffiti or our own pop-upkeyboard • Word prediction and completion

16. For People with Muscular Disabilities • Using handhelds as interface to PCs • People with Muscular Dystrophyhave fine-motor control but losegross motor control • Difficulties with mouse and keyboard,but stylus OK • So use Remote Commanderas PC’s keyboard and mouse

17. PocketPC version • Get PC’s screen onto PocketPC • Full view, or one-to-one zooming • Scroll with iPaq’s buttons • Control or scribble

18. Power Point Control • Use PC to give the presentation • Use hand-held to control the PC • Two-way communication • Hand-held shows picture of slide,notes, timer

19. SlideShow Commander, cont. • See list of slide titles • Scribble • Tap on on-screen buttonsand links • Preview otherslides • Controldemonstrations

20. Control at a Distance • Controlling a PC fromacross the room • Meetings, etc. • Use a laser pointer? • Studies • Difficult due to: • Jitter of hand (+/- 10 pixels) • No button on the laser pointer • People not accurate at wherebeam will go on and off

21. Idea:“Semantic Snarfing” • Grab contents to handheld • Picture • “Semantic” because reformat based on the meaning of the contents • Menus • Text • Re-visualization

22. Command Post of the Future • Large DARPA funded project • Make commanders more effective • Private drill-down of public information

23. Classroom Investigations • Investigate improving large lectures with frequent in-class “concept tests” • Provide immediate feedback to instructor • Easy for instructor

24. Scrolling Desktop Applications • Scrolling using the non-dominant hand • Parallel and efficient use of both hands • Generates Windowsscrolling events • Study showed can befaster for some tasks

25. Shortcutter • User-created panels of controls • Create custom interfaces and extensions to PC applications • And then take them with you • Direct manipulationfor edit, then setproperties • PocketPC or Palm

26. Shortcutter Widgets • Buttons • Virtual Joy stick • Virtual Knob • Sliders • Mouse pad • Gesture panel

27. Shortcutter Actions • Send any keyboard key, mouse button, scrolling action or string to PC • Open a file or URL • Run an application • Invoke any PC menu or button • Windows message • Recorded • Switch to a different Shortcutter panel • Control the Mouse

28. Shortcutter Actions, cont. • Control external devicesthrough PC’s serial port • Directly (e.g., projectors) • X-10 for electrical devices • Macro • Can be multi-application • Application-specific • Same button, different messages • Useful for application sets:browsers, compilers

29. More Scenarios of Use • Lean-back mail reading • Controlling media players • … and many others

30. Controlling Appliances PhD research of Jeffrey Nichols http://www.pebbles.hcii.edu/puc/

31. Problem • Too many complex devices, eachwith its own idiosyncratic interface • Stereo system • Telephones • ATM • Fax machine • Photocopier • Hotel alarm clock • Increasingly computerized • Low usability

32. Problem

33. Existing “universal” controls • Pre-programmed at the factory with a subset, • Or, Laboriously hand-programmed by the user

34. Important Work by Others • Xweb [Olsen Jr. 2000] • INCITS V2 standardization effort: Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI’02] • Stanford iRoom, iCrafter[Ponnekanti, UBICOMP 2001]

35. Our Approach • Handheld is a “Personal Universal Controller” (PUC) • Two-way communication • Appliance describes its functions • Handheld PUC • Automaticallycreates interface • Controls the appliance • Displays feedback about appliance status Specifications Control Feedback

36. Automatic Generation of UIs • Benefits • All interfaces consistent for a user • With conventions of the handheld • Even from multiple manufacturers • Addresses hotel alarm clock problem • Can take into account user preferences • Multiple modalities (GUI + Speech UI) • A Hard Problem • Previous automatic systems have not generated high quality interfaces

37. Research Challenges • Automatic Design of Graphical User Interfaces • Automatic Design of Speech User Interfaces • Connection with real devices • Through various protocols • X-10, AV/C, HAVi, UPnP, etc. • Also, custom hardware and software

38. Hand-Generated Graphical Interfaces • First, Hand-Designed PocketPC interfaces: • AIWA Shelf Stereo (Tape,CD,Tuner) • AT&T Telephone/AnsweringMachine • Used Embedded Visual Basic • Ensured quality with heuristicanalysis and think-aloud studies • Compared with manufacturer’sinterfaces

39. Results of Comparison • Using PUC, users took 50% less time & made 50% fewer errors • All differences are significant (p < 0.05)

40. Discussion of Comparison • Our hand designed interfaces succeeded for several reasons • Good organization • Each button has one function • Good labels • Only available functions are active • Others, hidden on tabs or grayed out • Better feedback and error messages

41. Current PUC Specification Language • XML • Full documentation for the specification language and protocol: http://www.pebbles.hcii.cmu.edu/puc/ • Contains sample specification for a stereo

42. Properties of PUC Language • State variables & commands • Each can have multiple labels • Useful when not enough room • Typed variables • Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc. • Optional labels for values • Hierarchical Structure • Groups

43. Dependency Information • Crucial for high-quality interfaces • Expressed as <active-if> clauses • Operations: • Equals, Less-Than,Greater-Than • Combined Logically • AND, OR • Used for: • Dynamic graying out • Layout • Widget selection

44. Specifications • Have working specifications for: • Audiophase stereo • X-10 lights control • Sony CamCorder • Windows Media Player • Audio ReQuest hardware MP3 player • WinAmp Media Player • Elevator

45. Examples of Generated GUIs • Stereo and X-10

46. More Examples • Elevator

47. Generating Speech Interfaces • “Universal Speech Interface” (USI) project • Prof. Roni Rosenfeld of CMU • http://www.cs.cmu.edu/~usi • Creates grammar, language model and pronunciation dictionary from PUC specification • Pronunciation from labels using phonetic rules • Can provide other pronunciations as labels for fine-tuning • Will use dependency information to help with disambiguation and explanation • Supports queries and spoken feedback • Paraphrases as confirmation

48. PUC Architecture

49. “Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly Custom hardware Audiophase Stereo Custom software ARQ MP3 player SIMA MP3 player (future) X-10 Light switches, etc. AV/C (standard protocol) Sony CamCorder PlayStation, etc. (future) HAVi (current work) Mitsubishi HDTV 65” TV Mitsubishi HDTV VCR UPnP (future) ?? Adaptors

50. Adaptor using Custom Hardware for Stereo • Pretends to send IR codes • Reads LED panel signals to decode state • Created by Pittsburgh company: Maya Design