Using Hand-Held Computers and PCs Together: The Pebbles Project

1. Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Using Hand-Held Computersand PCs Together:The Pebbles Project Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles

2. Multiple Devices • Most of our time spent in places where there is embedded technology • Offices, meeting rooms, classrooms, homes • Often multiple devices will be available: • Mobile phone and PDA and a PC • Multiple people’s PDAs • PDAs in a “Smart Room”

3. Handhelds will be communicating • 802.11 • BlueTooth • Cell-phone network • (Infrared)

4. Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be inclose, interactive communication.” • Brad Myers, “Using Hand-Held Devices andPCs Together,” Comm. ACM,Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.

5. Research Agenda How can multiple devicesbe used effectively together,at the same time? How can the user interface and functionality be spread across multiple devices?

6. Research Agenda “Multi-MachineUser Interfaces” How can multiple devicesbe used effectively together,at the same time? How can the user interface and functionality be spread across multiple devices?

7. Example: Power Point Control • Use PC to give the presentation • Use hand-held to control the PC • Two-way communication • Hand-held shows picture of slide,notes, list of titles, timer, etc.

8. Pebbles is: DAs for ntry of oth ytes and ocations from xternal ources. P E B B L E S http://www.pebbles.hcii.cmu.edu/

9. Handhelds in an Office • Augment desktop applications • Use multiple devices at the same time

10. Handhelds in Meetings • Attendees use handhelds to interact andannotate presentation • Augment collaboration • Take notes

11. Handhelds in Classrooms • Data projectors for instructor’s slides • Students could have computers for: • Notetaking linked to instructor’s slides • In-class testing • Running simulations

12. Handhelds in Homes • Interact with embedded computation • “Smart homes” • Not just speech and vision as interfaces • Interact with appliances, lights, etc. • “Personal Universal Controller” (PUC)

13. Office Use (For Individuals) How can handhelds augment desktop applications?

14. Applications for Individuals • Extra input and output devices have been shown to be useful • But can be expensive and hard to configure • People have PDAs and are attached to PC • For example, cradles for recharging • Customizable, extensible • Extend desktop applications

15. Scrolling with the PDA • For scrolling using the non-dominant hand • Studies showed parallel and efficient uses of both hands together • Generates Windows scrolling events

16. Winner! Results of Study of Scrolling with PDA • Scrolling with buttons on PDA was fastest • PDA scrollers similar to mouse speed • Using 2 hands is effective!

17. Remote Clipboard • Transfer information between PDA and PC • Connects their clipboards together • Transfer content or reference • Works with all applications • Also between multiplecomputers

18. Shortcutter • User-created panels of controls • Create custom interfaces and extensions to PC applications • And then take them with you • Direct manipulationfor edit, then setproperties • Palm or PocketPC

19. Shortcutter Widgets • Buttons • Sliders • Knobs • Mouse pad • Graffiti Pad (Palm) • Gesture panel

20. Shortcutter Actions • Send any keyboard key, mouse button, scrolling action or string to PC • Open a file or URL • Run an application • Invoke any PC menu or button • Windows message • Recorded • Switch to a different Shortcutter panel • Control the Mouse

21. Shortcutter Actions, cont. • Control external devicesthrough PC’s serial port • Directly (e.g., projectors) • X-10 for electrical devices • Macro • Can be multi-application • Application-specific • Same button, different messages • Useful for application sets:browsers, compilers

22. More Scenarios of Use • Lean-back mail reading • Controlling WinAmp • … and many others

23. Study of Individual Use • Time to tap on button depended on size • Few errors • People often didn’t look at PDA

24. Study of Individual Use • Moving hands to both PDA and mouse only about 15% slower than just moving to the mouse msec 1H Keyboard->Mouse 728 1H Keyboard->PDA 744 1H Mouse->Keyboard 701 % slower 1H PDA->Keyboard 639 Keyboard -> Mouse&PDA 838 15.1% Mouse&PDA -> Keyboard 791 12.8%

25. Use in Meetings Enhance group’s collaboration and control

26. Original Application:Remote Commander • Allow PDAs to control a PC • Can be used with any application • Uses the standard (single) cursor • Don’t have to jump up and grabmouse • Perform all mouse andkeyboard functions • Use PDA like touchpad • Graffiti or our ownpop-up keyboard

27. PocketPC version • Get PC’s screen onto PocketPC • Full view, or one-to-one zooming • Scroll with iPaq’s buttons

28. Scribble • Multiple people draw on top of whatever on PC screen, not just PowerPoint • Each user has own cursorand color • Save by PrintScreen • Erase by refresh

29. SlideShow Commander • For PowerPoint • PC Only: Windows 95, 98, NT, 2000, ME, XP • Use PC to give the presentation • Full features of PowerPoint • Use hand-held as “remote control” for PC • Can wander away from the keyboard • Two-way communication • View, Navigate, Highlight, Time

30. View on Hand-Held: slide • See thumb-nail of current slide • Black and white or color

31. View on Hand-Held: notes • See the notes of the current slide

32. View on Hand-Held: titles • See the list of titles

33. View on handheld: Time • Multi-function timer • View large • And in corner

34. Control Demonstrations on Palm • Control PC applications and external devices • Uses Shortcutter • Easy demo and resume show

35. Control Demonstrations on PocketPC • Task list of PC applications • Tap brings one to front • PowerPoint continuesbehind – easy to resume

36. Commercialized • SlideShow Commandercommercialized by: • Available at Office Depot,CompUSA, etc. • Palm and PocketPC Synergy Solutions, Inc. http://www.synsolutions.com/ http://www.slideshowcommander.com/

37. Studies of Presentations • Summer study of 2 HCI Master’s students • Contextual Inquiry of 9 presentations • Audience: 10 to hundreds • 7 used PowerPoint, 4 used NetMeeting • Found 220 “breakdowns” • Most were minor problems • Averaged 8.7 lost minutes per talk (14.5%) • Designed SlideShow Commander to eliminate some of these problems

38. MultiCursor • For special applications that are awareof multiple inputs • E.g.: Shared Whiteboard • Single Display Groupware • Multiple people, one display • A number of interestingissues: • Palettes, widgets • Section handles • Undo

39. Chat • Communicate to another PDA user through the PC • PC serves as a conduit • For side notes and messages • For example, in negotiationmeetings • Send to all or to a specificperson

40. Command Post of the Future • Large DARPA funded project • Make commanders more effective

41. Study of Laser Pointers • Studied properties of laserpointer interaction techniques • Hand-wiggle 8 pixels • Delay until moving average stable 1.5 sec • Delay until target acquired 1.5 sec • Different devices and poses do not help much

42. Tap Speed Study • Tapping directly onSmartBoard is fastest • Semantic Snarfing next • But high error rate • Then regular mouse • Laser pointing slowest • Used separate physicalbutton on handheld

43. “Semantic Snarfing” • Interacting at a distance • Grab contents to handheld • Picture • Menus • Text • Re-visualization • “Magic Lenses”

44. Classroom Use Using handheld computers in classrooms For testing Improve large lectures Provide immediate feedback to instructor

45. Hardware • HP donated 110 Jornada 680 and 100 720 computers • Windows CE • Lucent donated Wavelan wireless cards

46. Context • Collaborator: Prof. David Yaron of Chemistry • Chemistry 09-106: Modern Chemistry II • Spring, 2000 and Spring, 2001 • About 90 students • Mostly freshmen • Loaned one Jornada and Wavelan card to each student for the whole semester • So would get used to using it • Offset technical difficulties with benefits to student

47. Wireless Andrew • CMU has almost complete coverage of 802.11b support • Wavelan • Donations from Lucent, etc. • Funding from state of Pennsylvania, etc. • Can use computers wirelessly in all classrooms and offices • Most lawns, dorm rooms, etc. • Windows, Mac, Linux, Windows CE support

48. Pebbles Tests • Create through html pages • Arbitrary html in questions and answers • Pictures, formatting, links • Use FrontPage, etc. to author • Embed tags to show question and answers

49. Concept Test • Students allowed to answer multiple times • Server stores all answers • Prof. Yaron displayedquestions and answers onthe board • Evidence that significantlycontributes to learning • Mazur, E. (1997).Peer Instruction:A User's Manual, Prentice Hall.

50. Instructor’s View • In class, instructor sees a chart of answers • Optionally refreshes every 3 seconds • May be projected forwhole class tosee • Top displaysinstructions forstudents