Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments

1. Pick-and-Drop:A Direct Manipulation Technique for Multiple Computer Environments Jun Rekimoto Sony Computer Science Laboratory Inc Oct.15.1997

2. From Single Device UI to multi-Device UI Current PC/GUIs are designed for One-user -- One-computer environment Computer Wall Future Computing Environment must be designed for Multi-user -- Multi-computer environments PDA PDA PDA PDA Kiosk PC PC PDA UIST’97

3. Limitations of Single-Device GUIs • One-to-One dogma • Mouse and Keyboard are connected to one computer • cf. Paper and Pencil • No Direct-Manipulation technique that can interrelate two or more computers • e.g., Drag-and-drop between two computers UIST’97

4. Mouse Jungle UIST’97

5. Note: These computers are connected to the Network Data Copying through Human Body UIST’97

6. How people transfer information between nearby computers? % % % % A survey conducted on 40 engineers in Sony’s Software Labs. % % UIST’97

7. Multi-Computer (Device) UIs • No cognitive overload as the number of devices increases • Virtual / Physical Characteristics • Number, size, orientation, and location of devices • Effective combination of a variety of devices • Boundaries between computers/devices • “Seamful but beautifully designed seams”[M.Weiser, Invited Talk at UIST94] • An Example: Pick-and-Drop UIST’97

8. Inter Computer Pick-and-Drop Pick-and-Drop Drag&Drop UIST’97

9. Information transition between virtual and physical spaces Pen virtually Carries the data Display Surface Physical Space Pick Drop Computer Space 1 Computer Space 2 UIST’97

10. Data Shadow • Visual feedback for pen carrying data Display Surface UIST’97

11. Applications of Pick-and-Drop Picking up information from Kiosk (push media) terminal Information exchange between PDAs UIST’97

12. Wall-sized Display + Palmtop Computer UIST’97

13. Video UIST’97

14. System Configuration Wall Size Display (or Physical Wall) PDA (Nomadic PCs) Desktop Display Pen ID = 1 Network Pen ID = 2 ID Data Pen Manager 1 xx.gif UIST’97

15. Pick-and-Drop between Paper and Computer Page ID Object ID Camera Pick Position Page Code Printed Objects Tablet UIST’97

16. Discussions Symbolic • Traditional Data Transfer • Copy data from C:/ My Document of Machine A to /user/local …. • Pick-and-Drop • Copy data from here to there Spatial UIST’97

17. Summary • Multi-Computer (Multi-Device) UI • as a new paradigm of direct-manipulation techniques • Pick-and-Drop: • information transfer technique for multi-device environment UIST’97

18. Future Work • Generalized Pick-and-Drop • information exchange between electronic appliance (e.g., VCR), ubiquitous computers, and physical objects (paper, wall, etc.) • Pick-and-Drop of active objects • e.g., JavaBeans • Application to distributed environments • Pick-and-drop over video conferencing UIST’97

19. Pick-and-Drop over ClearBoard ENVISIONMENT UIST’97

20. UIST’97

21. State Transition Diagram PICK DROP DRAG MOVE DRAG MOVE PEN-DOWN PEN-UP DOWN Time Out Time Out MOVE PEN-UP DRAG PEN-UP Time Out UIST’97

22. Pick-Move, Pick-Copy Copy Move UIST’97

23. Related Work (1/2) • DigitalDesk [Wellner 93] • Data transfer from paper to the desk • LiveBoard + PARC TAB [Want et.al 95] • PARC Tab as a remote pointer for the LiveBoard • Graspable UIs [Fitzmaurice et.al 95] • Using physical handles (bricks) to manipulate computer objects UIST’97

24. Related Work (2/2) • PDA-ITV [Robertson et.al 96] • PDA as a remote controller for the interactive TV • PaperLink [Arai et.al 97] • Picking up information from paper • Ultra Magic Key (UMK) [Usuda et.al 97] • TV manipulation using a paper booklet • Tangible Bits [Ishii et.al 97] • Physical artifact for “Bits” (phicons) UIST’97