Toolkits for Physical Objects (TUIs)

1. Toolkits for Physical Objects (TUIs) Jack Li Advanced User Interface Software November 30 Fall 2004

2. Outline • Motivation for TUIs • Input Technologies • TUI Toolkits

3. MOTIVATION FOR TUIS Current Limitations of UIs • Desktops and laptops suck in our entire attention • As a result we live in two worlds: the real physical world and the virtual cyber world • Constantly “wired” in order to be in both worlds • Loss of rich interactions found in everyday things (particularly haptic)

4. MOTIVATION FOR TUIS Motivation • Driven by ubiquitous computing • Both people and technology moving away from the desktop into the environment • Input: different ways of interacting • Output: different ways of displaying

5. MOTIVATION FOR TUIS Tangible User Interfaces • TUIs augment the physical world by integrating digital information with everyday physical objects • Goal: to bridge the gap between bits and atoms • 1. Interactive surfaces • 2. Coupling information with physical objects • 3. Ambient media for background awareness

6. Outline • Motivation for TUIs • Input Technologies • TUI Toolkits

7. INPUT TECHNOLOGIES Physical Objects as Input • Pretty much anything … walls, papers, utensils, clothes hangers! • Inspiration • sci-fi novels, own imagination • everyday life, current practices

8. INPUT TECHNOLOGIES Devices • Cameras • Electronic tags (RFID technology) • Sensors • Light • Motion • Heat • Force • Motors • Orbs

9. INPUT TECHNOLOGIES Framework for the GUI (a)model-view-controller

10. INPUT TECHNOLOGIES Framework for the TUI (b)MCRpd: model-control-representation(physical and digital)

11. Outline • Motivation for TUIs • Input Technologies • TUI Toolkits • Phidgets (2001) • Calder (2004) • Papier-Mâché (2004)

12. Toolkit Goals • Same as when GUIs were built • Make easy for non-specialized experts to create (should not need a vision or hardware guru on the team) • Hide dirty implementation details • Abstract proper functionality with well-defined API

13. PHIDGETS Overview • First toolkit for physical objects • Reusable physical devices with proper abstraction for creating a wide variety of applications • An API that had the right amount of abstraction

14. PHIDGETS GUI Differences • Connection manager • Way of detecting when devices go on/off • Identification • Every physical object should have a virtual counterpart • Simulation • A way to test code even if phidget not present

15. PHIDGETS Architecture

16. PHIDGETS The Physical Device • Primitive device components • Sensors • Motors • Switches • Circuit board • CY7C63000 USB micro-controller • Communications layer • USB communication standard

17. PHIDGETS Wire Protocol • Low-level software in Windows and microcontroller for communication that’s hidden from end programmers • Commands from phidgets • Unique Id numbers • Device type (servo, powerbar) • Specific events (tag for RFID reader) • Commands from Windows • Set device state (output like servo)

18. PHIDGETS Computer Software • PhidgetManager • Connection manager • Event onAttach() • Event onDetach() • Int count • Enum item • Phidget-specific devices • All have: • String deviceType • Boolean isAttached • Long serialNumber • RFID • onTag() • Servo • onPositionChanged(), motorPosition(index)

19. PHIDGETS Simple Application • Phidget Manager • Phidget Servo

20. Outline • Motivation for TUIs • Input Technologies • TUI Toolkits • Phidgets (2001) • Calder (2004) • Papier-Mâché (2004)

21. CALDER Main goals • Primarily geared towards interaction/product designers • Support for more refined products • Support on the hardware side

22. CALDER Supported Devices Small size is big win!

23. CALDER Fluid prototyping Objects easily embedded!

24. CALDER Communication • Wired • PIC16C745 C with USB interface engine • Same as Phidgets (and iStuff) • Wireless • Tradeoff between size and power (perhaps Moore’s law will remedy) • Small batteries, antenna, and pulse-width modulation

25. CALDER Software • Surrogates (GUI widgets) to manipulate physical object output

26. Outline • Motivation for TUIs • Input Technologies • TUI Toolkits • Phidgets (2001) • Calder (2004) • Papier-Mâché (2004)

27. PAPIER-MACHE Overview • Emphasis on software side side • Input model that provides common programming abstractions • Focus on objects being lightweight

28. PAPIER-MACHE Software Abstractions • Phob—encapsulates information related to this physical object • All—source producer, time • RFIDPhob—tag • VisionPhob—mean color, bounding box, rotations • PhobListener—listens for Phobs produced • phobAdded(), phobUpdated(), phobRemoved() • PhobProducer—actual devices that generates events • PhobManager—maintains PhobProducers

29. PAPIER-MACHE A Common Event Model • PhobListener is the interface among all devices • Methods are the same • phobAdded(), phobUpdated(), phobRemoved() • Vision • phobAdded() when new object detected, phobUpdated() when old object changed (size, color), phobRemoved() when old object disappears • RFID (different from what the paper says) • phobAdded() when tag detected, no phobUpdated(), phobRemoved() when tag disappears • Facilitates easier retargetting and substitutions (allows for more exploration and faster, prototyping)

30. PAPIER-MACHE Marble Answering Example • 1. Get the manager that keeps track of all input devices (PhobProducers) RFIDManager manager = RFIDManager.getManager(); • 2. Set up a map associating all objects with an audio clip AssociationFactory factory = new AudioClipFactory(); AssociationTreeTable assocMap = new AssociationTreeTable(factory); • 3. Attach this map so new objects are assigned new clips and old ones get played manager.addPhobListenerForAllProducers(assocMap);

31. PAPIER-MACHE A Behavior: Associations • A common behavior we noticed • New objects get stored/assigned, old objects get retrieved a database of sorts • An extension of PhobListener

32. PAPIER-MACHE A Behavior: Associations • AssociationTreeTable • addClassifierFactoryPair(Classifier, Factory) • ObjectClassifier • isMemberOfClass(Phob phobToCompare) • AssociationFactory • createAssociationEltForPhob(Phob phob)

33. PAPIER-MACHE Monitoring Window

34. PAPIER-MACHE Device state—connection mgr

35. PAPIER-MACHE Camera Feedback

36. PAPIER-MACHE Association Map

37. PAPIER-MACHE Extras

38. Summary

39. Major Toolkit differences • Phidgets • Complete, balanced, varied assortment of physical objects easy to program with • Calder • Small size allows for more fluid prototyping at the PHYSICAL level • Papier-Mâché • Unified event model allows easier development and retargetting at the SOFTWARE level

40. Conclusion • TUIs seek to augment the physical world with digital information • Toolkit support allows non-experts to develop TUI application, like GUI toolkits • Successful toolkits must manage physical connections, gather/alter relevant physical state information, and provide good feedback for these added complexities

41. Future work • Input adaptation • M to N mapping of input devices • Input models • What other kinds of behaviors can be abstracted • Will be more clear with the development of more applications

42. References • Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States • Ullmer, B. and H. Ishii, Emerging Frameworks for Tangible User Interfaces, in Human-Computer Interaction in the New Millennium, Addison-Wesley. pp. 579 -- 601, 2001. • Saul Greenberg, Chester Fitchett. "Phidgets: easy development of physical interfaces through physical widgets," Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida • Scott R. Klemmer, Jack Li, James Lin, James A. Landay. "Papier-Mache: toolkit support for tangible input," Proceedings of the 2004 conference on Human factors in computing systems, p.399-406, April 24-29, 2004, Vienna, Austria • Lee, J.C.; Avrahami, D.; Hudson, S.E.; Forlizzi, J.; Dietz, P.H.; Leigh, D.L., "The Calder Toolkit: Wired and Wireless Components for Rapidly Prototyping Interactive Devices", Designing Interactive Systems (DIS), ISBN: 1-58113-787-7, pp. 167-175, August 2004