"Physical hypermedia": Organising collections of mixed physical and digital material

1. "Physical hypermedia": Organising collections of mixed physical and digital material Kaj Grønbæk , Jannie F. Kristensen , Peter Ørbæk , Mette Agger Eriksen, "Physical hypermedia": organising collections of mixed physical and digital material, Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, August 26-30, 2003, Nottingham, UK [doi>10.1145/900051.900056]

2. Dealing with information • Information exists in a mixture • Physical • Digital • Much information is available digitally but people don’t always want work with it that way • Print papers and photos out • Revise, annotate, sketch and diagram

3. Dealing with information • Other information or objects may only exist in a physical form with no realistic solution for fully digital information solutions • There needs to be a way to annotate and link information to these objects • When working on a project professionals need to be able to relate these physical and digital objects together

4. Dealing with information • Although he domain focus for this paper is architecture and design, this problem exists in many domains and fields (such as medicine and law) • Examples from this domain • Wood and foam models • Building material samples • These objects need to be able to be related back to the project description, drawings and specifications.

5. Ubiquitous Computing as a solution to the mix of media • So far in the ubiquitous computing field the focus has been • Developing infrastructure for display enabled devices on difference scales • PDAs, cell phones, watches • Making digital information available everywhere and shared • Much less research has focused on how to relate computing to physical objects such as paper, binders, samples, materials, containers

6. The presents an opportunity Augmented Reality • Field studies linking digital information to physical objects, spaces and locations • This is practically completed through an ID code (barcode), readers, and RFID (radio frequency identification) The authors focus on the ordinary physical artifacts of work • In the architecture field • And how these objects are used

7. Hypermedia and Spatial Relationships • A recent trends in hypermedia research, which studies links, composites, hierarchies, groupings and metadata has been to examine the aspect of spatial organization in digital information. • Described as a 2d canvas with cards being paced on a table • Cards can be changes in size, shape and color • May contain pointers to external content • Some cards can be “opened” to reveal a new space and different cards

8. Hypermedia and Spatial Relationships • Topos uses this approach in 3d, which was used by the authors in this paper. • The central concept Tops uses is “workspace” • Sets of spatially related and placed materials(drawings, models, notes, documents) • Supports internet sharing and collaboration • Supports 3d organization of models and documents • Can support abstract spatial hypermedia and concrete spatial hypermedia such as: • Open unfurnished spaces • Buildings or landscapes and backgrounds for relative material placement

9. Bridging physical and digital material • In the physical work environment people utilize things like paper, files, clipboards, clips, photos, sketches, wood and cardboard in their work. • Some of these items can be related digitally via folders and directories, but there isn’t much support for keeping these relationships constant.

10. Bridging physical and digital material • For this study the author’s utilize a RFID tag system and integrate it with the 3d Topos system • Provides support for linking physical material a digital environment • Through linking, grouping, annotating and tracking where physical objects were last located

11. How architects/designers manage physical materials • The process for organizing material can range: • From very formal (legal medial fields) • To open to definition by the individual or group

12. How architects/designers manage physical materials • Use of physical work space: • “nomadic” movements though the space for both collaborative and individual work • Space is not just used for organizing but also for exhibition of work and as a creative and inspirational atmosphere. • This implies a tagging system in this environment may have to be made up of many tag readers integrated into desks, worktops, walls and shelves

13. Organizing Physical Objects • Much of what this work entails is finding, collecting, producing and organizing many different types of physical materials • Paper, bricks, plants, plans, cardboard, models, samples, drawings • For inspiration, documentation, design process, reference, prototype

14. Meta-data Annotation • The three main purposes were discovered for organizing physical materials and objects • Documentation • Inspiration • Management • For all 3 purposes it is important to be able to able to re-find and recognize material for it’s later use

15. Meta-data Annotation • The ability to add annotations, and keep them related to the physical objects is also important in work situations • Example a series of time stamped models placed in order to represent progress and the ability to backtrack in a project • Collectional Artifacts • Can be described by the range of materials they are suited for organizing • In terms of the way they afford different aspects of collecting and organizing • Flexible(shelves), specific, mobile(stapler), link loosely(paperclip) or permanently • For this study collectional artifacts included: • Shelves, walls, boxes, folders, tape, clips, staplers

16. Abstractions of Collectional Artifacts • How do we map handling of physical materials to digital materials? • Notion of space in a mixed environment • Topos provides features supporting spatial organization of digital materials, the infrastructure supports spatial memory • the workspace concept is the main structuring mechanism • The next step is to map these physical organizational actions into semantics for the hypermedia system

17. Abstractions of Collectional Artifacts • How do we map handling of physical materials to digital materials? • Understanding collectional actions • Designers use a multitude of objects to organize physical collections together • From very formal to very informal • Regardless, these methods related to explicit and implicit meta-data actions • Example: Attaching a post it to a collection of papers while stapling it • This implies some reasoning based on category, associating materials by purpose over time tightly or loosely connected • Stapling more permanent that paper clipping • We wish to create open space of possibilities for coupling actions

18. Relationships between Physical and Digital Worlds • Based on abstractions of collections the following are possibilities: • Physical Only • Digital world has no trace of object • Physical w/ digital ID • Digital world has an ID plus meta-data relating to the object but no digital representation • Example: a brick with an RFID tag • Physical w/ low resolution digital • Example: a pen tracked drawing, scanned document or photo of an object

19. Relationships between Physical and Digital Worlds • Based on abstractions of collections the following are possibilities: • Physical generated from digital • A printed map, drawing or report • Digital Only • Content cannot be printed or made tangible outside of the digital • Video or sound source files

20. Relationships between Physical and Digital Worlds • Collections of materials from one category differ greatly from collections in other categories • Items from category 1 are difficult to map semantically into the digital environment • We can do this by description and properties and related the item to a tag or stamp so that is it transformed into cat 2. • Items from cat 2 can be identified ad described by meta-actions and grouped through collective actions • They still do not have distinct visual or auditory appearance in the digital environment • Items in cat 3 can have low res digital representation in the environment allowing them to appear in workspaces, but cannot be worked with dynamically • Allow collectional and metadata actions in the physical world to be mapped directly into digital actions

21. Relationships between Physical and Digital Worlds • Items from cat 4 correspond into common documents in Topos • Physical representations can be mapped digitally in mirrored actions • These materials may have physical representations with the same unique ID • Annotations or metadata added to any should be associated with the unique ID in digital world representing the material • For collectional actions for items in cat 5 mapping is done through an indirect relationship to the digital item augmented with a metadata ID referencing the actual digital material • Screen dump, transcription of a video or audio file

22. Physical Hypermedia Prototypes • The previous observations are the starting point for the prototype • The real world design took place in a workshop form over 3 days with 4 landscape architects • They were presented with a design task (real world) related to a site nearby • Given material and time to prepare pre-workshop • Asked to solve task in normal fashion utilizing the physical and digital work environments and prototypes with in the project • Discussion, observation or their work gave first iteration of user feedback on design issues and concepts of relating the the gap between physical and digital environments

23. Tagging- Registering and Detecting Collectional Actions • RFID tags were used on physical materials • Inexpensive, adhesive, placed in a cup on a table or workspace • Single or collectional • Tags are associated with an item through selection of the digital item and associating the tag • Extended the concept to support grouping into collections and link between materials • Topos pops up workspaces or objects when tagged items are placed near the tag reader antennas • RFID readers consist of: • a desktop model that recognizes several tags at once • Handheld models that see one tag at a time • Two modes: Tag arrived, tag left when events occur

24. Tagging- Registering and Detecting Collectional Actions • Easy tagging to track within office and electronic workspace • Tagging can take place after meetings as new documents are created • Metadata and annotations can be added to physical items through the digital representations • The RFID allows for concurrent contact less detection of numbers of tags containing unique IDs and can record that a collectional action has taken place in the vicinity • If physical objects have been removed from collectional artifact their digital counterpart displays this (with a red mark) to show it is missing from the collection

25. Representing Physical Material • Digitizing physical documents is important in the design process in bridging the two environments • Tagging physical material and linking it to digital material is ne way to do this • Other times having a digital “snapshot” in the electronic environment as a reminder is important. • A Snapscanner allows quickly adding a digital representation with an ID for a physical to digital link. • Many images found to be too low res to be helpful • Should be portable

26. Tool Tags • Physical cues for linking and grouping • Tool tag is an RID which is couples to a Topos command in hypermedia system rather than piece of info. • Commands are then issues by placing item on reader with or with out a physical item • Allow for tangible interaction, making a number of commands available even when mouse & keyboard are not • Link group much like clips and staplers • Users found it difficult to use tooltags on pure digital objects or accidentally applied more then one conflicting tags

27. Conclusions • Need to extend hypermedia to better integrate physical world • Have identified a set of abstract collectional and meta-data actions to support this • Goal to develop a physical hypermedia system to be at hand for users and pervasive • To be familiar in a common metaphor

28. Questions • In an environment like construction, or architecture, does this make sense to use this technology? • This process is very people labor intensive (add work) does it add the required value in this situation? • Ex. Many large business use similar processes (with barcodes) for inventory management, saving millions of dollars. • Where would it make more sense to use this technology?