Collaborative Image-based Tagging and Interactive Mapping on Tabletops

1. Collaborative Image-based Tagging and Interactive Mapping on Tabletops Presented By: Glen Whitaker Supervisor: Judy Kay

2. Research Question • How to Support Tabletop interaction for organisation of Content? • Why is it important? • Novelty of User Interface • Organising is fundamental • Especially file systems • Constraints? • Limited methods of input • Challenges? • No established toolkits, primitives, UI design • No guidelines • Research - Cruiser Tabletop as exploratory framework

3. Refined Research Question • How to Support Tabletop interaction for organisation of Content using Tagging and Maps? • Why? • Tagging text is similar to associating file name and location to files in a file system • Maps similar concept to the folder structure in a file system

4. Sub-Problem: How to Support Tagging at Tabletop • Adding Tags to Images • Linking Images to maps • Single text/image tags • Multiple Tags

5. Refinement: Elements of Tagging • Way to organise and find digital artefacts • Key interface tasks for tagging • Defining tags • Adding tags to objects • Viewing tags • Searching tags • Deleting tags • How to achieve this on Tabletop Interfaces?

6. Approach 1: MyTags • Inspired by TeamTag • Morris, M. R., A. Paepcke, T. Winograd, and J. Stamberger. "TeamTag: Exploring Centralized Versus Replicated Controls for Co-Located Tabletop Groupware." Proceedings of the SIGCHI conference on Human Factors in computing systems CHI: ACM Press, 2006. 1273 - 82. • Text-Based Tagging • Adding Text to Images • Adding Multiple Text to Images

7. Approach 2: MyMaps • Image-based Tagging in context of maps • Adding Images to Map Locations • Placing image on a single map location/region • Adding Images to Multiple Map Locations • Placing an image to multiple regions on map

8. Prototype

9. Defining Tags • Tags are predefined (externally in a file/database) • Tags represented by special objects

10. Adding Tags

11. Viewing Tags • MyTags • Flip image over to view list of text • MyMaps • Dwell on tag to view image • Viewing is same for single and multiple tags in both MyTags and MyMaps

12. Deleting Tags • Move tag to blackhole • Same for tags in both MyMaps and MyTags • Same for single and multiple tags

13. The Framework • Cruiser • Images • Blackhole (garbage collection object) • Tagging Primitives • Adding Text Tags • Adding Image based tags • Adding Regions • View Tags • Delete Tags

14. Previous Elements Blackhole Images • Previous Elements used by MyMaps: • Blackhole (garbage collection object) • Images (used to represent exhibits) • Audio can be associated with image • Images can have other images placed on back

15. New Elements • “New” elements of the Cruiser Tabletop • Map Resources • Map Region Creation Tool • “Pin Images” similar to pins on a notice board • Image Region Identification Map Back of Image Region Pin Images Text Attachment Region Tool Text Object

16. Architecture = Trent’s Cruiser Elements = New Elements

17. Evaluation • Performance • Successful Tagging with MyTag • Successful Tagging with MyMaps • Other usability elements • Comparison of affordances • Learnability • Error Rate • Error Recovery • Collocated Collaboration UI aspects • Duplicated Maps • UI Facilitates Collaboration • Evaluate Effects of Short term memory load • Images • Maps

18. Evaluation Design • Qualitative Evaluations • 6 Pairs (User A, User B) • Asked to design their own museum • Double cross over condition (TeamTag and MyMaps + MyTags) • User A and B given different sets of text attached to 10 images • Questionnaire for effective issues: • Preference, attitudes, perspective

19. Evaluation Steps • Tutorial • Task 1 • Design virtual Museum • Task 2 • Free-form virtual Museum • Tutorial of cross-over system • Repeat Tasks 1 and 2 for cross over system • Questionnaire Encourage Collaboration in Tagging Context Evaluate Collaboration and usability without design limitations Double cross-over to control first system effect

20. = Represents System used Second (always faster)

21. = Represents System used Second = Preferred Method X

22. 10/12 Preferred Second System 2 preferred MyMaps even as first system = Represents System used Second = Preferred Method X

23. 2 People (1 Pair) liked both Others evenly split Some commented regions suited task better and less planning needed = Represents System used Second = Preferred Method X

24. 5 People (2 Pairs + 1) liked but hardware limitations prevented effective use = Represents System used Second = Preferred Method X

25. All participants liked Image Tagging and could use comfortably = Represents System used Second = Preferred Method X

26. Manly yes, participants lost tags creating objects Pair 1 preferred flexibility to organise = Represents System used Second = Preferred Method X

27. Performance Results • Successful Tagging with MyTag • Association of Text to objects performed with low error rate • Pairs completed tasks successfully using method • Successful Tagging with MyMaps • Association of Images was performed with low error rate • Pairs completed tasks successfully using method

28. Other usability elements • Swipe action for text tagging was liked and not really in the design • Learnability • Easy to learn tagging primitives • Error Rate/Recovery • Low error rate for Swipe/movement text association technique • Errors were due to having multiple images flipped within path of movement • Low error rate for image tags on maps • Errors were mainly a result of lack planning suitable location • Image Tags • Errors fixed by deletion of reference point and further planning • Text Tags • Errors fixed by deletion of attached text

29. Collocated Collaboration • Duplicate Maps • Participants did not use replicated maps as expected Hardware limitations - multiple use - affected this • 3 out of 12 participants used second map as a thumbnail view of the focus map • UI Facilitates Collaboration • Participants shared text objects • Participants collaboratively planned image tag locations • Participants collaboratively planned text tags for each image

30. Effects on Short-Term Memory Load • Images • Images are flipped in tagging, causing short-term memory load but avoiding confusion • Images could not be semi-transparent since it would conflict with other functions • Maps • Dwelling needed to view images relating to tags • image tag design reduced clutter and confusion about map objects as the cost of the need to check what reference points represented • Participants were observed to have minor problems remembering tags with the small sample of images and text

31. Caveats • Hardware • Mimio capturing technology used for Cruiser only allows one interaction at a time • Software • Objects are 2-Dimensional representations • Text predefined in external file loaded at start up • Precision of interaction depends on accuracy of calibration

32. Future Work • Handwriting for tag inputs • Backend database for efficiency (minimise I/O) • Personalised recommendations based on tags • Link tagging to file system – (Anthony Collins thesis)

33. Contribution • Significant progress in understanding how to support collocated collaborative planning at a tabletop • Insight into new gestures for image tagging and text tagging on tabletops • Swipe gesture discovered

34. No more slides. Click to continue.

35. Results • Was 6 pairs enough? • Sample showed consistent patterns relating to Image Tagging • Provided substantial usability, performance, collaboration and affordance results • More detailed and extensive evaluations required for further results

36. Background: Virtual Museum Implementation • Virtual Museums • Digital Technology being used to display exhibits while preserving the contents of an exhibit in digital format • Augmented Reality and 3D Modelling advancements allow photorealistic exhibits • Huang, C., C. Chen, and P. Chung. "Projects in VR: Tangible Photorealistic Virtual Museum." IEEE Computer Graphics and Applications: IEEE, 2005. pp 15 - 17.

37. Direct Touch Interfaces • DiamondTouch Tabletop • Direct-Touch Based Interaction • Unique identification based on circuits through chairs/pads • Other Touch based tables include: SmartSkin, Entertaible and Misto • Ryall, K., M.R. Morris, K. Everitt, C. Forlines, and C. Shen. "Experiences with and Observations of Direct-Touch Tabletops." First IEEE International Workshop on Horizontal Interactive Human-Computer Systems: IEEE, 2006.

38. Lumisight – Rear Projection Design • Lumisight Tabletop • Projectors inside the table project different images in different directions • Number of users limited to number of projectors • Kakehi, Y., T. Hosomi, M. Iida, T. Naemura, and M. Matsushita. "Transparent Tabletop Interface for Multiple Users on Lumisight Table." First IEEE International Workshop on Horizontal Interactive Human-Computer Systems: IEEE, 2006.

39. TeamSearch • TeamSearch • Use of special widgets to represent metadata shared by all users • Search images through use of widgets • Search through manually tagged images using widgets • Morris, M.R., A. Paepcke, and T. Winograd. "Teamsearch: Comparing Techniques for Co-Present Collaborative Search of Digital Media." First IEEE International Workshop on Horizontal Interactive Human-Computer Systems: IEEE, 2006. pp 97 - 104. Widget Personal Space Query Objects Metadata inside widgets

40. TeamSearch • Added Tags • Tags in the original implementation were added manually to each image • Searching • Users search the photos by placing query object onto the metadata values • Thumbnails of matching photo’s appear in that user’s personal space • Tags are • Metadata that matches the contents of the widgets

41. TeamTag • TeamTag • Presentation and association of metadata • Centralised-control (TeamSearch approach) and Replicated-control design • Morris, M. R., A. Paepcke, T. Winograd, and J. Stamberger. "TeamTag: Exploring Centralized Versus Replicated Controls for Co-Located Tabletop Groupware." Proceedings of the SIGCHI conference on Human Factors in computing systems CHI: ACM Press, 2006. 1273 - 82. Centralised-Controls Design Replicated-Controls Design

42. TeamTag • Centralised-controls design • Implementation of TeamSearch widgets • Users attach metadata by tapping image, then subsequently selecting a piece of metadata in a widget • Replicated-controls design • Presents metadata categorised into columns that are presented in front of each user • Users attach metadata by tapping image, then subsequently selecting a piece of metadata in the collection presented in front of them

43. TeamTag • Added Tags • Tapping the image to be tagged and the subsequent piece of metadata selected by that user is attached • Multiple tags can be associated to images using this method • Tags are • Defined sets of metadata • Viewing Tags • Popup lists present what metadata is associated with an image • Popup lists activated by a user touching an image with two fingers

44. Centralised Vs Replicated • Evaluated in context of zoo, labelling each image with appropriate metadata in groups of 4 • 19 out of 24 subjects preferred replicated metadata interface • Collaborative evaluation of replicated interface showed on average 1.24 users contributed to the tagging of an image • Group collaboration and communication levels presented no significant difference

45. Google Maps • Google Maps • Interactively move around map of world • Zoom in on satellite and road maps of areas around the world • Identify businesses on the maps by providing business info to Google to tag based on street address • WikiMapia provides user defined region tagging • Google Maps Tour. Google. Available: http://www.google.com/help/maps/tour/. 10th of August 2006.

46. Google Maps • Added Tags • WikiMapia – Drawing of a box on the map to indicate a region • Google Maps – Entering of data with the location being labelled by the system based on data inputted about a business • Searching • Searches performed by comparing text inputted to the metadata that is associated with a tag • Tags • WikiMapia – regions are user defined boxes with a text tag identifying what the region is • Google Maps – identifies single addresses with business details associated with that address (limited to USA, Japan, UK, China and Canada) • Point Vs Region • WikiMapia allows user identification of regions • Google maps allows identification of single addresses on maps

47. The Cruiser • Cruiser Tabletop • Uses Mimio® to Capture Interaction • Uses Pen stylus devices for interaction • Number of users limited to Pen Numbers • MyMaps and current software on the cruiser can be implemented using DiamondTouch interfaces • Trent Apted’s First Year of PhD presentation on Photo sharing with Sharepic system and the Cruiser Tabletop

48. Map Resources • Contribution • Explore how collaborative planning activities can be performed on tabletop interfaces • Provides a context for image based tagging • Transparency levels represent the interactive state • Maps are replicated for the number of users Interactive Map Non-Interactive Map Map With some Tags Walls in Museum

49. Regions • Contribution • Categorisation through areas on map objects • Evaluate the effectiveness of planning through the identification of regions • Using a region creation tool to identify these regions • Image tagged to maps within regions inherit any text for those regions Attached Regions Region Creation Tool E.g. Greek E.g. Roman

50. Image Tags • Contribution • Evaluation of the effectiveness of image based tagging on collaborative interfaces • Evaluate how this assists in planning tasks on tabletop interfaces • Assigned by performing a gesture on the image to be attached while placed on top of the map Map with Image Tags of 2 different Users Images