Perspectives – surface computing

1. Perspectives – surface computing Surface computing evolution And design of the interaction

2. Overview • A little history • Affordances of surface interaction • Tabletops, with touch interaction • Walls • Bodies • GUIs: WIMP versus surface computing • File system • Command interface • Network • Recent NUI interfaces

3. Touch affordanceshttp://www.billbuxton.com/multitouchOverview.html • Keyboards: physical v onscreen • Very long history of touch • From late 60’s single touch • Multi-touch (Bob Boie, Bell Labs, 1984) • Capacitive array of touch sensors over CRT • Buxton, U Toronto 1985 • Digital Desk (Wellner, 1991) • Front projected • Optical and acoustic sensing hand/finger/object • 2 finger scaling, translation, pinch

4. Seminal systems • PLATO (U Illinois) 1972 • Simon (IBM, Bell South, 1992) • First smart phone, single touch • Diamond Touch (MERL, 2001)

5. TouchLight (Wilson, Microsoft, 2004) • Rear projected • Image processing from acrylic plastic • PlayAnywhere (Wilson 2005) • Objects • Plastic Logic (Cambridge, UK, 2006) • Flexible e-ink display • Multi-touch pad

6. Jeff Han (2005) • Perceptive Pixel (2006) • Rear projection • TFIR • Apple iPhone (2007) • Microsoft surface (2007)

7. Rethinking GUIs for NUI

8. Core OS functions What are the most fundamental facilities of the main OS-GUI? (eg Windows, Linux….) Are they fundamentally different on an interactive surface, compared with a desktop?

9. Core OS function File system http://rp-www.cs.usyd.edu.au/~judy/Talk/OnTop-hiqual.avi

13. How to evaluate a new file access interface? How would you evaluate an associative file access mechanism?

14. Studies • Compare Focus Associative Search with…. • Results: • Collaboration: Focus encouraged more interaction with each other’s files than with Explorer – people discussed more files • Screen use: Focus lead to greater utilisation of the tabletop space • Hierarchical still has an important role Collins, A. Bezerianos, G. McEwan, M. Rittenbruch, R. Wasinger, and J. Kay. Understanding file access mechanisms for embedded ubicomp collaboration interfaces. In UbiComp '09: Proceedings of the 11th International Conference on Ubiquitous Computing, pages 135-144, New York, NY, USA, 2009. ACM

15. Core OS functions Start new applications Switch files used by an application Alter configuration for an application Activate/remove interface elements (cf commands) C. J.Ackad, A. Collins, and J. Kay. Moving beyond the tabletop as an appliance. In Adjunct Proceedings of ITS '09, the ACM International Conference on Interactive Tabletops and Surfaces, 2009.

16. Case study: tabletop affordance for capturing group interaction and making it visible

17. Tabletops for collaborative problem solving and knowledge creation

18. Identifying the user How important is it to identify who does each task? How much did it figure in your use of concept mapping, brainstorming and WellMet?

19. Collaid Learner’s physical differentiation

20. Our enriched interactive tabletop • Logs: • Differentiated tabletop actions • Snapshots of the artefact Kinect sensor Multi-touch tabletop R. Martinez, A. Collins, J. Kay, and K. Yacef. Who did what? who said that? Collaid: an environment for capturing traces of collaborative learning at the tabletop. In ACM International Conference on Interactive Tabletops and Surfaces, ITS 2011, pages 172-181, 2011. 

21. Our gear 2 Cmate Concept Mapping at the Tabletop

22. Automatically distinguish high from low collaboration Model trained on a dataset captured from a multi-display setting Low collaboration group Highly collaborative group A total of 453 and 88 frequent patterns for high and low collaboration groups respectively

25. Can tabletops automatically help a teacher determine the group most in need of attention?

26. Awareness and Control

27. Collaboration and equality Participation: quantity of learner’s actions t(47) = 2.71, p < .0094 Participation was more equal in Activity 1 “everyone starts with the same basis so at the beginning they don’t know who’s the leader, who understood the case …” Teacher:

28. Collaboration and equality Contribution: quantity of actions that changed the content of the artefact …some groups agreed that they all were going to discuss and only one student would perform the physical actions. “… [next time] I should emphasise and make sure that everyone is fine to use the tabletop and include their ideas…” Teacher:

29. Adherence to the class script (14 tutorials) There was not enough time for activity 2 as planned Implications This was the most important activity from the learning perspective It forced the teacher to use more time than the 50 minutes

30. Learning outcomes for activities This analysis suggests the low achieving groups took longer to get started. “It would be more valuable to get this information per each group during the tutorials”. Teacher: * high achieving groups had more than 50% of the “crucial” propositions

31. Hardware independence What are the affordances of different tabletop hardware, present and predicted for future?

32. Hardware independence How to design for that? (And operating system independence)

33. Case study for hardware independence • Core touch tabletop operations • Rotate • Resize • Why are they critical? • How is this different from desktops? Mobiles? Walls? • How to implement? • Low level, basic software elements • Gesture design

34. Other recent interaction approaches CHI2013

35. High-precision pointing techniques capable of acquiring targets as small as 4 millimeters on a 5.5 meters wide display while leaving up to 93% of a typical tablet device’s screen space available for task-specific widgets. Head-Pad (head orientation) ARC-Pad (tablet/smartphone) High-Precision Pointing on Large Wall Displays using Small Handheld Devices Mathieu Nancel, Olivier Chapuis, Emmanuel Pietriga. Xing-Dong Yang Michel Beaudouin-Lafon1,2

36. StrikeAPose: Revealing Mid-Air Gestures on Public DisplaysRobert Walter, Gilles Bailly, Jorg Muller

37. SideWays: A Gaze Interface for Spontaneous Interaction with Situated Displays Yanxia Zhang, Andreas Bulling, Hans Gellersen

38. Information Capacity of Full-Body Movements AnttiOulasvirta, TeemuRoos, ArttuModig, Laura Leppänen

39. Body-centric Design Space for Multi-surface Interaction Julie Wagner, Mathieu Nancel, Sean Gustafson, StephaneHuot, Wendy E. Mackay

40. NailDisplay:Bringing an Always-Available Visual Display to Fingertips Chao-Huai Su, Liwei Chan, Chien-Ting Weng,Rong-Hao Liang, Kai-Yin Cheng∗ Bing-Yu Chen

41. Figure 1: WatchIt enables interacting with the wristband using simple gestures: (a) with a finger pointing on the internal strap, (b) with a fingerslidingon the internal strap, (c) with two fingers on opposite straps WatchIt: Simple Gestures and Eyes-free Interaction for Wristwatches and Bracelets Simon T. Perrault Eric Lecolinet James Eagan Yves Guiard

42. NUI… • Touch • User identification • Authentication • Gesture interaction • Speech • Gaze • Digital ecosystems • Carried devices • Embedded • Worn • Persistence of “files” • Brain….