Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections

1. Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections Hui-Shan Kao

2. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

3. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

4. Introduction • Regular interactive surface only support • Multi-Touch • Tangible input • Extend more possibility on interactive surface • Add another display • Enable intuitive 3D manipulation

5. Introduction • Beyond-Surface Interactions • Base on regular interactive surface • Embedded IR Markersincontent • 6DOFofIRcamera • Invisible Location & Orientation IR Projector

6. Outline • Introduction • Related Work • 3D Interactive Tabletop • Localization of Device • Invisible Projection • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

7. 3D Interactive Tabletop • Second Light • Use an electronically switchable diffuser • Turn any translucent sheet above the surface into a mobile display via the second projection • With the camera that sees through the surface, it can localize a mobile panel in six-degrees. Izadi, S.etc Going beyond the display: a surface technology with an electronically switchable diffuser. In Proc UIST’08

8. Localization of Device • Enable3Dinteraction on tabletop display • Needtorecognizethe6DOFofdevice • Thewaytoknow6DOF • Magnetic tracker • Penlight • Vision based tracker • Handheld projector • JanusVF

9. Invisible projection • Invisible projection • Spectrum • IR/Color • polarization • Time • high frequency • Synchronization of camera and projector • Encoding in content • Embedded code in color channel

10. Invisible projection – Spectrum • Hybrid Infrared and Visible Light Projection for Location Tracking • a projector capable of projecting • visible images for application content • invisible infrared images for location discovery and tracking • Using Gray-coded pattern to locate the sensors. Johnny Lee, etc. Hybrid Infrared and Visible Light Projection for Location Tracking In Proc UIST’07

11. Invisible projection – Frequency • Dynamic adaptation of projected imperceptible codes • Using high frequent temporal image modulation to project an adaptive invisible pattern • Invisible to the human eye • Reconstructed by a synchronized camera • Dynamic multi-resolution marker tracking method • Ensure a continuous camera tracking • Adapt to the camera pose A. Grundh¨ofer, etc.“Dynamic adaptation of projected imperceptible codes,” In Proc. ISMAR ’07 (International Symposiumon Mixed and Augmented Reality)

12. Outline • Introduction • Related Work • System Design • Hardware Configuration • Markers System • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

13. System Design • Goal: • Support multi-touchand multi-user • DI based touch detection • Add beyond surface interaction • Using invisible marker tracking

14.  Hardware Configuration IRProjection ColorProjection ColorProjector IRCamera IRCamera IRProjector Mirror

15.  Hardware Design • The order of glass layer and diffuse layer • diffuse layer should on top • Not to degrade the luminance of pico projector • The reflection of pico projector may offending the user

16. Hardware Configuration • Problem: • IR rays will be reflected by the touch-glass and resulting in IR spot regions in camera views • Use two cameras to reduce the IR spot

17. Markers System • ARToolkitPlus • Fiducial marker • Self-identify • Enable error correct bit • Localization • Camera and projector calibration • Camera pose estimation

18. Outline • Introduction • Related Work • System Design • System Calibration • Tabletop System Calibration • Projector and Camera System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

19. Tabletop System Calibration • Original tabletop system calibration • Finding the homographs between table, camera, and projector • The corners are manually specified by the users • Time-consuming • complicate • Pixel-level accuracy

20. Tabletop System Calibration • Only need four points to be manually specified • Adding an additional IR-Color camera • Project predefined markers for calibration User IR-Color Camera IR Camera IR Projector Color Projector Color-IR Camera

22. Pico projector and camera calibration • The projector as an inverse camera • Mapping pixel from a 2D image into 3D rays • Using standard camera calibration procedure • Find the 3D points of the projected pattern and the 2D points of the image projected Rc , Tc Rp , Tp Rcp , Tcp

23. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • 3D posture estimation • Multi-Resolution Markers • Working with multi-touch • Application • Conclusion

24. Beyond Surface Implementation

25. 3D  Posture Estimation • ARToolKitPlus for 3D estimation • Camera looks within image for markers • Markers are predefined base on the table coordinate • Encode identity • Allow recovery of camera pose relative to marker • To improve the marker detection • Adaptive threshold • Non-uniform distribution of IRprojection • over-exposed of camera • Kalman Filtering • smooth the estimation

26. Multi-Resolution Markers • Uni-resolution marker • Camera could observe the markers too small or big • The marker with unfit size will not be recognized • Multi-resolution marker • Systemresizes the IR markers according to the camera’s posture

27. Marker Split and Merge • Marker Split • Not enough : split the markers into smaller size • Marker Merge • Too much : merge the markers for higher accuracy • How to re-arrange the layout of IR makers? • Ensure that camera will see at least 4 markers • Only re-arrange the layout in camera’s view field • The nearest camera will have high priority

28. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Foreground detection • Background Simulation • Software Synchronization • Application • Conclusion

29. Working with multi-touch • Traditional DI process • Take few frames for building background • subtract the background • obtain the foreground • IRmarkers projection will interrupt the traditional detection of multi-touch • Foreground can not be recognized

30. Marker On and Off • Divide the marker into two stages • Marker onfor camera positioning • Marker off for finger detection • ROI detection control marker on/off ROI Without Marker Off With Marker Off

31. Foreground Detection • Define ROI • Simply subtract the background • Some markers still remain • Background is not similar to real • Simulate background • Foreground detection only perform in ROI • Reduce the computation

32. Background Simulation • Background simulation • Background is dynamically re-arrange • Model the background in real-time • Pixellevel • Pixel will interactive influence • Tag level • Interactive influence between tags can be thought as a global variance

33. Background Simulation • Off-line process of simulation • Since the layout of markers are predefined • Saving each marker as a patch image and record the position of marker • As the layout re-arrange, the simulating background can be built by the saving patch and their position

35. IR Camera of Mobile Device Applications Real scene Smoothing IR Cameras KalmanFiltering Color Projector IR Projector Finger Touches Observed Image Foregrounds Layout Manager Simulated Background Tangible Objects ROI Prediction

36. Software Synchronization • The camera and projectors are two independent systems • The simulated background will not synchronize to the capture image • Some of the markers will be treat as foreground Simulate BG Capture Image Real BG

37. Software Synchronization • Keep the simulated backgrounds in a buffer by time • Find the most similar background by subtraction Real BG Background Candidate queue BG Candidate Capture Image Real BG

38. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

39.  Application • 3 types of the application provide intuitive and natural manipulation. iLamp iFlash iView

40. iLamp • Goal: Project high-resolution content, bring more detailed and fine-grained information • Combine a Pico projector and an IR camera

41. iFlashlight • A mobile version of iLamp, can be moved easily.

42. iView • Tablet PC + IR Camera • An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives.

43. iView • Problem in iView • The user will lose the connection with the surface. • Adding the boundary of surface • Instruct user to manipulate the surface for farther information.

44. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

45. Conclusion • A new interactive surface based on the programmable invisible markers. • Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera. • Bring another level of information on interactive surface.