Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections

1. Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections Hui-Shan Kao

2. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

3. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

4. Introduction • Regular interactive surface only support • Multi-Touch • Tangible input • Extend more possibility on interactive surface • Add another display • Enable intuitive 3D manipulation

5. Introduction • Beyond-Surface Interactions • Base on regular interactive surface • Embedded IR markers • Invisible • 6DOFofIRcamera • pico projector • Tablet PC Location & Orientation IR Projector

6. Outline • Introduction • Related Work • Beyond-surface Interaction • Localization of Device • Invisible Projection • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

7. Beyond-surface Interaction • Second Light • Use an electronically switchable diffuser • Turn any translucent sheet above the surface into a mobile display via the second projection • With the camera that sees through the surface, it can localize a mobile panel in six-degrees. Izadi, S.etc Going beyond the display: a surface technology with an electronically switchable diffuser. In Proc UIST’08

8. Localization of Device • Enable3Dinteraction on tabletop display • Needtorecognizethe6DOFofdevice • Thewaytoknow6DOF • Magnetic tracker • Penlight • Vision based tracker • Handheld projector • JanusVF • Marker based tracker Visible

9. Invisible projection • Invisible projection • Spectrum • IR/Color • polarization • Time • high frequency • Synchronization of camera and projector • Encoding in content • Embedded code in color channel

10. Invisible projection • Hybrid Infrared and Visible Light Projection for Location Tracking • A projector capable of projecting visible images and infrared images • Using gray-coded pattern to locate the sensors. • Dynamic adaptation of projected imperceptible codes • Using high frequent temporal image modulation to project an invisible pattern Johnny Lee, etc. Hybrid Infrared and Visible Light Projection for Location Tracking In Proc UIST’07 A. Grundh¨ofer, etc.“Dynamic adaptation of projected imperceptible codes,” In Proc. ISMAR ’07

11. Outline • Introduction • Related Work • System Design • Hardware Configuration • Markers System • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

12. System Design • Goal: • Support multi-touchand multi-user • DI based touch detection • Add beyond surface interaction • Using invisible marker

13.  Hardware Configuration Tablet PC + IR Camera Picoprojector + IR Camera IRProjection ColorProjection ColorProjector IRCamera IRCamera IRProjector Mirror

14.  Hardware Design • The order of glass layer and diffuse layer • diffuse layer should on top • Not to degrade the luminance of pico projector • The reflection of pico projector may offending the user

15. Hardware Configuration • Problem: • IR rays will be reflected by the touch-glass and resulting in IR spot regions in camera views • Use two cameras to reduce the IR spot

16. Markers System • ARToolkitPlus • Fiducial marker • Self-identify • Enable error correct bit • Localization • Camera and projector calibration • Camera pose estimation

17. Outline • Introduction • Related Work • System Design • System Calibration • Tabletop System Calibration • Projector and Camera System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

18. Tabletop System Calibration • Original tabletop system calibration • Finding the homographs between table, camera, and projector • The corners are manually specified by the users • Time-consuming • Human intervention • Pixel-level accuracy

19. Tabletop System Calibration • Only need four points to be manually specified • Adding an additional IR-Color camera • Project predefined markers for calibration IRCamera IRProjector IR-ColorCamera Color Projector

20. Pico projector and camera calibration • The projector as an inverse camera • Mapping pixel from a 2D image into 3D rays • Using standard camera calibration procedure • Find the 3D points of the projected pattern and the 2D points of the image projected Rc , Tc Rp , Tp Rcp , Tcp

21. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • 3D posture estimation • Multi-Resolution Markers • Working with multi-touch • Application • Conclusion

22. Beyond Surface Implementation 3D posture?

23. 3D  Posture Estimation • ARToolKitPlus for 3D estimation • Camera looks within image for markers • Encode identity • Allow recovery of camera pose relative to marker

24. Multi-Resolution Markers • Uni-resolution marker • Camera could observe the markers too small or big • The marker with unfit size will not be recognized • Multi-resolution marker • Systemresizes the IR markers according to the camera’s posture

25. Marker Split and Merge • Marker Split • Not enough : split the markers into smaller size • Marker Merge • Too much : merge the markers for higher accuracy • How to re-arrange the layout of IR makers? • Ensure that camera will see at least 4 markers • Only re-arrange the layout in camera’s view field • The nearest camera will have high priority

26. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Foreground detection • Background Simulation • Software Synchronization • Application • Conclusion

27. Working with multi-touch • Traditional DI process • Take few frames for building background • subtract the background • obtain the foreground • IRmarkers projection will interrupt the traditional detection of multi-touch • Foreground can not be recognized

28. ROIGeneration 31 IR Camera of Mobile Device Real scene Observed Image IR Cameras Color Projector IR Projector Foregrounds Layout Manager Simulated Background ROI

29. Foreground Detection IR Camera of Mobile Device Applications Real scene Smoothing IR Cameras KalmanFiltering Color Projector IR Projector Finger Touches Observed Image Foregrounds Layout Manager Simulated Background Tangible Objects ROI Prediction

30. Software Synchronization • The camera and projectors are two independent systems • The simulated background will not synchronize to the capture image • Some of the markers will be treat as foreground Simulate BG Capture Image Real BG

31. Software Synchronization • Keep the simulated backgrounds in a buffer by time • Find the most similar background by subtraction Real BG Background Candidate queue BG Candidate Capture Image Real BG

32. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

33.  Application • 3 types of the application provide intuitive and natural manipulation. iLamp iFlash iView

34. iLamp • Goal: Project high-resolution content, bring more detailed and fine-grained information • Combine a Pico projector and an IR camera

35. iFlashlight • A mobile version of iLamp, can be moved easily.

36. iView • Tablet PC + IR Camera • An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives.

37. iView • Problem in iView • The user will lose the connection with the surface. • Adding the boundary of surface • Instruct user to manipulate the surface for farther information.

38. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

39. Conclusion • A new interactive surface based on the programmable invisible markers. • Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera. • Bring another level of information on interactive surface.