1 / 46

Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections

Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections. Hui -Shan Kao. Outline. Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion. Outline. Introduction

damian
Download Presentation

Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections Hui-Shan Kao

  2. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

  3. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

  4. Introduction • Regular interactive surface only support • Multi-Touch • Tangible input • Extend more possibility on interactive surface • Add another display • Enable intuitive 3D manipulation

  5. Introduction • Beyond-Surface Interactions • Base on regular interactive surface • Embedded IR Markersincontent • 6DOFofIRcamera • Invisible Location & Orientation IR Projector

  6. Outline • Introduction • Related Work • 3D Interactive Tabletop • Localization of Device • Invisible Projection • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

  7. 3D Interactive Tabletop • Second Light • Use an electronically switchable diffuser • Turn any translucent sheet above the surface into a mobile display via the second projection • With the camera that sees through the surface, it can localize a mobile panel in six-degrees. Izadi, S.etc Going beyond the display: a surface technology with an electronically switchable diffuser. In Proc UIST’08

  8. Localization of Device • Enable3Dinteraction on tabletop display • Needtorecognizethe6DOFofdevice • Thewaytoknow6DOF • Magnetic tracker • Penlight • Vision based tracker • Handheld projector • JanusVF

  9. Invisible projection • Invisible projection • Spectrum • IR/Color • polarization • Time • high frequency • Synchronization of camera and projector • Encoding in content • Embedded code in color channel

  10. Invisible projection – Spectrum • Hybrid Infrared and Visible Light Projection for Location Tracking • a projector capable of projecting • visible images for application content • invisible infrared images for location discovery and tracking • Using Gray-coded pattern to locate the sensors. Johnny Lee, etc. Hybrid Infrared and Visible Light Projection for Location Tracking In Proc UIST’07

  11. Invisible projection – Frequency • Dynamic adaptation of projected imperceptible codes • Using high frequent temporal image modulation to project an adaptive invisible pattern • Invisible to the human eye • Reconstructed by a synchronized camera • Dynamic multi-resolution marker tracking method • Ensure a continuous camera tracking • Adapt to the camera pose A. Grundh¨ofer, etc.“Dynamic adaptation of projected imperceptible codes,” In Proc. ISMAR ’07 (International Symposiumon Mixed and Augmented Reality)

  12. Outline • Introduction • Related Work • System Design • Hardware Configuration • Markers System • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

  13. System Design • Goal: • Support multi-touchand multi-user • DI based touch detection • Add beyond surface interaction • Using invisible marker tracking

  14.  Hardware Configuration IRProjection ColorProjection ColorProjector IRCamera IRCamera IRProjector Mirror

  15.  Hardware Design • The order of glass layer and diffuse layer • diffuse layer should on top • Not to degrade the luminance of pico projector • The reflection of pico projector may offending the user

  16. Hardware Configuration • Problem: • IR rays will be reflected by the touch-glass and resulting in IR spot regions in camera views • Use two cameras to reduce the IR spot

  17. Markers System • ARToolkitPlus • Fiducial marker • Self-identify • Enable error correct bit • Localization • Camera and projector calibration • Camera pose estimation

  18. Outline • Introduction • Related Work • System Design • System Calibration • Tabletop System Calibration • Projector and Camera System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

  19. Tabletop System Calibration • Original tabletop system calibration • Finding the homographs between table, camera, and projector • The corners are manually specified by the users • Time-consuming • complicate • Pixel-level accuracy

  20. Tabletop System Calibration • Only need four points to be manually specified • Adding an additional IR-Color camera • Project predefined markers for calibration User IR-Color Camera IR Camera IR Projector Color Projector Color-IR Camera

  21. Pico projector and camera calibration • The projector as an inverse camera • Mapping pixel from a 2D image into 3D rays • Using standard camera calibration procedure • Find the 3D points of the projected pattern and the 2D points of the image projected Rc , Tc Rp , Tp Rcp , Tcp

  22. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • 3D posture estimation • Multi-Resolution Markers • Working with multi-touch • Application • Conclusion

  23. Beyond Surface Implementation

  24. 3D  Posture Estimation • ARToolKitPlus for 3D estimation • Camera looks within image for markers • Markers are predefined base on the table coordinate • Encode identity • Allow recovery of camera pose relative to marker • To improve the marker detection • Adaptive threshold • Non-uniform distribution of IRprojection • over-exposed of camera • Kalman Filtering • smooth the estimation

  25. Multi-Resolution Markers • Uni-resolution marker • Camera could observe the markers too small or big • The marker with unfit size will not be recognized • Multi-resolution marker • Systemresizes the IR markers according to the camera’s posture

  26. Marker Split and Merge • Marker Split • Not enough : split the markers into smaller size • Marker Merge • Too much : merge the markers for higher accuracy • How to re-arrange the layout of IR makers? • Ensure that camera will see at least 4 markers • Only re-arrange the layout in camera’s view field • The nearest camera will have high priority

  27. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Foreground detection • Background Simulation • Software Synchronization • Application • Conclusion

  28. Working with multi-touch • Traditional DI process • Take few frames for building background • subtract the background • obtain the foreground • IRmarkers projection will interrupt the traditional detection of multi-touch • Foreground can not be recognized

  29. Marker On and Off • Divide the marker into two stages • Marker onfor camera positioning • Marker off for finger detection • ROI detection control marker on/off ROI Without Marker Off With Marker Off

  30. Foreground Detection • Define ROI • Simply subtract the background • Some markers still remain • Background is not similar to real • Simulate background • Foreground detection only perform in ROI • Reduce the computation

  31. Background Simulation • Background simulation • Background is dynamically re-arrange • Model the background in real-time • Pixellevel • Pixel will interactive influence • Tag level • Interactive influence between tags can be thought as a global variance

  32. Background Simulation • Off-line process of simulation • Since the layout of markers are predefined • Saving each marker as a patch image and record the position of marker • As the layout re-arrange, the simulating background can be built by the saving patch and their position

  33. IR Camera of Mobile Device Applications Real scene Smoothing IR Cameras KalmanFiltering Color Projector IR Projector Finger Touches Observed Image Foregrounds Layout Manager Simulated Background Tangible Objects ROI Prediction

  34. Software Synchronization • The camera and projectors are two independent systems • The simulated background will not synchronize to the capture image • Some of the markers will be treat as foreground Simulate BG Capture Image Real BG

  35. Software Synchronization • Keep the simulated backgrounds in a buffer by time • Find the most similar background by subtraction Real BG Background Candidate queue BG Candidate Capture Image Real BG

  36. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

  37.  Application • 3 types of the application provide intuitive and natural manipulation. iLamp iFlash iView

  38. iLamp • Goal: Project high-resolution content, bring more detailed and fine-grained information • Combine a Pico projector and an IR camera

  39. iFlashlight • A mobile version of iLamp, can be moved easily.

  40. iView • Tablet PC + IR Camera • An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives.

  41. iView • Problem in iView • The user will lose the connection with the surface. • Adding the boundary of surface • Instruct user to manipulate the surface for farther information.

  42. Outline • Introduction • Related Work • System Design • System Calibration • Beyond Surface Implementation • Working with multi-touch • Application • Conclusion

  43. Conclusion • A new interactive surface based on the programmable invisible markers. • Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera. • Bring another level of information on interactive surface.

More Related