1 / 43

Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections

Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections. Hui -Shan Kao Advisor : Dr. Yi-Ping Hung. Outline. Introduction Related Work System Design Interaction Techniques Applications Conclusion. Outline. Introduction Related Work

chaka
Download Presentation

Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Supporting Beyond-Surface Interaction for TabletopDisplay Systems by Integrating IR Projections Hui-Shan Kao Advisor : Dr. Yi-Ping Hung

  2. Outline • Introduction • Related Work • System Design • Interaction Techniques • Applications • Conclusion

  3. Outline • Introduction • Related Work • System Design • Interaction Techniques • Applications • Conclusion

  4. Introduction • Interaction with surfaces mainly support • Multi-touch • Tangible input • Interaction beyond surfaces Combine with mobile displays iPhone iPad Pico Projector Tablet PC

  5. Introduction • Two scenarios • Type 1: • Using pico projector • Multi-resolutionpresentation • Augmenting personal information • Type 2: • Using tablet PC • Interactive 3D viewer Type2 Type1

  6. Challenge How to know the 3D spatial relationship with surface in real-time? Type2 Type1 ?

  7. Outline • Introduction • Related Work • System Design • Interaction Techniques • Applications • Conclusion

  8. Localization of Device • Enable3Dinteraction on tabletop display • Needtorecognizethe6DOFofdevice • Thewaytoknow6DOF • Magnetic tracker • Penlight • H.Song, T.Grossman, G. Fitzmaurice, F. Guimbretiere, A. Khan, R. Attar, and G. Kurtenbach. Penlight: combining a mobile projector and a digital pen for dynamic visual overlay. In Proc. CHI ’09, 2009

  9. Localization of Device • Vision based tracker • Markers with known structure • Interactive handheld projector • ARToolkit VisibleMarker ! • Xiang Cao, Clifton Forlines, and RavinBalakrishnan. Multi-user interaction using handheld projectors. In Proc. UIST ’07, 2007. • D. Wagner and D. Schmalstieg. ARToolKitPlus for Pose Tracking on Mobile Devices. In Proc CVWW’07: • Proceedings of 12th Computer Vision Winter Workshop, 2007.

  10. Outline • Introduction • Related Work • System Design • Hardware Configuration • Interaction Techniques • Applications • Conclusion

  11.  Hardware Configuration IR Camera IRProjection  invisible markers ColorProjection  visible content ColorProjector IRCamera IRCamera IRProjector Mirror

  12.  Hardware Configuration • Multi-touch Surface • Glass layer  support touch force • Diffuser layer  display image • Placement of glass layer and diffuser layer • Glass on top of diffuser • Diffuser on top of glass

  13.  Hardware Configuration  • Option 1: Glass on top of diffuser pico-projection reflection • degrade illumination • damage user’s eye Pico-projector IRcamera touch-glass diffuser Hard to Solve! IR & colorprojector IRcamera

  14.  Hardware Configuration  • Option 2: Diffuser on top of Glass IR-projection reflection • spot effect in IR images Pico-projector diffuser touch-glass IR & color projector spot spot IRcamera IRcamera

  15. Hardware Configuration • Removing IR spot by using two cameras IRCamera 1 IRCamera 2 Weighting Mask (1) IR Camera(1) Stitched View Weighting Mask (2) IR Camera(2)

  16. Outline • Introduction • Related Work • System Design • Interaction Techniques • Adaptive Markers for Camera Estimation • Dynamic Markers for Multi-Touch • Applications • Conclusion

  17. 3D Position Estimation Type2 Type1 3D position? 3D position?

  18. 3D Position Estimation • ARToolKitPlus for 3D estimation • Fiducial marker • Self-identify by ID • Projecting a grid of markers on tabletop • Each marker with • Unique ID • Associated position • Compute the camera’s position based on the marker perceived (0,0) ID = 0 (1,1)

  19. Multi-Level Markers • Uni-level marker • Camera might observe the markers too small or too big • Multi-level marker • Systemresizes the IR markers according to camera position

  20. Marker Split and Merge • Adapting the maker size • Markers merge, when camera moves far enough • Markers split, when camera moves too close • Ensure camera to see at least 4 markers

  21. Cooperating with multi-touch • Marker on for camera positioning • Marker off for finger detection • Foreground-ROI detection for marker on/off

  22. Cooperating with multi-touch ?

  23. Cooperating with multi-touch

  24. Background Simulation • Offline • Save each marker as a patch image and record the position of marker (offline) :: for each marker _ = patch patch collection base view

  25. Background Simulation • Online • As the layout re-arrange, the simulated background can be built by the saving patch in real time (online) :: for each marker in layout ( ) _ = base view patch of marker simulated background

  26. Background Simulation

  27. ROIGeneration 1st frame after finger touch 2nd and the other frames

  28. Summary of Foreground Detection IR Camera of Mobile Device Applications Real scene Smoothing IR Cameras KalmanFiltering Color Projector IR Projector Observed Image Finger Touches Foregrounds Layout Manager Simulated Background Tangible Objects ROI Prediction

  29. Outline • Introduction • Related Work • System Design • Interaction Techniques • Applications • Conclusion

  30. Three Applications • Three applications provide intuitive and natural manipulation. iLamp iFlashlight iView

  31. iLamp

  32. iLamp • Combine a pico projector and an IR camera • Project seamless high-resolution content, bringing more detailed information • Act as a desk lamp for personal use

  33. iLamp RpcTpc RpcTpc • [Rp|Tp ] = [Rpc|Tpc ]x[Rc|Tc ] Real-time RcTc ? RpTp

  34. Pico projector and camera calibration • Off-line : Find transformation between camera and pico projector • The projector can be viewed as a dual of camera • RpcTpcare the Rotation and translation of the attached camera RpcTpc

  35. Pico projector and camera calibration • On-line : Find transformation between pico projector and tabletop • Estimate RcTcin real-time • Compute RpTp • [Rp|Tp ] = [Rpc|Tpc ]x[Rc|Tc ] RpcTpc

  36. iFlashlight

  37. iFlashlight • A mobile version of iLamp, can be moved easily. • Multi-user cooperation

  38. iView

  39. iView • An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives.

  40. Outline • Introduction • Related Work • System Design • Interaction Techniques • Applications • Conclusion

  41. Conclusion • A new interactive surface based on the programmable invisible markers. • Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera. • Bring another level of information on interactive surface.

  42. Thank you

More Related