1 / 23

Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras

Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras. Yanli Liu 1,2 and Xavier Granier 2,3,4 1: College of Computer Science, Sichuan University, P.R.China 2: INRIA Bordeaux Sud-Ouest, France 3: LP2N (CNRS, Univ. Bordeaux, Institut d'Optique)

nirav
Download Presentation

Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Online Tracking of Outdoor Lighting Variations for Augmented Reality with Moving Cameras Yanli Liu1,2 and Xavier Granier2,3,4 1: College of Computer Science, Sichuan University, P.R.China 2: INRIA Bordeaux Sud-Ouest, France 3:LP2N (CNRS, Univ. Bordeaux, Institut d'Optique) 4:LaBRI (CNRS, University of Bordeaux)

  2. Motivation • Augmented reality • mobility

  3. Motivation • Two consistency • Geometric consistency • Devices • Camera position • GPS, UWB, Omnisense • WiFi, cell information • Camera pose • Linear accelerometers • Tracking via computer vision • [Cornelis et al. LNCS 2001, zhang et al. CVPR 2007, Xu et al. image and Vision Computing 2008] • Illumination consistency • outdoor lighting is largely dependent on weather and time

  4. Motivation • Two problems • Online process • first step toward real-time solutions • Moving viewpoints • Handhold camera jitter

  5. Previous Work • Markers or lighting probes[Debevec Siggraph’ 98, Agusanto ISMAR’03, Kanbara ICPR’04, Hensley I3D’07] • too dense sampling • our method does not require any supplemental devices Debevec Siggraph’ 98

  6. Previous Work • Three components of shading • BRDF • geometry • lighting • Fix other one or two components [Wang PG’02, Li ICCV’03, Hara PAMI’05, Andersen ICPR’06, Sun ICCV’09] • 3D reconstruction • controlled environment (indoor or lab) rendered image original image [Wang PG’02]

  7. Previous Work • Time-lapse outdoor video analysis [Sunkavalli Siggraph’07, Sunkavalli CVPR 08] • take whole video sequence as input • Post-processing [Sunkavalli Siggraph’07]

  8. Previous Work • Learning based outdoor illumination estimation [Liu TVC’09, Liu CAVW’10, Xing C&G’11] • offline stage learning • fixed viewpoint • moving viewpoints Liu CAVW’10

  9. Our Method • Key ideas • Tracking illumination variation by tracking feature points • Feature points tracking is error prone. • Select reliablefeature points using global illumination constraint and spatial-temporal coherency.

  10. Illumination and BRDF model • Outdoor lighting [Sunkavalli SIG’07, Sunkavalli CVPR 08, Madsen InTech 2010] • the sunlight • directional light • colored intensity • sun direction • the skylight • ambient light • colored intensity

  11. Illumination and BRDF model • Neutral reflection model [Lee PAMI’90, Montoliu LNCS’05, Eibenberger ICIP 2010, ICCV 2011] • the color of the specular reflection is the same as the color of the incident lighting. • Phong-like model

  12. System Initialization • Tracking illumination variation by tracking feature points • 3D geometry vs normals • planar feature points plane segmentation [Hoiem IJCV’07] KLT feature-points mean-shift color segmentation first frame threshold-based Shadow detection clustered feature-points

  13. System Initialization • BRDF initialization • pixels difference at in sun lit regions depend on specular parameters and : • Assuming piecewise constant , and • Spatially varying diffuse

  14. Tracking Lighting Variation with Reliable Feature Points • Energy function • Outdoor lighting is nearly constant during time intervals less than 1/5 second. Alignment-based weight control the smooth degree of skylight

  15. Tracking Reliable Features and Their Attributes • Feature points labeling • Three attributes: • Normal (plane, homography matrix) • BRDF parameters • Shadow situation Spatial & temporal coherency

  16. Tracking Reliable Features and Their Attributes • Feature points labeling current point is not in shadow paired point is labeled in compute lighting t -1 t

  17. Results and Discussion • Quantitative results • PC: Intel i7 2.67GHz and 6GB RAM • MATLAB • Video resolution 640 480 Average fps and average number of feature points estimated on 1,000 frames

  18. Results and discussion • Quantitative results Average percentage of different steps in total computational cost

  19. Results and Discussion • Visual results • Building scene • Wall scene

  20. Conclusion • Fully image-based pipeline • online tracking of lighting variations of outdoor videos. • Manages lighting changes and misalignment of feature points • Ensure a stable estimation on a sparse set feature points.

  21. Limitations and Future Work • Rough shadow detection • 3D reconstruction vs shadow detection • Sun-lit features • Initialization • automatic initialization: easy but may fail in some cases • manual initialization: may be tedious for a non-expert user. • Semi-assisted initialization

  22. Limitations and Future Work • Tracking independently on R, G, and B channels • priori model of outdoor illumination color or spectra • difficult to optimization • The first step of a long march to a seamless and real-time AR solution for videos with moving viewpoints.

  23. Thanks for your attention! Questions?

More Related