1 / 22

photosynth/view.aspx?cid=82e0166f-0367-47a8-abf4-87a075bb347e

Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA. Image-based maps. http://www.photosynth.net/view.aspx?cid=82e0166f-0367-47a8-abf4-87a075bb347e. Key Steps. Structure from motion (Sfm)

deborah-meyers
Download Presentation

photosynth/view.aspx?cid=82e0166f-0367-47a8-abf4-87a075bb347e

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Recent work in image-based rendering from unstructured image collections and remaining challengesSudipta N. SinhaMicrosoft Research, Redmond, USA

  2. Image-based maps • http://www.photosynth.net/view.aspx?cid=82e0166f-0367-47a8-abf4-87a075bb347e

  3. Key Steps • Structure from motion (Sfm) • Robust depth-map estimation • Rendering

  4. Recent results • Structure from motion (Sfm) • Robust depth-map estimation • Image-based navigation A multi-stage linear approach to structure from motion Sinha, Steedly & Szeliski, RMLE –ECCV workshop 2010 Piecewise planar stereo for image-based rendering Sinha, Steedly & Szeliski, ICCV 2009 Image-based walkthroughs from incremental and partial scene reconstructions Kumar, Ahsan, Sinha & Jawahar, BMVC 2010

  5. Sequential Sfm Fitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10

  6. Sequential Sfm Fitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10 Initial seed pair Pose estimation, triangulation Refinement

  7. Sequential Sfm Fitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10 Initial seed pair Pose estimation, triangulation Refinement

  8. Linear multi-stage approach to structure from motion Sinha et. al. 2010 (ECCV-RMLE workshop) Contributions • Vanishing point (VP) constraints reduces drift in rotations • more accurate than [Govindu’04, Martinec’07] for urban scenes. • Faster pairwise matching + geometric verification • New practical linear structure and translation estimation • more stable than the known linear method [Rother’03] • robust to outliers in 2D observations • easy to parallelize • faster than sequential Sfm • much faster than L∞ - methods

  9. Linear multi-stage approach to structure from motion Sinha et. al. 2010 (ECCV-RMLE workshop) Linear Reconstruction Images 2-view Reconstruction Robust Alignment Global Scale & Translation Estimation Feature Extraction interest pts relative rotations Pair Matching 2 – VP + 2 point RANSAC Vanishing Point (VP) Detection VPs VP tracks relative pose estimates VP tracks Global Rotation Estimation relative rotations global camera orientations Full Sfm initialization Final Bundle Adjustment

  10. Results

  11. Timings

  12. Break-up of Timings

  13. Comparison with sequential Sfm

  14. Comparison with sequential Sfm STREET sequence OURS (65 cams, 52K pts) before Bundle Adjustment BUNDLER (65 cams, 22K pts) HALLWAY sequence BUNDLER (139 cams, 13K pts) OURS (184 cams, 27K pts)

  15. Piecewise Planar Stereo for image-based rendering Sinha et. al. ICCV 2009 Feature matching Graph-cut based energy minimization

  16. Piecewise Planar Stereo for image-based rendering Sinha et. al. ICCV 2009 Planar Stereo Results

  17. Piecewise Planar Stereo for image-based rendering also handle non-planar scenes now ...

  18. Image-based walkthroughs from incremental and partial scene reconstructions Kumar et. al. BMVC 2010 • Skip global scene reconstruction (Sfm) step, • Generate several overlapping, partial reconstructions instead. • During navigation, jump between local coordinate frames. • Scales easily, also parallelizable • Incremental matching & reconstruction (images appear over time) Fort sequence (~5800 images)

  19. Existing issues in unstructured Sfm • Accuracy vs. Connectedness • Reliable results from sparse, unstructured imagery • wide-baseline matching is still difficult • Representations: • metric vs. topological reconstructions ? hybrid ? • Reconstructing Indoors • Bottlenecks: doorways, corridors. • fewer features, non-Lambertian surfaces

  20. Dynamic Image-based Maps: Challenges • Acquisition • Images vs. video • Short-term dynamics vs. long-term dynamics • Need truly incremental Sfm • Start with scratch but keep going … ? • Interleaving matching, Sfm and dense stereo • Hybrid matching (2D—2D , 2D – 3D, 3D – 3D)

  21. Dynamic Image-based Maps: Challenges • Temporal appearance changes • Illumination: • day/night, seasons, weather, lights on/off • Cyclic, predictable • Albedo changes • Store-fronts, ads-billboards, • irreversible • Geometric changes: • temporary vs. permanent • Mid-level features for higher level recognition

  22. Questions ?

More Related