1 / 91

Multi-View Stereo for Community Photo Collections

Multi-View Stereo for Community Photo Collections. Michael Goesele , Noah Snavely , Brian Curless , Hugues Hoppe, Steven M. Seitz. photos varies substantially in lighting, foreground clutter, scale due to various cameras, time, weather.

zaide
Download Presentation

Multi-View Stereo for Community Photo Collections

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multi-View Stereo for Community Photo Collections Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, Steven M. Seitz

  2. photos varies substantially in lighting, foreground clutter, scale due to various cameras, time, weather

  3. Images of Notre Dame (a variation in sampling rate of more than 1,000)

  4. Images taken in the wild—wide variety Lots of photographers Different cameras Sampling rates Occlusion Different time of day, weather Post processing

  5. The problem statement Design an adaptive view selection process Given the massive number of images, find a compatible subset Multi View Stereo (MVS) Reconstruct robust & accurate depth maps from this subset

  6. Previous work Global View Selection assume a relatively uniform viewpoint distribution and simply choose the k nearest images from each reference view Local View Selection use shiftable windows in time to adaptively choose frames to match

  7. CPC non-uniformly distributed in 7D viewpoint (translation, rotation, focal length) space • represents an extreme case of unorganized images sets Algorithm overview: - Calibrating Internet Photos - Global View Selection - Local View Selection - Multi-View Stereo Reconstruction

  8. Calibrating Internet Photos • PTLens extracts camera and lens information and corrects for radial distortion based on a database of camera and lens properties • Discard images cannot be corrected • Remaining images entered into a robust, metric structure-from-motion (SfM) system (uses SIFT feature detector) - generate a sparse scene reconstruction from the matched features - list of images where feature was detected Remove Radiometric Distortions - all input images into a linear radiometric space (sRGB color space)

  9. Global View Selection For each reference view R, global view selection seeks a set N of neighboring views that are good candidates for stereo matching in terms of scene content, appearance, and scale SIFT selects features with similar appearance - Shared feature points: collocation problem - Scale invariance: stereo matching problem need a measurement to deal these two problems !

  10. Global score gRfor each view V within a candidate neighborhood N (which includes R) FV: set of feature points in View V FV∩ FR: common feature points of View V and R wN(f): measure angular separation of two views, the larger, the more separated in angulation ws(f): measures similarity in scale of two views, the larger, the more similar in scale

  11. Calculating wN(f) α is the angle between the lines of sight from Vi and Vj to f αmax set to 10 degrees

  12. Calculating ws(f) r = sR(f) / sV(f) sR(f): diameter of a sphere centered at f whose projected diameter in view V equals the pixel spacing in V - favors the case 1 ≤ r <2

  13. Add scores of all feature points for all view V and select top N Rescaling views If scaleR(Vmin) is smaller than 0.6 (threshold), which means 5x5 R vs 3x3 V, need rescale Find lowest resolution view Vmin, resample R Resample view whose scaleR(V) > 1.2 to match the scale of R

  14. Local View Selection Global view selection determines a set N of good matching candidates for a reference view R Select a smaller set A∈N (|A|=4) of active views for stereo matching at a particular location in the reference view

  15. Stereo Matching Use nxn window centered on point in R Goal: To maximize photometric consistency of this patch to its projections into the neighboring views Scene Geometry Model Photometric Model

  16. Scene Geometry Model Window centered at pixel (s, t) oR is the center of projection of view R rR(s,t) is the normalized ray direction through the pixel Reference view corresponds to a point xR(s,t) at a distance h(s,t) along the viewing ray rR(s,t)

  17. Photometric Model Simple model for reflectance effects—a color scale factor ck for each patch projected into the k-th neighboring view • Models Lambertian reflectance for constant illumination over planar surfaces • Fails for shadow boundaries, caustics, specular highlights, bumpy surfaces

  18. Results Several Internet CPCs gather from Flickr varying widely in terms of size, number of photographers and scale

  19. Output

  20. Video demo

  21. Thank you!

  22. Reconstructing Building Interiors from Images Yasutaka Furukawa Brian Curless Steven M. SeitzUniversity of Washington, Seattle, USA Richard Szeliski Microsoft Research, Redmond, USA

  23. Reconstruction & Visualizationof Architectural Scenes • Manual (semi-automatic) • Google Earth & Virtual Earth • Façade[Debevec et al., 1996] • CityEngine [Müller et al., 2006, 2007] • Automatic • Ground-level images [Cornelis et al., 2008, Pollefeys et al., 2008] • Aerial images [Zebedin et al., 2008] Google Earth Virtual Earth Müller et al. Zebedin et al.

  24. Reconstruction & Visualizationof Architectural Scenes • Manual (semi-automatic) • Google Earth & Virtual Earth • Façade[Debevec et al., 1996] • CityEngine [Müller et al., 2006, 2007] • Automatic • Ground-level images [Cornelis et al., 2008, Pollefeys et al., 2008] • Aerial images [Zebedin et al., 2008] Google Earth Virtual Earth Müller et al. Zebedin et al.

  25. Reconstruction & Visualizationof Architectural Scenes Little attention paid to indoor scenes Google Earth Virtual Earth Müller et al. Zebedin et al.

  26. Our Goal • Fully automatic system for indoors/outdoors • Reconstructs a simple 3D model from images • Provides real-time interactive visualization

  27. Challenges - Reconstruction • Multi-view stereo (MVS) typically produces a dense model • We want the model to be • Simple for real-time interactive visualization of a large scene (e.g., a whole house) • Accurate for high-quality image-based rendering

  28. Challenges – Indoor Reconstruction Texture-poor surfaces Complicated visibility Prevalence of thin structures (doors, walls, tables)

  29. Outline • System pipeline (system contribution) • Algorithmic details (technical contribution) • Experimental results • Conclusion and future work

  30. System pipeline Images Images

  31. System pipeline Structure-from-Motion Bundler by Noah Snavely Structure from Motion for unordered image collections http://phototour.cs.washington.edu/bundler/ Images

  32. System pipeline Multi-view Stereo PMVS by Yasutaka Furukawa and Jean Ponce Patch-based Multi-View Stereo Software http://grail.cs.washington.edu/software/pmvs/ Images SFM

  33. System pipeline Manhattan-world Stereo [Furukawa et al., CVPR 2009] Images SFM MVS

  34. System pipeline Manhattan-world Stereo [Furukawa et al., CVPR 2009] Images SFM MVS

  35. System pipeline Manhattan-world Stereo [Furukawa et al., CVPR 2009] Images SFM MVS

  36. System pipeline Manhattan-world Stereo [Furukawa et al., CVPR 2009] Images SFM MVS

  37. System pipeline Manhattan-world Stereo [Furukawa et al., CVPR 2009] Images SFM MVS

  38. System pipeline Axis-aligned depth map merging (our contribution) Images SFM MVS MWS

  39. System pipeline Rendering: simple view-dependent texture mapping Images SFM MVS MWS Merging

  40. Outline • System pipeline (system contribution) • Algorithmic details (technical contribution) • Experimental results • Conclusion and future work

More Related