1 / 33

3D Reconstruction of Objects behind Dense Occlusions

3D Reconstruction of Objects behind Dense Occlusions. Vaibhav Vaish. Joint work with Rick Szeliski, Larry Zitnick, Sing Bing Kang, Brian Curless. SAP: Seeing through occlusions. Images: 82 Aperture: 1m. SAP: Seeing through occlusions. Images: 45 Aperture: 2m. SAP: Crowd scene.

sai
Download Presentation

3D Reconstruction of Objects behind Dense Occlusions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. 3D Reconstruction of Objects behind Dense Occlusions Vaibhav Vaish Joint work with Rick Szeliski, Larry Zitnick, Sing Bing Kang, Brian Curless

  2. SAP: Seeing through occlusions Images: 82 Aperture: 1m

  3. SAP: Seeing through occlusions Images: 45 Aperture: 2m

  4. SAP: Crowd scene Images: 60 Aperture: 90o

  5. SAP: Crowd scene Images: 90 Aperture: 3m

  6. Synthetic Aperture: Strengths • Can see through occluders of high density • No segmentation of foreground required • Works with noisy images, no color calibration required If we have enough cameras spanning a wide enough aperture, we can see through occlusions

  7. 3D Reconstruction • Challenges: • Find 3D location and shape of occluded objects • Construct an image with non-planar focal surface so that all occluded objects are in focus • Questions: • Does shape from focus do better than shape from stereo ? • Can we design alternatives to focus / stereo which are more robust in the presence of occlusions ?

  8. Outline • 3D Reconstruction via plane sweep • Stereo v/s Focus • Alternative methods • Results • Future Work

  9. z Plane Sweep: Stereo

  10. Plane Sweep: Stereo z

  11. Plane Sweep: Stereo Computing depth(x,y): for each depth z for each pixel (x,y) V(x,y,z)  variance of rays through (x,y,z) (x,y) is assigned the depth z for which the variance is minimum z

  12. Plane Sweep: Focus z

  13. Plane Sweep: Focus z

  14. Plane Sweep: Focus Computing depth(x,y): for each depth z compute SAP image focused at depth z for each pixel (x,y) F(x,y,z)  gradient of SAP image at (x,y) (x,y) is assigned the depth z for which F(x,y,z) is maximum z

  15. Depth Estimation Framework • Initial depth estimate • For each depth assignment (x, y, d), compute a cost measure • For each pixel, find the minimum cost depth • Global Optimization • Graph cuts, belief propagation • Enforce smoothness constraints • Iterative Refinement

  16. Variance vs. focus: intensity ramp

  17. Variance vs. focus: intensity ramp In regions of constant gradient, variance can recover correct depth, but focus is unresponsive. Focus requires non-zero 3rd order gradients to recover depth.

  18. Range of depths Occlusions Depth of background object Foreground occluders

  19. Median Color • Both the stereo and focus operators use the mean as an estimate of surface color • In presence of occlusions the mean may not be an accurate measure – even when the depth is right • Could we do better by using a more robust measure like median ?

  20. Median Color Computing depth(x,y): for each depth z for each pixel (x,y) compute median color of rays through (x,y,z) M(x,y,z)  sum of distances to median (x,y) is assigned the depth z which minimizes M(x,y,z)

  21. Entropy At the correct depth, the intensity histogram should be peaked

  22. Entropy At incorrect depths, the histogram should be more uniform Use histogram entropy H = - x log x as a cost function

  23. Test Scene 105 images (21x5 light field) 60cm x 10cm synthetic aperture Image Resolution: 650x515

  24. Synthetically Focused Images

  25. Synthetic Focus Mean color at estimated depth

  26. Variance Mean color at estimated depth

  27. Median Color Median color at estimated depth

  28. Entropy Mean color at estimated depth

  29. - = +…+ = Adding per-view occlusion maps yields occlusion map for points on the CD surface Comparison: Ground truth and Occlusion Map

  30. Performance vs. Occlusion

  31. Demo: Peek

  32. Future Work • Compare with algorithms that try to delete occluding objects • Voxel Coloring [Seitz, CVPR 97] • Analyze effect of different kinds of occluders • color distribution, feature size • Aperture shape, size and sampling • Impact of number of cameras • Add global optimization

  33. Acknowledgements • Rick, Larry, Sing Bing, Brian … • Fellow Interns (Jan, Noah, Ashish, Karteek, Ashley, Le, Manuel …) • Assistance in acquisition • Augusto Roman, David Koller, Neel Joshi

More Related