3D Reconstruction of Objects behind Dense Occlusions

1. 3D Reconstruction of Objects behind Dense Occlusions Vaibhav Vaish Joint work with Rick Szeliski, Larry Zitnick, Sing Bing Kang, Brian Curless

2. SAP: Seeing through occlusions Images: 82 Aperture: 1m

3. SAP: Seeing through occlusions Images: 45 Aperture: 2m

4. SAP: Crowd scene Images: 60 Aperture: 90o

5. SAP: Crowd scene Images: 90 Aperture: 3m

6. Synthetic Aperture: Strengths • Can see through occluders of high density • No segmentation of foreground required • Works with noisy images, no color calibration required If we have enough cameras spanning a wide enough aperture, we can see through occlusions

7. 3D Reconstruction • Challenges: • Find 3D location and shape of occluded objects • Construct an image with non-planar focal surface so that all occluded objects are in focus • Questions: • Does shape from focus do better than shape from stereo ? • Can we design alternatives to focus / stereo which are more robust in the presence of occlusions ?

8. Outline • 3D Reconstruction via plane sweep • Stereo v/s Focus • Alternative methods • Results • Future Work

9. z Plane Sweep: Stereo

10. Plane Sweep: Stereo z

11. Plane Sweep: Stereo Computing depth(x,y): for each depth z for each pixel (x,y) V(x,y,z)  variance of rays through (x,y,z) (x,y) is assigned the depth z for which the variance is minimum z

12. Plane Sweep: Focus z

13. Plane Sweep: Focus z

14. Plane Sweep: Focus Computing depth(x,y): for each depth z compute SAP image focused at depth z for each pixel (x,y) F(x,y,z)  gradient of SAP image at (x,y) (x,y) is assigned the depth z for which F(x,y,z) is maximum z

15. Depth Estimation Framework • Initial depth estimate • For each depth assignment (x, y, d), compute a cost measure • For each pixel, find the minimum cost depth • Global Optimization • Graph cuts, belief propagation • Enforce smoothness constraints • Iterative Refinement

16. Variance vs. focus: intensity ramp

17. Variance vs. focus: intensity ramp In regions of constant gradient, variance can recover correct depth, but focus is unresponsive. Focus requires non-zero 3rd order gradients to recover depth.

18. Range of depths Occlusions Depth of background object Foreground occluders

19. Median Color • Both the stereo and focus operators use the mean as an estimate of surface color • In presence of occlusions the mean may not be an accurate measure – even when the depth is right • Could we do better by using a more robust measure like median ?

20. Median Color Computing depth(x,y): for each depth z for each pixel (x,y) compute median color of rays through (x,y,z) M(x,y,z)  sum of distances to median (x,y) is assigned the depth z which minimizes M(x,y,z)

21. Entropy At the correct depth, the intensity histogram should be peaked

22. Entropy At incorrect depths, the histogram should be more uniform Use histogram entropy H = - x log x as a cost function

23. Test Scene 105 images (21x5 light field) 60cm x 10cm synthetic aperture Image Resolution: 650x515

24. Synthetically Focused Images

25. Synthetic Focus Mean color at estimated depth

26. Variance Mean color at estimated depth

27. Median Color Median color at estimated depth

28. Entropy Mean color at estimated depth

29. - = +…+ = Adding per-view occlusion maps yields occlusion map for points on the CD surface Comparison: Ground truth and Occlusion Map

30. Performance vs. Occlusion

31. Demo: Peek

32. Future Work • Compare with algorithms that try to delete occluding objects • Voxel Coloring [Seitz, CVPR 97] • Analyze effect of different kinds of occluders • color distribution, feature size • Aperture shape, size and sampling • Impact of number of cameras • Add global optimization

33. Acknowledgements • Rick, Larry, Sing Bing, Brian … • Fellow Interns (Jan, Noah, Ashish, Karteek, Ashley, Le, Manuel …) • Assistance in acquisition • Augusto Roman, David Koller, Neel Joshi