From photohulls to photoflux optimization

1. From photohulls to photoflux optimization Yuri Boykov University of Western Ontario Victor Lempitsky Moscow State University

2. Overview • Current methods for multi-view reconstruction • Photohulls and space carving • “Deformable models” • Flux for image segmentation • Photoflux for stereo • Proof of concept

3. binary, monotonic photoconsistency local decision to “carve” inconsistent points is OK region growing it is guaranteed that Kutulakos and Seitz, 2000Photohull S2 X S1 3 1 2

4. A slide from Kutulakos and Seitz (IJCV 2000) Which shape do you get? V V • The Photo Hullis the UNION of all photo-consistent scenes in V • It is a photo-consistent scene reconstruction • Tightest possible bound on the true scene

5. V Which shape do you get? V • The Photo Hull is the UNION of all photo-consistent scenes in V • It is a photo-consistent scene reconstruction • Tightest possible bound on the true scene?

6. Photohull may not be well defined in some cases Rectangle of any size is a photoconsistent surface for this camera configuration object of arbitrary shape and surface texture

7. a larger photoconsistent surface photohull [K&S’00] Space carving may underestimate the tightest bound on the true scene Why?

8. inconsistent voxel consistent surface current voxels Photohull surface pointsSpace Carving voxels current surface

9. non-monotonic photoconsistency NOT safe to “carve” inconsistent points Non-monotone photoconsistency S2 X S1 3 1 2 it is possible that

10. non-binary photoconsistency “local” answer no longer available To carve or not to carve? Non-binary photoconsistency S2 X S1 3 1 2

11. So,… what exactly is photohull ? • Largest continuous photoconsistent surface? • Defined only for binary monotone photoconsistency • In some cases it may not exist OR • The output of voxel carving algorithm? • Which version? • approach to voxel vs. surface consistency • threshold for making 0-1 photoconsistency decisions • order of voxel carving for non-monotone photoconsistency

12. Faugeras and Keriven, 98 Pons et al., 05 Vogiatzis et al., 05 “continuous space carving” ? Minimal surfaces for multi-view reconstruction non-binary photoconsistency X S1 3 1 2

13. S2 X Is the surface guaranteed to contract? NO surface can both contract or expand even for monotonic photoconsistency X S1 3 1 2

14. Bottom line • Deformable models • different surface based concept • surface based algorithms • solution exists for solid PDE’s • real-valued photoconsistency • - energy-based regularization • no spatial monotonisity • not greedy, may backtrack/expand • no noise but may oversmooth • Photohull • surface based concept • voxel based algorithms • may not exist • binary photoconsistency • thresholding, may leak • spatial monotonicity • greedy carving • can get fine details but noisy

15. Multi-view Reconstruction versusImage Segmentation Flux-based methods Regularization Greedy methods Snakes Kass et al., 1988 Level-sets Malladi et al., 1994 Graph cuts Boykov and Jolly, 2001 Level-sets Vasilevsky and Sidiqqi, 2002 Kimmel et al., 2003 Graph cuts Kolmogorov and Boykov, 2005 Thresholding Region growing image segmentation Mash-based Esteban and Schmitt, 2004 Level-sets Faugeras and Keriven, 1998 Graph cuts Vogiatzis et al., 2005 Voxel coloring Seitz and Dyer, 1997 Space carving Kutulakos and Seitz, 2002 multi-view reconstruction (volumetric approach) This work

16. C1 n n C2 flux(C1) > flux(C2) Flux • vector field: some vector defined at each point p • “stream of water” with a given speed at each location • flux: “amount of water” passing through a given contour • changes sign with orientation

17. Gauss-Ostrogradsky (divergence) theorem Why flux? • Regularization: shrinks • Flux: intelligent ballooning flux and length are geometric properties of boundary with opposite effect

18. boundary with large flux What flux is good for?Segmentation of thin objects [Vasilevsky,Siddiqi’02] • Flux of image gradients:

19. What flux is good for?Segmentation of thin objects [Vasilevsky,Siddiqi’02] _ + _ + • Flux of image gradients: • Laplacian of image intensities _ + + _ + _ + _ + _ + …or region optimizing Laplacian boundary with large flux

20. flux Laplacian zero crossings (if vectors are gradients of some potential field) What flux is good for?Data-driven ballooning Vasilevsky and Sidiqqi, 2002 Kimmel and Bruckstein, 2003 (in the context of level-sets) image intensities on a scan line Laplacian of intensities + + - - Kolmogorov and Boykov 2005 (in the context of graph cuts)

21. Why flux?Riemannian length + Flux [Kimmel,Bruckstein’03] Riemannian length Flux of Riemannian length + Flux

22. Integrating Laplacian Zero-crossingsinto Graph Cuts [Kolmogorov&Boykov’05] graph cuts Flux = smart ballooning counteract shrinking bias The image is courtesy of David Fleet University of Toronto

23. “Shrinking” bias in stereo CVPR’05 slides from Vogiatzis, Torr, Cippola

24. Uniform ballooning CVPR’05 slides from Vogiatzis, Torr, Cippola

25. Our approach:introduce flux for multi-view stereo • Capture some properties of photohull through a novel surface functional photoflux • Photoflux can be combined with regularization • combines benefits of space carving and deformable models • can recover fine details while keeping the noise low • Photofluxaddresses shrinking or “over-smoothing” bias of standard regularization methods for N-view reconstruction • data-driven intelligent ballooning addresses shrinking bias • regularized Laplacian zerocrossings for boundary alignment

26. photohull X X’ for all points X on photohull for all points X’ right outside photohull From photohull to photoflux binary photoconsistency

27. X’ for all points X on photohull Gradient of photoconsistency is large for all points X on photohull for all points X’ right outside photohull From photohull to photoflux non-binary photoconsistency photohull X

28. f(S) = outward surface normal at point X fixed shape S allows to compute global visibility of point X Photoflux X 3 1 2

29. E(S) = Photoflux photoflux + regularization • Optimization • Local via level-sets or PDE-cuts [BKCD, ECCV’06] • maintaining global visibility could be expensive in practice • local patch dS can approximate visibility • global graph cuts on a complex [Lempitsky et al. ECCV’06] • other options also…

30. f(S) = Photoflux visibility vector

31. photoconsistency Visibility vector 4 X 3 5 2 1

32. f(S) = unknown visibility given point X is on some surface patch based visibility w=w(N) Generalization of photoflux idea if visibility w at point X is given, e.g. w=w(S)

33. Photoflux for estimated local photoconsistency gradients Vector field can be computed at every X without fixing any surfce S

34. Back to example from K&S’00

35. Back to example from K&S’00 gradients and divergence of “photoconsistency” zoom

36. Back to example from K&S’00 photohull [K&S, IJCV’00] gradients and divergence of “photoconsistency”

37. Back to example from K&S’00 larger photoconsistent surface gradients and divergence of “photoconsistency”

38. Back to example from K&S’00 16 cameras 4 cameras

39. Textured and non-textured objects 16 cameras 4 cameras

40. Regularizing Photoflux photoflux + regularization • Photoflux does not have regularization by itself • Regularized Liplacian zero-crossings E(S) =

41. Algorithms • Level-sets • Continuous max-flow techniques • Discrete max-flow (graph cuts) • implicit • explicit

42. 0 0 0 1 0 1 0 0 1 0 1 1 “Implicit” graph cuts • Most current graph cuts technique implicitly use surfaces represented via binary (interior/exterior) labeling of pixels

43. “Explicit” graph cuts • Except, a recent explicit surfaces representation method • - Kirsanov and Gortler, 2004

44. “Explicit” graph cuts • for multi-view reconstruction • Lempitsky et al., ECCV 2006 • Explicit surface patches allow local estimation of visibility when computing globally optimal solution local oriented visibility estimate Compare with Vogiatzis et al., CVPR’05 approach for visibility

45. “Explicit” graph cuts Regularization + uniform ballooning some details are still over-smoothed Lempitsky et al., ECCV 2006

46. This work Regularization + “intelligent ballooning” Low noise and no shrinking

47. Space carving

48. Also tried surface based flux E(S) = sometimes not submodular

49. Photoflux II “Camel”

50. Photoflux II “Hand” data courtesy of K. Kutulakos and S Seitz