How to fit a surface to a point cloud? or optimization of surface functionals in computer vision

1. TRICS seminar Computer Science Department How to fit a surface to a point cloud? or optimization of surface functionals in computer vision Yuri Boykov

2. Optimization of surface functionals in computer vision • Computer vs. human vision • Model fitting in computer vision • templates, pictorial structures, trees, deformable models, contours/snakes, meshes, surfaces, complexes, graphs, weak-membrane model, Mumford-Shah, Potts model,…… • Optimization in computer vision • dynamic programming, gradient descent, PDEs, shortest paths, min. spanning trees, linear and quadratic programming, primal-dual schema, network flow algorithms, QPBO, ... • Applications • segmentation, stereo, multi-view reconstruction, optical flows • surface fitting

3. Contours

4. +Shading 3D shape understanding M.C. Escher Drawing hands

5. +Color Recognition Da Vinci Madonna Litta

6. +Texture recognizing material Magritte Souvenir de Voyage

7. +Texture recognizing 3D perspective The New Yorker Album of Drawings, The Viking Press, NY, 1975

8. What do humans get by ‘looking’? basic image cues: • Contours • Shading • Color • Texture • … J. Vermeer The Guitar Player

9. What do humans get by ‘looking’? basic image cues: higher-level perception: • Segmentation • Motion • 3D shape perception • 3D scene geometry • Detection/Recognition • … Contours Shading Color Texture …

10. What do computers get by ‘looking’? x I(x,y) x y y y 3D plot of image intensity I(x,y) x

11. What do computers get by ‘looking’? higher-level grouping? basic image cues: • Intensity discontinuities(contours) • Intensity gradients (shading) • Multi-valued intensities (color) • Filtering (e.g. texture) • … P. Picasso The Guitar Player

12. Bayesian approach Prior + Data Fit some prior model into data model Low-level cues (local info) high-level knowledge (global picture)

13. Face template image Rigid Template Matching image translation, rotation, scaling • In matching we estimate “position” of a rigid template in the image • “Position” includes global location parameters of a rigid template: • - translation, rotation, scale,…

14. Non-rigid (parametric) matching panorama mosaicing • Pick one image (red) • Warpthe other images to match it (homographic transform) • Blend

15. e.g.… using homographies

16. e.g…. using flexible templates • In flexible template matching we estimate “position” of each rigid component of a template • For tree-structured models, efficient global optimization is possible via DP (Felzenswalb&Huttenlocher 2002)

17. tracking parameters => activity recognition Bottom-up tracker

18. initial final intermediate deformable contours (“snakes”) Kass, Witkin, Terzopoulos 1987 • 2D curve which matches to image data • Initialized near target, iteratively refined • Can restore missing data Optimization gets harder when a loop is introduced. DP does not apply. One solution: gradient descent

19. local minima, fixed contour topology Cremers, Tischhäuser, Weickert, Schnörr, “Diffusion Snakes”, IJCV '02

20. Level set function u(x,y) is normally discretized/stored over image pixels • Values of u(p) can be interpreted asdistancesorheights of image pixels -0.8 -0.8 -1.7 0.2 A contour may be approximated from u(x,y) with near sub-pixel accuracy -0.5 0.5 -0.6 -0.4 0.7 -0.2 0.3 0.6 Implicit representation of contours Osher&Sethian 1989 C

21. [Visualization is courtesy of O. Juan]Simple evolution Morphological Operation: Erosion

22. Visualization is courtesy of O. JuanExample of gradient descent evolution Gradient descent w.r.t. Euclideanlength

23. Example of gradient descent evolution Gradient descent w.r.t. Euclideanlength Laplacian Osher&Sethian 1989

24. [example from Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE TIP ’01]Geodesic Active Contours via Level-sets

25. W Other geometric energy functionals besides length[courtesy of Ron Kimmel] Geometric measures commonly used in segmentation gradient descent evolution Functional E( C ) weighted length weighted area alignment (flux)

26. in 3D…deformable meshes, level-sets, … Estimation of position for mesh points Many loops. optimization - gradient descent Typical problems: - local minima (clutter, outliers) -over-smoothing GOALS: global optima (?) “right” functional (?)

27. Global Optimization and Surface Functionals

28. More generally...Estimate labels for graph nodes observed noisy image I image labeling L (restored intensities) I L NOTE: similar to robust regression model estimation p along one scan line in the image

29. (simple example) Piece-wise smooth restoration • Markov Random Fields (MRF) approach weak membrane model(Geman&Geman’84, Blake&Zisserman’83,87) (Continuous analogue: Mamford-Shah functional, 1989) discontinuity preserving prior optimizing E(L) is NP hard!

30. (simple example) Piece-wise constant restoration observed noisy image I image labeling L (restored intensities) I L p along one scan line in the image

31. (simple example) Piece-wise constant restoration Potts model Boykov Veksler Zabih ’01 Greiget al.’89 (for 2 labels) “perceptual grouping” global optimization is still NP hard, but there are fast provably good combinatorial approximation algorithms, linear and quadratic programming, QPBO, primal-dual schema

32. Perceptual grouping from stereo constant label = plane (Birchfield &Tomasi’99)

33. Binary labeling(binary image restoration) Greig Porteous Seheult ’89 optimal binary labelingL original binary image I • Globally optimal solution is possible using combinatorial graph cut algorithms • pseudo-boolean optimization Hammer’65, Picard&Ratlif’75

34. Binary labeling(object extraction) object segmentation left ventricle of heart

35. Boykov&Jolly’01 Binary labeling(object extraction) surface extraction C left ventricle of heart • Globally optimal solution is possible using graph cut algorithms • pseudo-boolean optimization(Hammer’65, Picard&Ratlif’75)

36. 0 0 0 1 0 1 0 0 1 0 1 1 Implicitsurface representation via graph-cuts • Any contour (or surface in 3D) satisfying labeling of exterior/interior points (pixel centers) is acceptable if some explicit surface has to be output.

37. Tight characterization for geometric functionals of contour C that can be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05) disclaimer: for pairwise interactions only Global optimization of geometric surface functionals “edge alignment” • Regional bias • any scalar function f • Geometric length • any convex, • symmetric metric • (e.g. Riemannian) • Flux • any vector field v

38. Globally optimal surfaces in 3D Volumetric segmentation (BJ01,BK’03,KB’05)

39. Binary labeling(object extraction) Blake et al.’04, Rother et al.’04 iteratively re-estimate color models e.g. using mixture of Gaussians

40. Segmentation for Image Blending

41. Segmentation for Image Blending

42. Optimal surfaces in 3D Local cues: voxel’sphotoconsistency Prior: smoothness, projective geometry constraints 3D reconstruction Vogiatzis, Torr, Cippola’05

43. Globally optimal surfaces in 3D from a cheapdigital camera Lempitsky&Boykov, 2006

44. Globally optimal surfaces in 3D 3D model (texture mapped) multi-view reconstruction set up Furukawa&Ponce2006

45. 3D model: Surface fitting to point cloud a cloud of 3D points (e.g. from a laser scanner) surface fitting:

46. Surface fitting to point cloud

47. Surface fitting functional Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007) data fit prior

48. From 10 views No initialization is needed Optimal surfaces in 3D Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)

49. Global vs. local optimization Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007) initial solution regional potentials global minima local minima

50. Fitting to sparse data