1 / 32

Lecture 8 Segmentation of Dynamical Scenes

Lecture 8 Segmentation of Dynamical Scenes. One-body two-views. One-body multiple-views. A static scene: multiple 2D motion models. A dynamic scene: multiple 3D motion models. Motivation and problem statement. Given an image sequence, determine

hayden
Download Presentation

Lecture 8 Segmentation of Dynamical Scenes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lecture 8Segmentation of Dynamical Scenes Invitation to 3D vision

  2. One-body two-views Invitation to 3D vision

  3. One-body multiple-views Invitation to 3D vision

  4. A static scene: multiple 2D motion models A dynamic scene: multiple 3D motion models Motivation and problem statement • Given an image sequence, determine • Number of motion models (affine, Euclidean, etc.) • Motion model: affine (2D) or Euclidean (3D) • Segmentation: model to which each pixel belongs Invitation to 3D vision

  5. Local methods (Wang-Adelson ’93) Estimate one model per pixel using data in a window Cluster models with K-means Iterate Aperture problem Motion across boundaries Global methods (Irani-Peleg ‘92) Dominant motion: fit one motion model to all pixels Look for misaligned pixels & fit a new model to them Iterate Previous work on 2D motion segmentation • Normalized cuts (Shi-Malik ‘98) • Similarity matrix based on motion profile • Segment pixels using eigenvector Invitation to 3D vision

  6. Factorization techniques Orthographic/discrete: Costeira-Kanade ’98, Gear ‘98 Perspective/continuous: Vidal-Soatto-Sastry ’02 Omnidirectional/continuous: Shakernia-Vidal-Sastry ’03 Special cases: Points in a line (orth-discrete): Han and Kanade ’00 Points in a conic (perspective): Avidan-Shashua ’01 Points in a line (persp.-continuous): Levin-Shashua ’01 2-body case: Wolf-Shashua ‘01 Previous work on 3D motion segmentation Invitation to 3D vision

  7. Probabilistic approaches Generative model: data membership + motion model Obtain motion models using Expectation Maximization E-step: Given motion models, segment image data M-step: Given data segmentation, estimate motion models 2D Motion Segmentation Layered representation (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99) 3D Motion Segmentation EM+Reprojection Error: Feng-Perona’98 EM+Model Selection: Torr ’98 How to initialize iterative algorithms? Previous work: probabilistic techniques Invitation to 3D vision

  8. This work considers full perspective projection multiple objects general motions We show that Problem is equivalent to polynomial factorization There is a unique global closed form solution if n<5 Exact solution is obtained using linear algebra Can be used to initialize EM-based algorithms Image points Number of motions Multibody Fund. Matrix Epipolar lines Multi epipolar lines Multi epipole Epipoles Fundamental Matrices Motion segmentation Our approach to motion segmentation Invitation to 3D vision

  9. Number of models? One-dimensional segmentation Invitation to 3D vision

  10. One-dimensional segmentation • For n groups • Number of groups • Groups Invitation to 3D vision

  11. One-dimensional segmentation • Solution is unique if • Solution is closed form if and only if • Solves the eigenvector segmentation problem e.g. normalized cuts Invitation to 3D vision

  12. Generalized PCA(Vidal, Ma ’02) Solve for the roots of a polynomial of degree in one variable Solve for a linear system in variables Three-dimensional motion segmentation Invitation to 3D vision

  13. Rotation: Translation: Epipolar constraint Multiple motions Multibody epipolar constraint The multibody epipolar constraint • Satisfied by ALL points regardless of segmentation • Segmentation is algebraically eliminated!!! Invitation to 3D vision

  14. Embedding Lifting Embedding The multibody fundamental matrix Bilinear on embedded data! • Veronese map (polynomial embedding) • Multibody fundamental matrix Invitation to 3D vision

  15. Theorem:Given image points corresponding to motions, if at least 8 points correspond to each object, then 1 2 3 4 Minimum number of points 35 99 225 8 Estimation of the number of motions Invitation to 3D vision

  16. 1-body motion Estimation of multibody fundamental matrix n-body motion Invitation to 3D vision

  17. Given Fundamental matrices Multibody epipolar transfer Multibody epipole Segmentation of fundamental matrices rank condition for n motions linear system F Invitation to 3D vision

  18. Lifting Multibody epipolar transfer Multibody epipolar line Polynomial factorization Invitation to 3D vision

  19. The multibody epipole is the solution of the linear system Epipoles are obtained using polynomial factorization Lifting Multibody epipole • Number of distinct epipoles Invitation to 3D vision

  20. From images to epipoles Invitation to 3D vision

  21. Columns of are epipolar lines Polynomial factorization to compute them up to scale Scales can be computed linearly Fundamental matrices Invitation to 3D vision

  22. Image point Veronese map Embedded image point Multibody epipolar transfer Multibody epipolar line Polynomial Factorization Epipolar lines Linear system Multibody epipole Polynomial Factorization Epipoles Linear system Fundamental matrix The multibody 8-point algorithm Invitation to 3D vision

  23. Zero-mean Gaussian noise Constrained optimization problem on Optimal function for 1 motion Optimal function for n motions Solved using Riemanian Gradien Descent Optimal 3D motion segmentation Invitation to 3D vision

  24. Comparison of 1 body and n bodies Invitation to 3D vision

  25. Multibody epipole Recovery of epipoles Fundamental matrices Feature segmentation Minimum number of points 1 2 3 4 35 99 225 8 1 2 5 10 5 20 65 2 Other cases: linearly moving objects Invitation to 3D vision

  26. Affine motion segmentation: constant brightness constraint 3D motion segmentation: epipolar constraint Other cases: affine flows • In linear motions, geometric constraints are linear • Two-view motion constraints could be bilinear!!! Invitation to 3D vision

  27. 3D motion segmentation results N = 44 + 48 + 81 = 173 Invitation to 3D vision

  28. Results Invitation to 3D vision

  29. Results Invitation to 3D vision

  30. More results Invitation to 3D vision

  31. There is an analytic solution to 3D motion segmentation based on Multibody epipolar constraint: it does not depend on the segmentation of the data Polynomial factorization: linear algebra Solution is closed form iff n<5 A similar technique also applies to Eigenvector segmentation: from similarity matrices Generalized PCA: mixtures of subspaces 2-D motion segmentation: of affine motions Future work Reduce data complexity, sensitivity analysis, robustness Conclusions Invitation to 3D vision

  32. R. Vidal, Y. Ma, S. Soatto and S. Sastry. Two-view multibody structure from motion, International Journal of Computer Vision, 2004 R. Vidal and S. Sastry. Optimal segmentation of dynamic scenes from two perspective views, International Conference on Computer Vision and Pattern Recognition, 2003 R. Vidal and S. Sastry. Segmentation of dynamic scenes from image intensities, IEEE Workshop on Vision and Motion Computing, 2002. References Invitation to 3D vision

More Related