320 likes | 333 Views
Lecture 8 Segmentation of Dynamical Scenes. One-body two-views. One-body multiple-views. A static scene: multiple 2D motion models. A dynamic scene: multiple 3D motion models. Motivation and problem statement. Given an image sequence, determine
E N D
Lecture 8Segmentation of Dynamical Scenes Invitation to 3D vision
One-body two-views Invitation to 3D vision
One-body multiple-views Invitation to 3D vision
A static scene: multiple 2D motion models A dynamic scene: multiple 3D motion models Motivation and problem statement • Given an image sequence, determine • Number of motion models (affine, Euclidean, etc.) • Motion model: affine (2D) or Euclidean (3D) • Segmentation: model to which each pixel belongs Invitation to 3D vision
Local methods (Wang-Adelson ’93) Estimate one model per pixel using data in a window Cluster models with K-means Iterate Aperture problem Motion across boundaries Global methods (Irani-Peleg ‘92) Dominant motion: fit one motion model to all pixels Look for misaligned pixels & fit a new model to them Iterate Previous work on 2D motion segmentation • Normalized cuts (Shi-Malik ‘98) • Similarity matrix based on motion profile • Segment pixels using eigenvector Invitation to 3D vision
Factorization techniques Orthographic/discrete: Costeira-Kanade ’98, Gear ‘98 Perspective/continuous: Vidal-Soatto-Sastry ’02 Omnidirectional/continuous: Shakernia-Vidal-Sastry ’03 Special cases: Points in a line (orth-discrete): Han and Kanade ’00 Points in a conic (perspective): Avidan-Shashua ’01 Points in a line (persp.-continuous): Levin-Shashua ’01 2-body case: Wolf-Shashua ‘01 Previous work on 3D motion segmentation Invitation to 3D vision
Probabilistic approaches Generative model: data membership + motion model Obtain motion models using Expectation Maximization E-step: Given motion models, segment image data M-step: Given data segmentation, estimate motion models 2D Motion Segmentation Layered representation (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99) 3D Motion Segmentation EM+Reprojection Error: Feng-Perona’98 EM+Model Selection: Torr ’98 How to initialize iterative algorithms? Previous work: probabilistic techniques Invitation to 3D vision
This work considers full perspective projection multiple objects general motions We show that Problem is equivalent to polynomial factorization There is a unique global closed form solution if n<5 Exact solution is obtained using linear algebra Can be used to initialize EM-based algorithms Image points Number of motions Multibody Fund. Matrix Epipolar lines Multi epipolar lines Multi epipole Epipoles Fundamental Matrices Motion segmentation Our approach to motion segmentation Invitation to 3D vision
Number of models? One-dimensional segmentation Invitation to 3D vision
One-dimensional segmentation • For n groups • Number of groups • Groups Invitation to 3D vision
One-dimensional segmentation • Solution is unique if • Solution is closed form if and only if • Solves the eigenvector segmentation problem e.g. normalized cuts Invitation to 3D vision
Generalized PCA(Vidal, Ma ’02) Solve for the roots of a polynomial of degree in one variable Solve for a linear system in variables Three-dimensional motion segmentation Invitation to 3D vision
Rotation: Translation: Epipolar constraint Multiple motions Multibody epipolar constraint The multibody epipolar constraint • Satisfied by ALL points regardless of segmentation • Segmentation is algebraically eliminated!!! Invitation to 3D vision
Embedding Lifting Embedding The multibody fundamental matrix Bilinear on embedded data! • Veronese map (polynomial embedding) • Multibody fundamental matrix Invitation to 3D vision
Theorem:Given image points corresponding to motions, if at least 8 points correspond to each object, then 1 2 3 4 Minimum number of points 35 99 225 8 Estimation of the number of motions Invitation to 3D vision
1-body motion Estimation of multibody fundamental matrix n-body motion Invitation to 3D vision
Given Fundamental matrices Multibody epipolar transfer Multibody epipole Segmentation of fundamental matrices rank condition for n motions linear system F Invitation to 3D vision
Lifting Multibody epipolar transfer Multibody epipolar line Polynomial factorization Invitation to 3D vision
The multibody epipole is the solution of the linear system Epipoles are obtained using polynomial factorization Lifting Multibody epipole • Number of distinct epipoles Invitation to 3D vision
From images to epipoles Invitation to 3D vision
Columns of are epipolar lines Polynomial factorization to compute them up to scale Scales can be computed linearly Fundamental matrices Invitation to 3D vision
Image point Veronese map Embedded image point Multibody epipolar transfer Multibody epipolar line Polynomial Factorization Epipolar lines Linear system Multibody epipole Polynomial Factorization Epipoles Linear system Fundamental matrix The multibody 8-point algorithm Invitation to 3D vision
Zero-mean Gaussian noise Constrained optimization problem on Optimal function for 1 motion Optimal function for n motions Solved using Riemanian Gradien Descent Optimal 3D motion segmentation Invitation to 3D vision
Comparison of 1 body and n bodies Invitation to 3D vision
Multibody epipole Recovery of epipoles Fundamental matrices Feature segmentation Minimum number of points 1 2 3 4 35 99 225 8 1 2 5 10 5 20 65 2 Other cases: linearly moving objects Invitation to 3D vision
Affine motion segmentation: constant brightness constraint 3D motion segmentation: epipolar constraint Other cases: affine flows • In linear motions, geometric constraints are linear • Two-view motion constraints could be bilinear!!! Invitation to 3D vision
3D motion segmentation results N = 44 + 48 + 81 = 173 Invitation to 3D vision
Results Invitation to 3D vision
Results Invitation to 3D vision
More results Invitation to 3D vision
There is an analytic solution to 3D motion segmentation based on Multibody epipolar constraint: it does not depend on the segmentation of the data Polynomial factorization: linear algebra Solution is closed form iff n<5 A similar technique also applies to Eigenvector segmentation: from similarity matrices Generalized PCA: mixtures of subspaces 2-D motion segmentation: of affine motions Future work Reduce data complexity, sensitivity analysis, robustness Conclusions Invitation to 3D vision
R. Vidal, Y. Ma, S. Soatto and S. Sastry. Two-view multibody structure from motion, International Journal of Computer Vision, 2004 R. Vidal and S. Sastry. Optimal segmentation of dynamic scenes from two perspective views, International Conference on Computer Vision and Pattern Recognition, 2003 R. Vidal and S. Sastry. Segmentation of dynamic scenes from image intensities, IEEE Workshop on Vision and Motion Computing, 2002. References Invitation to 3D vision