Segmentation of Dynamic Scenes

1. Segmentation of Dynamic Scenes René Vidal Department of EECS, UC Berkeley

2. A static scene: multiple 2D motion models A dynamic scene: multiple 3D motion models Motivation and problem statement • Given an image sequence, determine • Number of motion models (affine, Euclidean, etc.) • Motion model: affine (2D) or Euclidean (3D) • Segmentation: model to which each pixel belongs

3. Previous work on 2D motion segmentation • Local methods (Wang-Adelson ’93) • Estimate one model per pixel using a data in a window • Cluster models with K-means • Iterate • Aperture problem • Motion across boundaries • Global methods (Irani-Peleg ‘92) • Dominant motion: fit one motion model to all pixels • Look for misaligned pixels & fit a new model to them • Iterate • Normalized cuts (Shi-Malik ‘98) • Similarity matrix based on motion profile • Segment pixels using eigenvector

4. Previous work on 3D motion segmentation • Factorization techniques • Orthographic/discrete: Costeira-Kanade ’98, Gear ‘98 • Perspective/continuous: Vidal-Soatto-Sastry ’02 • Omnidirectional/continuous: Shakernia-Vidal-Sastry ’03 • Special cases: • Points in a line (orth-discrete): Han and Kanade ’00 • Points in a conic (perspective): Avidan-Shashua ’01 • Points in a line (persp.-continuous): Levin-Shashua ’01 • 2-body case: Wolf-Shashua ‘01

5. Previous work: probabilistic techniques • Probabilistic approaches • Generative model: data membership + motion model • Obtain motion models using Expectation Maximization • E-step: Given motion models, segment image data • M-step: Given data segmentation, estimate motion models • 2D Motion Segmentation • Layered representation (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99) • 3D Motion Segmentation • EM+Reprojection Error: Feng-Perona’98 • EM+Model Selection: Torr ’98 • How to initialize iterative algorithms?

6. This work considers full perspective projection multiple objects general motion We show that Problem is equivalent to polynomial factorization There is a unique global closed form solution if n<5 Exact solution is obtained using linear algebra Can be used to initialize EM-based algorithms Our approach to motion segmentation Image points Number of motions Multibody Fund. Matrix Epipolar lines Multi epipolar lines Multi epipole Epipoles Fundamental Matrices Motion segmentation

7. Number of models? One-dimensional segmentation

8. One-dimensional segmentation • For n groups • Number of groups • Groups

9. Three-dimensional motion segmentation Generalized PCA(Vidal et.al. ‘02) • Solve for the roots of a polynomial of degree in one variable • Solve for a linear system in variables

10. Multibody epipolar constraint The multibody epipolar constraint • Rotation: • Translation: • Epipolar constraint • Multiple motions • Satisfied by ALL points regardless of segmentation • Segmentation is algebraically eliminated!!!

11. Embedding Lifting Embedding The multibody fundamental matrix Bilinear on embedded data! • Veronese map (polynomial embedding) • Multibody fundamental matrix

12. 1 2 3 4 Minimum number of points 35 99 225 8 Estimation of the number of motions • Theorem:Given image points corresponding to motions, if at least 8 points correspond to each object, then

13. 1-body motion Estimation of multibody fundamental matrix n-body motion

14. Given Fundamental matrices Multibody epipolar transfer Multibody epipole Segmentation of fundamental matrices rank condition for n motions linear system F

15. Lifting Multibody epipolar transfer Multibody epipolar line Polynomial factorization

16. The multibody epipole is the solution of the linear system Epipoles are obtained using polynomial factorization Lifting Multibody epipole • Number of distinct epipoles

17. Fundamental matrices • Columns of are epipolar lines • Polynomial factorization to compute them up to scale • Scales can be computed linearly

18. Image point Veronese map Embedded image point Multibody epipolar transfer Multibody epipolar line Polynomial Factorization Epipolar lines Linear system Multibody epipole Polynomial Factorization Epipoles Linear system Fundamental matrix The multibody 8-point algorithm

19. Optimal 3D motion segmentation • Zero-mean Gaussian noise • Constrained optimization problem on • Optimal function for 1 motion • Optimal function for n motions • Solved using Riemanian Gradient Descent

20. Comparison of 1 and n bodies

21. Multibody epipole Recovery of epipoles Fundamental matrices Feature segmentation Minimum number of points 1 2 3 4 35 99 225 8 1 2 5 10 5 20 65 2 Linearly moving objects

22. 3D motion segmentation results N = 44 + 48 + 81 = 173

23. Conclusions • There is an analytic solution to 3D motion segmentation based on • Multibody epipolar constraint: it does not depend on the segmentation of the data • Polynomial factorization: linear algebra • Solution is closed form iff n<5 • A similar technique also applies to • Eigenvector segmentation: from similarity matrices • Generalized PCA: mixtures of subspaces • 2-D motion segmentation: of affine motions • Future work • Reduce data complexity, sensitivity analysis, robustness

24. References • R. Vidal, Y. Ma, S. Soatto and S. Sastry. Two-view multibody structure from motion, International Journal of Computer Vision, 2004 • R. Vidal and S. Sastry. Optimal segmentation of dynamic scenes from two perspective views, International Conference on Computer Vision and Pattern Recognition, 2003 • R. Vidal and S. Sastry. Segmentation of dynamic scenes from image intensities, IEEE Workshop on Vision and Motion Computing, 2002.