Space Time Tracking ECCV 2002

1. Space Time Tracking ECCV 2002 Lorenzo Torresani Christoph Bregler

2. Outline • Problem • Background • Structure from Motion • Matrix Decomposition • Non-Rigid Motion Estimation • Non-Rigid Shapes Estimation • Results

3. Problem “To track feature points on non-rigid objects without using any prior model”

4. M N M N Rank of a Matrix Rank (A) = Number of linearly independent vectors in Rank (B) = Number of linearly independent vectors in For M x N the Rank of A ≤ min (M,N)

5. Rank of a Matrix cont’d Rank C = ? Columns of C are Linear combination of Columns of A Rank C = N There are only N independent vectors in A

6. VT n x n D n x n A m x n U m x n SVD • SVD for a matrix A writes A as a product of three matrices: • U • D • V • Every m x n matrix has a singular value decomposition U,V have orthogonal columns

7. Frame 1 Frame 2 ………… Frame F Tomasi Kanade Structure from Motion • Given N 2D trajectories taken over F images, recover 3D structure and motion (Camera pose) • Assumption: • 3D Object is rigid • Orthographic Projection • Tracks can be computed using any standard tracker (KLT etc)

8. Tomasi Kanade Structure from Motion cont’d • Assume a set of P 3D points on a rigid object (structure) S = [P1, P2 …….. PP ] • Orthographic Projection where (u,v) are image coordinates and M is orthographic projection matrix • Subtract mean of all u’s and v’s to center the world coordinate frame at the center of the object. • This will get rid of T in the above equation

9. Tomasi Kanade Structure from Motion cont’d • 2D coordinates of N points over F images can be written in one matrix • W is called the measurement/tracking matrix Rank 3

10. From W to R and S Force the rank of W to be 3 SVD

12. Steps • Matrix Decomposition of W matrix for non-rigid objects • Estimate Motion Matrix using reliable set of points • Estimate shape basis (S) for all other feature points (unreliable)

13. For Non Rigid Constraint

14. S l1S1 l2S2 lKSK + + … + = S S S 1 2 K ……………… 3D Non Rigid Shape Model • Linear Combination of K Basis Shapes • Each basis shape is Si3 x P matrix describing P points Courtesy Christopher Bregler

15. W Q M Matrix Decomposition • Project P points of shape S • Scaled Orthographic Projection • Move world coordinate to object centeroid (This will get rid of T)

17. 2F x 3K 3K x P Tracking Matrix Complete 2D Tracks or Flow Rank of W 3K In Tomasi Kanade it was 3

18. ? ? = known reliable tracks W Q’ M’ Non Rigid Motion Estimation • Since W isrank-deficient, Q can be estimated w/o the full availability of W • r <= 3K point tracks will span the space of the trajectories of all the points (as rank of W is r) Courtesy Christopher Bregler

19. r = 9

20. t=F . t=2 = 3D positions of point i for the K modes of deformation . t=1 . . . . . . frames wi : full trajectory mi Trajectory Constraint Q’ • Generate m trajectories (hypothesis) using Factored Sampling • Evaluate w by computing sum of square difference around point i. Courtesy Christopher Bregler

21. Each column mi of unreliable M is computed as expected value of posterior.

23. Results