250 likes | 332 Views
Space Time Tracking ECCV 2002. Lorenzo Torresani Christoph Bregler. Outline . Problem Background Structure from Motion Matrix Decomposition Non-Rigid Motion Estimation Non-Rigid Shapes Estimation Results. Problem.
E N D
Space Time Tracking ECCV 2002 Lorenzo Torresani Christoph Bregler
Outline • Problem • Background • Structure from Motion • Matrix Decomposition • Non-Rigid Motion Estimation • Non-Rigid Shapes Estimation • Results
Problem “To track feature points on non-rigid objects without using any prior model”
M N M N Rank of a Matrix Rank (A) = Number of linearly independent vectors in Rank (B) = Number of linearly independent vectors in For M x N the Rank of A ≤ min (M,N)
Rank of a Matrix cont’d Rank C = ? Columns of C are Linear combination of Columns of A Rank C = N There are only N independent vectors in A
VT n x n D n x n A m x n U m x n SVD • SVD for a matrix A writes A as a product of three matrices: • U • D • V • Every m x n matrix has a singular value decomposition U,V have orthogonal columns
Frame 1 Frame 2 ………… Frame F Tomasi Kanade Structure from Motion • Given N 2D trajectories taken over F images, recover 3D structure and motion (Camera pose) • Assumption: • 3D Object is rigid • Orthographic Projection • Tracks can be computed using any standard tracker (KLT etc)
Tomasi Kanade Structure from Motion cont’d • Assume a set of P 3D points on a rigid object (structure) S = [P1, P2 …….. PP ] • Orthographic Projection where (u,v) are image coordinates and M is orthographic projection matrix • Subtract mean of all u’s and v’s to center the world coordinate frame at the center of the object. • This will get rid of T in the above equation
Tomasi Kanade Structure from Motion cont’d • 2D coordinates of N points over F images can be written in one matrix • W is called the measurement/tracking matrix Rank 3
From W to R and S Force the rank of W to be 3 SVD
Steps • Matrix Decomposition of W matrix for non-rigid objects • Estimate Motion Matrix using reliable set of points • Estimate shape basis (S) for all other feature points (unreliable)
S l1S1 l2S2 lKSK + + … + = S S S 1 2 K ……………… 3D Non Rigid Shape Model • Linear Combination of K Basis Shapes • Each basis shape is Si3 x P matrix describing P points Courtesy Christopher Bregler
W Q M Matrix Decomposition • Project P points of shape S • Scaled Orthographic Projection • Move world coordinate to object centeroid (This will get rid of T)
2F x 3K 3K x P Tracking Matrix Complete 2D Tracks or Flow Rank of W 3K In Tomasi Kanade it was 3
? ? = known reliable tracks W Q’ M’ Non Rigid Motion Estimation • Since W isrank-deficient, Q can be estimated w/o the full availability of W • r <= 3K point tracks will span the space of the trajectories of all the points (as rank of W is r) Courtesy Christopher Bregler
t=F . t=2 = 3D positions of point i for the K modes of deformation . t=1 . . . . . . frames wi : full trajectory mi Trajectory Constraint Q’ • Generate m trajectories (hypothesis) using Factored Sampling • Evaluate w by computing sum of square difference around point i. Courtesy Christopher Bregler
Each column mi of unreliable M is computed as expected value of posterior.