Structure from Motion

1. Structure from Motion ECE 847:Digital Image Processing Stan Birchfield Clemson University

2. Acknowledgment Many slides are courtesy of others

3. SVD • Any mxn matrix A can be decomposed as • where • This is the singular value decomposition (SVD) mxm mxn nxn

4. = Tall and short matrices Tall matrix m>n, p = n mxm mxn nxn m<n, p = m Short matrix = mxm mxn nxn

5. Compact version Tall matrix Tall matrix m>n, p = n = mxm mxn nxn m<n, p = m Short matrix Short matrix = mxm mxn nxn

6. Compact version (cont.) Tall matrix Tall matrix m>n, p = n = mxn nxn nxn m<n, p = m Short matrix Short matrix = mxm mxm mxn

7. SVD reveals structure • Let r be the index of the smallest non-zero singular value • Then • Easy to show:

8. Eigen / singular • Singular values and singular vectorswork likeeigenvalues and eigenvectors: • First p eigenvalues of ATA (or AAT) are squares of the singular values of A:

9. Condition number • A is non-singular if and only if • In real life, matrices are never singular. • The condition number of A is • If 1/C is near the machine’s precision, then A is ill-conditioned. It is dangerous to invert A.

10. Norms Singular values readily yield norms: • Induced Euclidean norm: • Frobenius norm:(Euclidean norm, treating matrix as vector)

11. Least squares where The set of equations is solved as or

12. Least squares (cont.) • Minimum norm least squares solution to Ax=b, i.e., the shortest vector x that achieves is unique and is given by • where pseudoinverse inverts all nonzero singular values

13. Homogeneous system • What if b is all zeros? • Then the minimum-norm solution is not interesting, b/c it will be x=0 always • Instead, find unit-norm solution • Solution is given by (the right singular vector associated with the smallest singular value)

14. Enforcing constraints Find closest matrix to A in the sense of Frobenius norm that satisfies constraints exactly: • Factorize A = USVT • Change S to S’ to satisfy constraints • Put back together: A’ = US’VT Example: Enforce rank of A by setting small singular values to zero

15. Geometric interpretation of SVD

16. Structure from motion • Structure from motion (SFM) recovers • scene geometry • camera motion from a sequence of images • Could be called structure (or shape) and motion from video (SAMV), but nobody does this

17. SFM preliminaries • Collect F frames of P points (with correspondence) • Camera coordinate system: centered at focal point and aligned with image axes (x and y in image, positive z along optical axis) • World coordinate system is coincident with first camera (arbitrary)

18. SFM under perspective projection pth point xp-tf xp • Perspective imaging: • Equation counting: • 2FP+1 equations (extra equation from scale ambiguity) • 3P + 6(F-1) unknowns • Required:2FP+1 >= 3P + 6(F-1)With 2 frames, need at least 5 points if fth camera coord sys. tf world coord sys. jf

19. Perspective: 2 frames of 5 points • Show graphically that with fewer than 5 points, there is always wiggle room between camera frames

20. 8-point algorithm • Longuet-Higgins • Hartley normalization

21. SFM under orthographic projection • Orthographic imaging ignores depth: • Equation counting: • 2FP+F equations (extra eqn. for each frame: set z motion to 0) • 3P + 6(F-1) unknowns (same as perspective) • But equations are not independent (complicated proof omitted) • 2 frames is not enough • With 3 frames, need at least 4 points

22. Orthography: 3 frames of 4 points • Show graphically the wiggle room with < 3 frames or < 4 points

23. Factorization rotation • Recall: • Stack into measurement matrix: 4xP 2FxP 2Fx4 (Tomasi and Kanade 1992) measurement = motion x shape

24. Subtracting centroid • Place world origin at centroid of points: • Then subtract centroid of image coordinates per frame:

25. Registered measurements • This leads to the registered measurement matrix: 3xP 2FxP 2Fx3 registered measurement = rotation x shape

26. 3 3 0 3 0 0 Rank theorem • Similarly, • Use SVD to enforce rank constraint: • This reduces effects of noise in a robust, stable way

27. Euclidean constraints • But our choice was arbitrary • Solution is unique only up to affine transformation • Impose metric constraints to solve for Q: for any invertible 3x3 matrix Q use least squares, then Cholesky decomposition

28. Algorithm summary Tomasi-Kanade factorization for SFM: (Quadratic equations require nonlinear minimization)

29. Results

30. More results

31. Handling occlusion Unknown image measurement pair (ufp,vfp) in frame f can be reconstructed if • p is visible in 3 image frames • 3 other points are visible in 4 frames

32. Occlusion results ping pong ball rotated 450 degrees 84% of data hallucinated from 16%

33. Factorization extensions • Poelman and Kanade (1994): Paraperspective • Costeira and Kanade (1995): Multibody factorization • Sturm and Triggs (1996): Perspective, fixed rank algorithm to speed computation multibody (Costeira and Kanade) results

34. Planar parallax • See Irani

35. Using dynamics • We have looked at batch methods. Now incremental methods. • A. Davison real-time reconstruction

36. Texture mapping Depth image Texture image Triangle mesh • Pollefeys Textured 3D Wireframe model