Structure From Motion

1. Structure From Motion Sebastian Thrun, Gary Bradski, Daniel Russakoff Stanford CS223B Computer Vision http://robots.stanford.edu/cs223b

2. Structure From Motion (1) [Tomasi & Kanade 92]

3. Structure From Motion (2) [Tomasi & Kanade 92]

4. Structure From Motion (3) [Tomasi & Kanade 92]

5. Structure From Motion • Problem 1: • Given n points pij =(xij, yij) in m images • Reconstruct structure: 3-D locations Pj =(xj, yj, zj) • Reconstruct camera positions (extrinsics) Mi=(Aj, bj) • Problem 2: • Establish correspondence: c(pij)

6. Orthographic Camera Model Extrinsic Parameters Rotation Orthographic Projection Limit of Pinhole Model:

7. Orthographic Projection Limit of Pinhole Model: Orthographic Projection

8. The Affine SFM Problem

9. Count # Constraints vs #Unknowns • m camera poses • n points • 2mn point constraints • 8m+3n unknowns • Suggests: need 2mn  8m + 3n • But: Can we really recover all parameters???

10. How Many Parameters Can’t We Recover? We can recover all but… Place Your Bet!

11. The Answer is (at least): 12

12. Points for Solving Affine SFM Problem • m camera poses • n points • Need to have: 2mn  8m + 3n-12

13. Affine SFM Fix coordinate system by making p0=origin Rank Theorem: Q has rank 3 Proof:

14. The Rank Theorem 2m elements n elements

15. Tomasi/Kanade 1992 Singular Value Decomposition

16. Tomasi/Kanade 1992 Gives also the optimal affine reconstruction under noise

17. Back To Orthographic Projection Find C and d for which constraints are met

18. Back To Projective Geometry Orthographic (in the limit) Projective

19. Projective Camera: Non-Linear Optimization Problem: Bundle Adjustment!

20. Structure From Motion • Problem 1: • Given n points pij =(xij, yij) in m images • Reconstruct structure: 3-D locations Pj =(xj, yj, zj) • Reconstruct camera positions (extrinsics) Mi=(Aj, bj) • Problem 2: • Establish correspondence: c(pij)

21. The Correspondence Problem View 1 View 2 View 3

22. Correspondence: Solution 1 • Track features (e.g., optical flow) • …but fails when images taken from widely different poses

23. Correspondence: Solution 2 • Start with random solution A, b, P • Compute soft correspondence: p(c|A,b,P) • Plug soft correspondence into SFM • Reiterate • See Dellaert/Seitz/Thorpe/Thrun 2003

24. Example

25. Results: Cube

26. Animation

27. Tomasi’s Benchmark Problem

28. Reconstruction with EM

29. 3-D Structure