Stanford CS223B Computer Vision, Winter 2007 Lecture 8 Structure From Motion

1. Stanford CS223B Computer Vision, Winter 2007Lecture 8 Structure From Motion Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens Slide credit: Gary Bradski, Stanford SAIL

2. Summary SFM • Problem • Determine feature locations (=structure) • Determine camera extrinsic (=motion) • Two Principal Solutions • Bundle adjustment (nonlinear least squares, local minima) • SVD (through orthographic approximation, affine geometry) • Correspondence • (RANSAC) • Expectation Maximization

3. Structure From Motion features camera Recover: structure (feature locations), motion (camera extrinsics)

4. SFM = Holy Grail of 3D Reconstruction • Take movie of object • Reconstruct 3D model • Would be commercially highly viable live.com

5. Structure From Motion (1) [Tomasi & Kanade 92]

6. Structure From Motion (2) [Tomasi & Kanade 92]

7. Structure From Motion (3) [Tomasi & Kanade 92]

8. Structure From Motion (4a): Images Marc Pollefeys

9. Structure From Motion (4b) Marc Pollefeys

10. Structure From Motion (5) http://www.cs.unc.edu/Research/urbanscape

11. Structure From Motion • Problem 1: • Given n points pij =(xij, yij) in m images • Reconstruct structure: 3-D locations Pj =(xj, yj, zj) • Reconstruct camera positions (extrinsics) Mi=(Aj, bj) • Problem 2: • Establish correspondence: c(pij)

12. Structure From Motion features camera Recover: structure (feature locations), motion (camera extrinsics)

13. Recovery Problems

14. SFM: General Formulation O X -x Z f

15. SFM: Bundle Adjustment O X -x Z f

16. Bundle Adjustment • SFM = Nonlinear Least Squares problem • Minimize through • Gradient Descent • Conjugate Gradient • Gauss-Newton • Levenberg Marquardt common method • Prone to local minima

17. Count # Constraints vs #Unknowns • m camera poses • n points • 2mn point constraints • 6m+3n unknowns • Suggests: need 2mn  6m + 3n • But: Can we really recover all parameters???

18. How Many Parameters Can’t We Recover? We can recover all but… m = #camera poses n = # feature points Place Your Bet!

19. Count # Constraints vs #Unknowns • m camera poses • n points • 2mn point constraints • 6m+3n unknowns • Suggests: need 2mn  6m + 3n • But: Can we really recover all parameters??? • Can’t recover origin, orientation (6 params) • Can’t recover scale (1 param) • Thus, we need 2mn  6m + 3n -7

20. Are we done? • No, bundle adjustment has many local minima.

21. The “Trick Of The Day” Replace Perspective by Orthographic Geometry Replace Euclidean Geometry by Affine Geometry Solve SFM linearly via PCA (“closed” form, globally optimal) Post-Process to make solution Euclidean Post-Process to make solution perspective By Tomasi and Kanade, 1992

22. Orthographic Camera Model Extrinsic Parameters Rotation Orthographic Projection Orthographic = Limit of Pinhole Model:

23. Orthographic Projection Limit of Pinhole Model: Orthographic Projection

24. The Orthographic SFM Problem subject to

25. The Affine SFM Problem drop the constraints subject to

26. Count # Constraints vs #Unknowns • m camera poses • n points • 2mn point constraints • 8m+3n unknowns • Suggests: need 2mn  8m + 3n • But: Can we really recover all parameters???

27. How Many Parameters Can’t We Recover? We can recover all but… Place Your Bet!

28. The Answer is (at least): 12 = A ' A C i i

29. Points for Solving Affine SFM Problem • m camera poses • n points • Need to have: 2mn  8m + 3n-12

30. Affine SFM Fix coordinate system by making pi0=P0=origin Rank Theorem: Q has rank 3 Proof:

31. The Rank Theorem 2m elements n elements

32. Singular Value Decomposition

33. Affine Solution to Orthographic SFM Gives also the optimal affine reconstruction under noise

34. Back To Orthographic Projection Find C for which constraints are met Search in 9-dim space (instead of 8m + 3n-12)

35. Back To Projective Geometry Orthographic (in the limit) Projective

36. Back To Projective Geometry O X -x Z f Optimize Using orthographic solution as starting point

37. The “Trick Of The Day” Replace Perspective by Orthographic Geometry Replace Euclidean Geometry by Affine Geometry Solve SFM linearly via PCA (“closed” form, globally optimal) Post-Process to make solution Euclidean Post-Process to make solution perspective By Tomasi and Kanade, 1992

38. Structure From Motion • Problem 1: • Given n points pij =(xij, yij) in m images • Reconstruct structure: 3-D locations Pj =(xj, yj, zj) • Reconstruct camera positions (extrinsics) Mi=(Aj, bj) • Problem 2: • Establish correspondence: c(pij)

39. The Correspondence Problem View 1 View 2 View 3

40. Correspondence: Solution 1 • Track features (e.g., optical flow) • …but fails when images taken from widely different poses

41. Correspondence: Solution 2 • Start with random solution A, b, P • Compute soft correspondence: p(c|A,b,P) • Plug soft correspondence into SFM • Reiterate See Dellaert/Seitz/Thorpe/Thrun, Machine Learning Journal, 2003

42. Example

43. Results: Cube

44. Animation

45. Tomasi’s Benchmark Problem

46. Reconstruction with EM

47. 3-D Structure

48. Correspondence: Alternative Approach • Ransac [Fisher/Bolles] = Random sampling and consensus • Will be discussed Wednesday

49. Summary SFM • Problem • Determine feature locations (=structure) • Determine camera extrinsic (=motion) • Two Principal Solutions • Bundle adjustment (nonlinear least squares, local minima) • SVD (through orthographic approximation, affine geometry) • Correspondence • (RANSAC) • Expectation Maximization