CSCE 643 Computer Vision: Structure from Motion

1. CSCE 643 Computer Vision: Structure from Motion Jinxiang Chai

2. Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints

3. Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints How to estimate camera parameters? - where is the camera? - where is it pointing? - what are internal parameters, e.g. focal length?

4. Calibration from 2D motion • Structure from motion (SFM) • - track points over a sequence of images • - estimate for 3D positions and camera positions • - calibrate intrinsic camera parameters before hand • Self-calibration: • - solve for both intrinsic and extrinsic camera parameters

5. SFM = Holy Grail of 3D Reconstruction • Take movie of object • Reconstruct 3D model • Would be • commercially • highly viable

6. How to Get Feature Correspondences • Feature-based approach • - good for images • - feature detection (corners or sift features) • - feature matching using RANSAC (epipolar line) • Pixel-based approach • - good for video sequences • - patch based registration with lucas-kanade algorithm • - register features across the entire sequence

7. A Brief Introduction on Feature-based Matching • Find a few important features (aka Interest Points) • Match them across two images • Compute image transformation function h

8. Feature Detection • Two images taken at the same place with different angles • Projective transformationH3X3

9. Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3

10. Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3 How do we match features across images? Any criterion?

11. Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3 How do we match features across images? Any criterion?

12. Feature Matching • Intensity/Color similarity • The intensity of pixels around the corresponding features should have similar intensity

13. Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD

14. Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD • Distance constraint • The displacement of features should be smaller than a given threshold

15. Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD • Distance constraint • The displacement of features should be smaller than a given threshold • Epipolar line constraint • The corresponding pixels satisfy epipolar line constraints.

16. Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD • Distance constraint • The displacement of features should be smaller than a given threshold • Epipolar line constraint • The corresponding pixels satisfy epipolar line constraints. Fundamental matrix H

17. Feature-space Outlier Rejection bad Good

18. Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points?

19. Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points?

20. Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points? • No! Still too many outliers…

21. Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points? • No! Still too many outliers… • What can we do?

22. Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points? • No! Still too many outliers… • What can we do? Robust estimation!

23. Robust Estimation: A Toy Example • How to fit a line based on a set of 2D points?

24. RANSAC for Estimating Projective Transformation • RANSAC loop: • Select four feature pairs (at random) • Compute the transformation matrix H (exact) • Compute inliers where SSD(pi’, H pi) < ε • Keep largest set of inliers • Re-compute least-squares H estimate on all of the inliers For more detail, check - http://research.microsoft.com/en-us/um/people/zhang/INRIA/software-FMatrix.html - Philip H. S. Torr (1997). "The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix". International Journal of Computer Vision24 (3): 271–300

25. Structure from Motion • Two Principal Solutions • Bundle adjustment (nonlinear optimization) • Factorization (SVD, through orthographic approximation, affine geometry)

26. Projection Matrix Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: K R T P

27. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM?

28. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D

29. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D

30. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?

31. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?

32. SFM: Bundle Adjustment • SFM = Nonlinear Least Squares problem • Minimize through • Gradient Descent • Conjugate Gradient • Gauss-Newton • Levenberg Marquardt common method • Prone to local minima

33. Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N + 4 (unknowns) • Suggests: need 2mn  6m + 3n+4 • But: Can we really recover all parameters???

34. Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N+4 unknowns (known intrinsic camera parameters) • Suggests: need 2mn  6m + 3n+4 • But: Can we really recover all parameters??? • Can’t recover origin, orientation (6 params) • Can’t recover scale (1 param) • Thus, we need 2mn  6m + 3n+4 -7

35. Are We Done? • No, bundle adjustment has many local minima.

36. Image World SFM Using Factorization • Assume an orthographic camera

37. Image World SFM Using Factorization • Assume orthographic camera Subtract the mean

38. SFM Using Factorization Stack all the features from the same frame:

39. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

40. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

41. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: W

42. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD:

43. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: How to compute the matrix ? W

44. SFM Using Factorization M is the stack of rotation matrix:

45. SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1

46. SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1

47. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 0 1

48. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1

49. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1 How to compute QQT? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix)

50. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1 How to compute QQT? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) How to compute Q from QQT: SVD again: