580 likes | 718 Views
CSCE 643 Computer Vision: Structure from Motion. Jinxiang Chai. Stereo reconstruction. Given two or more images of the same scene or object, compute a representation of its shape. known camera viewpoints. Stereo reconstruction.
E N D
CSCE 643 Computer Vision: Structure from Motion Jinxiang Chai
Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints
Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints How to estimate camera parameters? - where is the camera? - where is it pointing? - what are internal parameters, e.g. focal length?
Calibration from 2D motion • Structure from motion (SFM) • - track points over a sequence of images • - estimate for 3D positions and camera positions • - calibrate intrinsic camera parameters before hand • Self-calibration: • - solve for both intrinsic and extrinsic camera parameters
SFM = Holy Grail of 3D Reconstruction • Take movie of object • Reconstruct 3D model • Would be • commercially • highly viable
How to Get Feature Correspondences • Feature-based approach • - good for images • - feature detection (corners or sift features) • - feature matching using RANSAC (epipolar line) • Pixel-based approach • - good for video sequences • - patch based registration with lucas-kanade algorithm • - register features across the entire sequence
A Brief Introduction on Feature-based Matching • Find a few important features (aka Interest Points) • Match them across two images • Compute image transformation function h
Feature Detection • Two images taken at the same place with different angles • Projective transformationH3X3
Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3
Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3 How do we match features across images? Any criterion?
Feature Matching ? • Two images taken at the same place with different angles • Projective transformation H3X3 How do we match features across images? Any criterion?
Feature Matching • Intensity/Color similarity • The intensity of pixels around the corresponding features should have similar intensity
Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD
Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD • Distance constraint • The displacement of features should be smaller than a given threshold
Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD • Distance constraint • The displacement of features should be smaller than a given threshold • Epipolar line constraint • The corresponding pixels satisfy epipolar line constraints.
Feature Matching • Feature similarity (Intensity or SIFT signature) • The intensity of pixels around the corresponding features should have similar intensity • Cross-correlation, SSD • Distance constraint • The displacement of features should be smaller than a given threshold • Epipolar line constraint • The corresponding pixels satisfy epipolar line constraints. Fundamental matrix H
Feature-space Outlier Rejection bad Good
Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points?
Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points?
Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points? • No! Still too many outliers…
Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points? • No! Still too many outliers… • What can we do?
Feature-space Outlier Rejection • Can we now compute H3X3 from the blue points? • No! Still too many outliers… • What can we do? Robust estimation!
Robust Estimation: A Toy Example • How to fit a line based on a set of 2D points?
RANSAC for Estimating Projective Transformation • RANSAC loop: • Select four feature pairs (at random) • Compute the transformation matrix H (exact) • Compute inliers where SSD(pi’, H pi) < ε • Keep largest set of inliers • Re-compute least-squares H estimate on all of the inliers For more detail, check - http://research.microsoft.com/en-us/um/people/zhang/INRIA/software-FMatrix.html - Philip H. S. Torr (1997). "The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix". International Journal of Computer Vision24 (3): 271–300
Structure from Motion • Two Principal Solutions • Bundle adjustment (nonlinear optimization) • Factorization (SVD, through orthographic approximation, affine geometry)
Projection Matrix Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: K R T P
Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM?
Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D
Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D
Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?
Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?
SFM: Bundle Adjustment • SFM = Nonlinear Least Squares problem • Minimize through • Gradient Descent • Conjugate Gradient • Gauss-Newton • Levenberg Marquardt common method • Prone to local minima
Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N + 4 (unknowns) • Suggests: need 2mn 6m + 3n+4 • But: Can we really recover all parameters???
Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N+4 unknowns (known intrinsic camera parameters) • Suggests: need 2mn 6m + 3n+4 • But: Can we really recover all parameters??? • Can’t recover origin, orientation (6 params) • Can’t recover scale (1 param) • Thus, we need 2mn 6m + 3n+4 -7
Are We Done? • No, bundle adjustment has many local minima.
Image World SFM Using Factorization • Assume an orthographic camera
Image World SFM Using Factorization • Assume orthographic camera Subtract the mean
SFM Using Factorization Stack all the features from the same frame:
SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W
SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W
SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: W
SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD:
SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: How to compute the matrix ? W
SFM Using Factorization M is the stack of rotation matrix:
SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1
SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1
SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 0 1
SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1
SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1 How to compute QQT? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix)
SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1 How to compute QQT? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) How to compute Q from QQT: SVD again: