1 / 42

CSCE 641 Computer Graphics: Image-based Modeling (Cont.)

CSCE 641 Computer Graphics: Image-based Modeling (Cont.). Jinxiang Chai. Image-based Modeling and Rendering. Model. Panoroma. Image-based rendering. Image based modeling. Images + Depth. Geometry+ Images. Camera + geometry. Images user input range scans. Images. Geometry+ Materials.

liv
Download Presentation

CSCE 641 Computer Graphics: Image-based Modeling (Cont.)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai

  2. Image-based Modeling and Rendering Model Panoroma Image-based rendering Image based modeling Images + Depth Geometry+ Images Camera + geometry Imagesuser input range scans Images Geometry+ Materials Light field Kinematics Dynamics Etc.

  3. Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints

  4. Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints How to estimate camera parameters? - where is the camera? - where is it pointing? - what are internal parameters, e.g. focal length?

  5. Spectrum of IBMR Model Panoroma Image-based rendering Image based modeling Images + Depth Geometry+ Images Camera + geometry Imagesuser input range scans Images Geometry+ Materials Light field Kinematics Dynamics Etc.

  6. Calibration from 2D motion • Structure from motion (SFM) • - track points over a sequence of images • - estimate for 3D positions and camera positions • - calibrate intrinsic camera parameters before hand • Self-calibration: • - solve for both intrinsic and extrinsic camera parameters

  7. SFM = Holy Grail of 3D Reconstruction • Take movie of object • Reconstruct 3D model • Would be • commercially • highly viable

  8. How to Get Feature Correspondences • Feature-based approach • - good for images • - feature detection (corners or sift features) • - feature matching using RANSAC (epipolar line) • Pixel-based approach • - good for video sequences • - patch based registration with lucas-kanade algorithm • - register features across the entire sequence

  9. Structure from Motion • Two Principal Solutions • Bundle adjustment (nonlinear optimization) • Factorization (SVD, through orthographic approximation, affine geometry)

  10. Projection Matrix Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: K R T P

  11. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM?

  12. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D

  13. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D

  14. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?

  15. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?

  16. SFM: Bundle Adjustment • SFM = Nonlinear Least Squares problem • Minimize through • Gradient Descent • Conjugate Gradient • Gauss-Newton • Levenberg Marquardt common method • Prone to local minima

  17. Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N unknowns • Suggests: need 2mn  6m + 3n • But: Can we really recover all parameters???

  18. Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N unknowns (known intrinsic camera parameters) • Suggests: need 2mn  6m + 3n • But: Can we really recover all parameters??? • Can’t recover origin, orientation (6 params) • Can’t recover scale (1 param) • Thus, we need 2mn  6m + 3n -7

  19. Are We Done? • No, bundle adjustment has many local minima.

  20. Image World SFM Using Factorization • Assume an othorgraphic camera

  21. Image World SFM Using Factorization • Assume othorgraphic camera Subtract the mean

  22. SFM Using Factorization Stack all the features from the same frame:

  23. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

  24. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

  25. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: W

  26. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: Is the solution unique? W

  27. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: Is the solution unique? No!

  28. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: How to compute the matrix ? W

  29. SFM Using Factorization M is the stack of rotation matrix:

  30. SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1

  31. SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1

  32. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 0 1

  33. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1

  34. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1 How to compute QQT? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix)

  35. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1 How to compute QQT? least square solution - 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) How to compute Q from QQT: SVD again:

  36. SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1 QQT: symmetric 3 by 3 matrix • Computing QQT is easy: • 3F linear equations • 6 independent unknowns

  37. SFM Using Factorization • Form the measurement matrix • Decompose the matrix into two matrices and using SVD • Compute the matrix Q with least square and SVD • Compute the rotation matrix and shape matrix: • and

  38. Weak-perspective Projection • Factorization also works for weak-perspective projection (scaled orthographic projection): z0 d

  39. Factorization for Full-perspective Cameras [Han and Kanade]

  40. SFM Summary Bundle adjustment (nonlinear optimization) - work with perspective camera model - work with incomplete data - prone to local minima Factorization: - closed-form solution for weak perspective camera - simple and efficient - usually need complete data - becomes complicated for full-perspective camera model Phil Torr’s structure from motion toolkit in matlab (click here) Voodoo camera tracker (click here)

  41. All Together Video Click here - feature detection - feature matching (epipolar geometry) - structure from motion - stereo reconstruction - triangulation - texture mapping

  42. SFM: Recent Development • Large Scale SFM: • Reconstructing the World from Internet Photos • Project website: • source code and data sets are available from http://www.cs.cornell.edu/projects/bigsfm/

More Related