CSCE 641 Computer Graphics: Image-based Modeling (Cont.)

1. CSCE 641 Computer Graphics: Image-based Modeling (Cont.) Jinxiang Chai

2. Image-based Modeling and Rendering Model Panoroma Image-based rendering Image based modeling Images + Depth Geometry+ Images Camera + geometry Imagesuser input range scans Images Geometry+ Materials Light field Kinematics Dynamics Etc.

3. Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints

4. Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape known camera viewpoints How to estimate camera parameters? - where is the camera? - where is it pointing? - what are internal parameters, e.g. focal length?

5. Spectrum of IBMR Model Panoroma Image-based rendering Image based modeling Images + Depth Geometry+ Images Camera + geometry Imagesuser input range scans Images Geometry+ Materials Light field Kinematics Dynamics Etc.

6. How can we estimate the camera parameters?

7. Camera calibration • Augmented pin-hole camera • - focal point, orientation • - focal length, aspect ratio, center, lens distortion Known 3D • Classical calibration • - 3D 2D • - correspondence Camera calibration online resources

8. Camera and calibration target

9. Classical camera calibration • Known 3D coordinates and 2D coordinates • - known 3D points on calibration targets • - find corresponding 2D points in image using feature detection • algorithm

10. Camera parameters Known 3D coords and 2D coords sx а u0 u 0 -sy v0 v 0 0 1 1 Viewport proj. Perspective proj. View trans.

11. Camera parameters Known 3D coords and 2D coords sx а u0 u 0 -sy v0 v 0 0 1 1 Viewport proj. Perspective proj. View trans. Intrinsic camera parameters (5 parameters) extrinsic camera parameters (6 parameters)

12. Camera matrix • Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into acamera matrix • M = K [R | t ] • (put 1 in lower r.h. corner for 11 d.o.f.)

13. Camera matrix calibration • Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

14. Camera matrix calibration • Linear regression: • Bring denominator over, solve set of (over-determined) linear equations. How?

15. Camera matrix calibration • Linear regression: • Bring denominator over, solve set of (over-determined) linear equations. How? • Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient

16. Nonlinear camera calibration • Perspective projection:

17. Nonlinear camera calibration • Perspective projection: K R T P

18. Nonlinear camera calibration • Perspective projection: • 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: K R T P

19. Nonlinear camera calibration • Perspective projection: • 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: K R T P

20. Multiple calibration images • Find camera parameters which satisfy the constraints from M images, N points: • for j=1,…,M • for i=1,…,N • This can be formulated as a nonlinear optimization problem:

21. Multiple calibration images • Find camera parameters which satisfy the constraints from M images, N points: • for j=1,…,M • for i=1,…,N • This can be formulated as a nonlinear optimization problem: Solve the optimization using nonlinear optimization techniques: - Gauss-newton - Levenberg-Marquardt

22. Nonlinear approach • Advantages: • can solve for more than one camera pose at a time • fewer degrees of freedom than linear approach • Standard technique in photogrammetry, computer vision, computer graphics - [Tsai 87] also estimates lens distortions (freeware @ CMU)http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html • Disadvantages: • more complex update rules • need a good initialization (recover K [R | t] from M)

23. How can we estimate the camera parameters?

24. Application: camera calibration for sports video images Court model [Farin et. Al]

25. Calibration from 2D motion • Structure from motion (SFM) • - track points over a sequence of images • - estimate for 3D positions and camera positions • - calibrate intrinsic camera parameters before hand • Self-calibration: • - solve for both intrinsic and extrinsic camera parameters

26. SFM = Holy Grail of 3D Reconstruction • Take movie of object • Reconstruct 3D model • Would be • commercially • highly viable

27. How to Get Feature Correspondences • Feature-based approach • - good for images • - feature detection (corners) • - feature matching using RANSAC (epipolar line) • Pixel-based approach • - good for video sequences • - patch based registration with lucas-kanade algorithm • - register features across the entire sequence

28. Structure from Motion • Two Principal Solutions • Bundle adjustment (nonlinear optimization) • Factorization (SVD, through orthographic approximation, affine geometry)

29. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM?

30. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D

31. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D

32. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?

33. Nonlinear Approach for SFM • What’s the difference between camera calibration and SFM? • - camera calibration: known 3D and 2D • - SFM: unknown 3D and known 2D • - what’s 3D-to-2D registration problem?

34. SFM: Bundle Adjustment • SFM = Nonlinear Least Squares problem • Minimize through • Gradient Descent • Conjugate Gradient • Gauss-Newton • Levenberg Marquardt common method • Prone to local minima

35. Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N unknowns • Suggests: need 2mn  6m + 3n • But: Can we really recover all parameters???

36. Count # Constraints vs #Unknowns • M camera poses • N points • 2MN point constraints • 6M+3N unknowns (known intrinsic camera parameters) • Suggests: need 2mn  6m + 3n • But: Can we really recover all parameters??? • Can’t recover origin, orientation (6 params) • Can’t recover scale (1 param) • Thus, we need 2mn  6m + 3n -7

37. Are We Done? • No, bundle adjustment has many local minima.

38. Image World SFM Using Factorization • Assume an othorgraphic camera

39. Image World SFM Using Factorization • Assume othorgraphic camera Subtract the mean

40. SFM Using Factorization Stack all the features from the same frame:

41. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

42. SFM Using Factorization Stack all the features from the same frame: Stack all the features from all the images: W

43. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: W

44. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD:

45. SFM Using Factorization Stack all the features from all the images: Factorize the matrix into two matrix using SVD: How to compute the matrix ? W

46. SFM Using Factorization M is the stack of rotation matrix:

47. SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1

48. SFM Using Factorization M is the stack of rotation matrix: 1 0 Orthogonal constraints from rotation matrix 0 1 1 0 0 1

49. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 0 1

50. SFM Using Factorization Orthogonal constraints from rotation matrices: 1 0 0 1 1 0 QQ: symmetric 3 by 3 matrix 0 1