1 / 33

CV: 3D to 2D mathematics

CV: 3D to 2D mathematics. Perspective transformation; camera calibration; stereo computation; and more. Roadmap of topics. Review perspective transformation Camera calibration Stereo methods Structured light methods Depth-from-focus Shape-from-shading. Review coordinate systems.

jarrett
Download Presentation

CV: 3D to 2D mathematics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CV: 3D to 2D mathematics Perspective transformation; camera calibration; stereo computation; and more MSU CSE 240 Fall 2003 Stockman

  2. Roadmap of topics • Review perspective transformation • Camera calibration • Stereo methods • Structured light methods • Depth-from-focus • Shape-from-shading MSU CSE 240 Fall 2003 Stockman

  3. Review coordinate systems Camera or sensor D Camera or sensor C Object or model M World or global W MSU CSE 240 Fall 2003 Stockman

  4. Convenient notation for points and transformations This point P has 2 real coordinates in the image This point P has 3 real world coordinates in coordinate system W This transformation maps each point in the real world W to a point in the image I MSU CSE 240 Fall 2003 Stockman

  5. Current goal Develop the theory in terms of modules (components) so that concepts are understood and can be put into practical application MSU CSE 240 Fall 2003 Stockman

  6. Perspective transformation Camera origin is center of projection, not lens X and Y are scaled by the ratio of focal length to depth Z MSU CSE 240 Fall 2003 Stockman

  7. In next homework & project • fit camera model to image with jig • jig has known precise 3D coordinates • examine accuracy of camera model • use camera model to do graphics • use two camera models to compute depth from stereo MSU CSE 240 Fall 2003 Stockman

  8. Notes on perspective trans. • 3D world scaled according to ratio of depth to focal length • scaling formulas are in terms of real numbers with the same units e.g. mm in the 3D world and mm in the image plane • real image coordinates must be further scaled to pixel row and column • entire 3D ray images to the same 2D point MSU CSE 240 Fall 2003 Stockman

  9. Goal: General perspective trans to be developed (accept for now) Camera matrix C transforms 3D real world point into image row and column using 11 parameters MSU CSE 240 Fall 2003 Stockman

  10. The 11 parameters Cij model • internal camera parameters: focal length f ratio of pixel height and width any shear due to sensor chip alignment • external orientation parameters: rotation of camera frame relative to world frame translation of camera frame relative to world The 11 parameters of this model are NOT independent. Radial distortion is not linear and is not modeled. MSU CSE 240 Fall 2003 Stockman

  11. Camera matrix via least squares Minimize the residuals in the image plane. Get 2 equations for each pair ((r, c), (x, y, z)) MSU CSE 240 Fall 2003 Stockman

  12. 2 equations for each pair Known 3D points Here, (u, v) is the point in the image where 3D point (x,y,z) is projected. The 11 unknowns d jk form the camera matrix. Known image points Camera parameters MSU CSE 240 Fall 2003 Stockman

  13. 2n linear equations from n pairs ((u,v) (x,y,z)) Standard linear algebra problem; easily solved in Matlab or by using a linear algebra package. Often, package replaces b’s with the residuals. MSU CSE 240 Fall 2003 Stockman

  14. Use a jig for calibration • Jig has known set of points • Measure points in world system W or use the jig to define W • Take image with camera and determine 2D points Get pairings ((r, c) (x, y, z)) MSU CSE 240 Fall 2003 Stockman

  15. Example calibration data # # IMAGE: g1view1.ras# # INPUT DATA | OUTPUT DATA# |Point Image 2-D (U,V) 3-D Coordinates (X,Y,Z) | 2-D Fit Data Residuals X Y | A 95.00 336.00 0.00 0.00 0.00 | 94.53 337.89 | 0.47 -1.89 B 0.00 6.00 0.00 | | C 11.00 6.00 0.00 | | D 592.00 368.00 11.00 0.00 0.00 | 592.21 368.36 | -0.21 -0.36 N 501.00 363.00 9.00 0.00 0.00 | 501.16 362.78 | -0.16 0.22 O 467.00 279.00 8.25 0.00 -1.81 | 468.35 281.09 | -1.35 -2.09 P 224.00 266.00 2.75 0.00 -1.81 | 224.06 266.43 | -0.06 -0.43# CALIBRATION MATRIX 44.84 29.80 -5.504 94.53 2.518 42.24 40.79 337.9 -0.0006832 0.06489 -0.01027 1.000 MSU CSE 240 Fall 2003 Stockman

  16. 3D points on jig Dimensions in inches MSU CSE 240 Fall 2003 Stockman

  17. Jig set in workspace Mapping is established between 3D points (x, y, z) and image points (u, v) MSU CSE 240 Fall 2003 Stockman

  18. Other jigs used at MSU • frame with wires and beads placed in car instead of the driver seat (to do stereo measurements of car driver) • frame with wires and beads as big as a harp to calibrate space for people walking (up to 6 cameras, persons wear tight body suit with reflecting disks, cameras compute 3D motion trajectory) MSU CSE 240 Fall 2003 Stockman

  19. Least squares set up A X = B 11 x 1 = 2n x 1 2n x 11 MSU CSE 240 Fall 2003 Stockman

  20. Least squares abstraction MSU CSE 240 Fall 2003 Stockman

  21. Justify the form of camera matrix • Another sequence of slides • Rotation, scaling, shear in 3D real world as a 3x3 (or 4x4) matrix • Projection to real 2D image as 4x4 matrix • Scaling real image coordinates to [r, c] coordinates as 4x4 matrix • Combine them all into one 4x4 matrix MSU CSE 240 Fall 2003 Stockman

  22. Other mathematical models Two camera stereo MSU CSE 240 Fall 2003 Stockman

  23. Baseline stereo: carefully aligned cameras MSU CSE 240 Fall 2003 Stockman

  24. Computing (x, y, z) in 3D from corresponding 2D image points MSU CSE 240 Fall 2003 Stockman

  25. 2 calibrated cameras view the same 3D point at (r1,c1)(r2,c2) MSU CSE 240 Fall 2003 Stockman

  26. Compute closest approach of the two rays: use center point V Shortest line segment between rays MSU CSE 240 Fall 2003 Stockman

  27. Connector is perpendicular to both imaging rays MSU CSE 240 Fall 2003 Stockman

  28. Solve for the endpoints of the connector Scaler mult. Fix book MSU CSE 240 Fall 2003 Stockman

  29. Correspondence problem: more difficult aspect MSU CSE 240 Fall 2003 Stockman

  30. Correspondence problem is difficult • Can use interest points and cross correlation • Can limit search to epipolar line • Can use symbolic matching (Ch 11) to determine corresponding points (called structural stereopsis) • apparently humans don’t need it MSU CSE 240 Fall 2003 Stockman

  31. Epipolar constraint With aligned cameras, search for corresponding point is 1D along corresponding row of other camera. MSU CSE 240 Fall 2003 Stockman

  32. Epipolar constraint for non baseline stereo computation Need to know relative orientation of cameras C1 and C2 If cameras are not aligned, a 1D search can still be determined for the corresponding point. P1, C1, C2 determine a plane that cuts image I2 in a line: P2 will be on that line. MSU CSE 240 Fall 2003 Stockman

  33. Measuring driver body position 4 cameras were used to measure driver position and posture while driving: 2mm accuracy achieved MSU CSE 240 Fall 2003 Stockman

More Related