1 / 56

Understanding Calibration, Triangulation, and Structure from Motion

This presentation covers concepts of calibration, triangulation, and structure from motion for robotic perception, including epipolar geometry, stereo matching, camera calibration methods, and more. Learn about camera parameters, depth information, and stereo vision applications.

bonniejones
Download Presentation

Understanding Calibration, Triangulation, and Structure from Motion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Calibration, Triangulation and Structure from Motion Thursday, 10th October 2019 Samyak Datta Disclaimer: These slides have been borrowed from Derek Hoiem, Kristen Grauman and the Coursera course on Robotic Perception (by Jianbo Shi and Kostas Danilidis). Derek adapted many slides from Lana Lazebnik, Silvio Saverese, Steve Seitz, and took many figures from Hartley & Zisserman.

  2. Reminder • PS3 is due on 10/16

  3. Reminder • PS3 is due on 10/16 • Project proposals were due last night • 2 project late days • We’ll assign project teams to TAs

  4. Topics overview • Intro • Features & filters • Grouping & fitting • Multiple views and motion • Homography and image warping • Image formation • Epipolar geometry and stereo • Structure from motion • Recognition • Video processing Slide credit: Kristen Grauman

  5. Multi-View Geometry • Last time: Epipolar geometry • Relates cameras from two positions • Last time: Scan-line stereo • Finding corresponding points in a pair of stereo images • Structure from motion • Multiple views

  6. Recap: Epipolar geometry X x x’ • Baseline – line connecting the two camera centers • Epipoles = intersections of baseline with image planes = projections of the other camera center • Epipolar Plane – plane containing baseline (1D family) • Epipolar Lines - intersections of epipolar plane with image planes (always come in corresponding pairs)

  7. Key idea: Epipolar constraint X X X x x’ x’ x’ Potential matches for x have to lie on the corresponding line l’. Potential matches for x’ have to lie on the corresponding line l.

  8. Basic stereo matching algorithm • For each pixel in the first image • Find corresponding epipolar line in the right image • Search along epipolar line and pick the best match • Triangulate the matches to get depth information • Simplest case: epipolar lines are scanlines • When does this happen?

  9. Simplest Case: Parallel images • Image planes of cameras are parallel to each other and to the baseline • Camera centers are at same height • Focal lengths are the same • Then, epipolar lines fall along the horizontal scan lines of the images

  10. Simplest Case: Parallel images Epipolar constraint: R = I t = (T, 0, 0) x x’ t The y-coordinates of corresponding points are the same

  11. Basic stereo matching algorithm • If necessary, rectify the two stereo images to transform epipolar lines into scanlines • For each pixel x in the first image • Find corresponding epipolar scanline in the right image • Search the scanline and pick the best match x’

  12. Correspondence search Left Right • Slide a window along the right scanline and compare contents of that window with the reference window in the left image • Matching cost: SSD or normalized correlation scanline Matching cost disparity

  13. Correspondence search Left Right scanline SSD

  14. Correspondence search Left Right scanline Norm. corr

  15. Failures of correspondence search Occlusions, repetition Textureless surfaces

  16. Depth from disparity X Similar triangles: z x x’ f f BaselineB O O’ disparity Disparity is inversely proportional to depth.

  17. Topics overview • Class Intro • Features & filters • Grouping & fitting • Multiple views and motion • Homography and image warping • Image formation • Epipolar geometry and stereo • Calibration, Triangulation and Structure from motion • Recognition • Video processing Slide credit: Kristen Grauman

  18. General Case: Stereo with calibrated cameras Given image pair, R, T, and camera parameters K, K’ Detect some features Compute essential matrix E Match features using the epipolar and other constraints Triangulate for 3d structure Not yet discussed: • How to calibrate a camera? • How to triangulate a 3D point?

  19. Today • Calibration • Triangulation • Structure from Motion • Example stereo and SfM applications

  20. How to calibrate the camera?

  21. Calibrating the Camera Method 1: Weak calibration • Using corresponding points between two images to estimate the fundamental matrix F

  22. Calibrating the Camera Method 1: Weak calibration • Use the 8-point algorithm • Only available approach if the camera is not accessible, e.g. internet images

  23. Calibrating the Camera Method 2: Use an object (calibration grid) with known geometry • Correspond image points to 3d points • Get least squares solution (or non-linear solution)

  24. Linear method • Solve using linear least squares Ax=0 form

  25. Calibrating the Camera • We have matrix P, which also depends on the camera pose. How to get K? • The first 3x3 submatrix, M, of P is the product (M = KR) of an upper triangular matrix K and rotation matrix R. • Factor M into KR using the QR matrix decomposition • t = K-1(p14, p24, p34)T

  26. Calibration with linear method • Advantages: easy to formulate and solve • Disadvantages • Doesn’t tell you camera parameters • Can’t impose constraints, such as known focal length • Doesn’t minimize projection error • Non-linear methods are preferred • Define error as difference between projected points and measured points • Minimize error using Newton’s method or other non-linear optimization • Use linear method for initialization

  27. Calibrating the Camera Vertical vanishing point (at infinity) Vanishing line Vanishing point Vanishing point Method 3: Use vanishing points • Find vanishing points corresponding to orthogonal directions

  28. Calibrating the Camera

  29. Calibrating the Camera

  30. Calibrating the Camera

  31. Calibrating the Camera

  32. Calibrating the Camera

  33. Calibrating the Camera

  34. Calibrating the Camera

  35. Today • Calibration • Triangulation • Structure from Motion • Example stereo and SfM applications

  36. Triangulation: Linear Solution

  37. Triangulation: Linear Solution

  38. Triangulation: Linear Solution

  39. Triangulation: Linear Solution

  40. Triangulation: Non-linear Solution

  41. Triangulation: Non-linear Solution

  42. Triangulation: Non-linear Solution

  43. Triangulation: Non-linear Solution

  44. General Case: Stereo with calibrated cameras Given image pair, R, T, and camera parameters K, K’ Detect some features Compute essential matrix E Match features using the epipolar and other constraints Triangulate for 3d structure

  45. Structure from motion

  46. Projective structure from motion Xj x1j x3j x2j P1 P3 P2 • Given: m images of n fixed 3D points • xij = Pi Xj , i = 1,… , m, j = 1, … , n • Problem: estimate m projection matrices Pi and n 3D points Xj from the mn corresponding 2D points xij Slides from Lana Lazebnik

  47. Reconstruction ambiguity • If two cameras are calibrated, then reconstruction is possible up to a similarity transform • With no calibration info, we face projective ambiguity. A projective transformation Q of the structure and camera positions does not change the measured points, i.e., X = PX = (PQ-1)(QX) Further reading: Hartley & Zisserman p. 264-266

  48. Projective structure from motion • Given: m images of n fixed 3D points • xij = Pi Xj , i = 1,… , m, j = 1, … , n • Problem: estimate m projection matrices Pi and n 3D points Xj from the mn corresponding points xij • With no calibration info, cameras and points can only be recovered up to a 4x4 projective transformation Q: • X → QX, P → PQ-1 • We can solve for structure and motion when • 2mn >= 11m +3n – 15 • For two cameras, at least 7 points are needed

  49. Sequential structure from motion • Initialize motion from two images using fundamental matrix • Initialize structure by triangulation • For each additional view: • Determine projection matrix of new camera using all the known 3D points that are visible in its image – calibration points cameras

  50. Sequential structure from motion • Initialize motion from two images using fundamental matrix • Initialize structure by triangulation • For each additional view: • Determine projection matrix of new camera using all the known 3D points that are visible in its image – calibration • Refine and extend structure: compute new 3D points, re-optimize existing points that are also seen by this camera – triangulation points cameras

More Related