Mastering 3D Vision: Step-by-Step Model Building Techniques

1. Lecture 7Step-by-Step Model Buidling Invitation to 3D vision

2. Review Feature selection Feature selection Feature correspondence Camera Calibration Landing Augmented Reality Euclidean Reconstruction Vision Based Control Sparse Structure and camera motion Invitation to 3D vision

3. Review Feature selection Feature selection Feature correspondence Camera Calibration Epipolar Rectification Dense Correspondence Texture mapping Euclidean Reconstruction Sparse Structure and motion 3-D Model Invitation to 3D vision

4. Partial Scene Knowledge Partial Motion Knowledge Partial Calibration Knowledge Review Feature selection Feature selection Feature correspondence Projective Reconstruction Epipolar Rectification Camera Self-Calibration Dense Correspondence Texture mapping Euclidean Reconstruction 3-D Model Invitation to 3D vision

5. Examples Invitation to 3D vision

6. Feature Selection • Compute Image Gradient • Compute Feature Quality measure for each pixel • Search for local maxima Feature Quality Function Local maxima of feature quality function Invitation to 3D vision

7. Feature Tracking • Translational motion model • Closed form solution • Build an image pyramid • Start from coarsest level • Estimate the displacement at the coarsest level • Iterate until finest level Invitation to 3D vision

8. Coarse to fine feature tracking 0 1 2 • compute • warp the window in the second image by • update the displacement • go to finer level • At the finest level repeat for several iterations Invitation to 3D vision

9. Optical Flow • Integrate around over image patch • Solve Invitation to 3D vision

10. Affine feature tracking Intensity offset Contrast change Invitation to 3D vision

11. Tracked Features Invitation to 3D vision

12. Wide baseline matching Point features detected by Harris Corner detector Invitation to 3D vision

13. Wide baseline Feature Matching • Select the features in two views • For each feature in the first view • Find the feature in the second view that maximizes • Normalized cross-correlation measure Select the candidate with the similarity above selected threshold Invitation to 3D vision

14. More correspondences and Robust matching • Select set of putative correspondences • Repeat 1. Select at random a set of 8 successful matches 2. Compute fundamental matrix 3. Determine the subset of inliers, compute distance to epipolar line 4. Count the number of points in the consensus set Invitation to 3D vision

15. RANSAC in action Inliers Outliers Invitation to 3D vision

16. Epipolar Geometry • Epipolar geometry in two views • Refined epipolar geometry using nonlinear estimation of F Invitation to 3D vision

17. Two view initialization calibrated • Recover epipolar geometry • Compute (Euclidean) projection matrices and 3-D struct. • Compute (Projective) projection matrices and 3-D struct. uncalibrated unknown Invitation to 3D vision

18. 3-D reconstruction from two views Projective Reconstruction Invitation to 3D vision

19. Repeat until convergence 2. Refine structure estimate given all the motions Multi-view reconstruction • Two view - initialized motion and structure estimates (scales) • Multi-view factorization - recover the remaining camera positions and refine the 3-D structure by iteratively computing 1. Compute i-th motion given the known structure iteration Invitation to 3D vision

20. Projective Ambiguity • calibrated case – projection matrices - and Euclidean structure • Uncalibrated case • for i-th frame • for 1st frame • Transform projection matrices and 3-D structure by H Invitation to 3D vision

21. Upgrade to Euclidean Reconstruction • From Projective to Euclidean Reconstruction • Remove the ambiguity characterized by • How to estimate H ? - Absolute quadric constraint • be recovered by a nonlinear minimization unknowns Invitation to 3D vision

22. Upgrade to Euclidean Reconstruction • Special case unknown focal length • other internal parameters are known (proj. center, aspect ratio) • Q can be parametrized in the following way • Zero entries in lead to linear constraints on Q • Initial estimate of Q can be obtained using linear techniques Invitation to 3D vision

23. Example of multi-view reconstruction Projective Reconstruction Euclidean reconstruction Invitation to 3D vision

24. Nonlinear Refinement • Euclidean Bundle adjustment • Initial estimates of are available • Final refinement, nonlinear minimization with respect to all unknowns Invitation to 3D vision

25. Example - Euclidean multi-view reconstruction Invitation to 3D vision

26. Example Original sequence Tracked Features Invitation to 3D vision

27. Recovered model Invitation to 3D vision

28. Euclidean Reconstruction Invitation to 3D vision

29. 1. Map the epipole to infinity Translate the image center to the origin Rotate around z-axis for the epipole lie on the x-axis Transform the epipole from x-axis to infinity 2. Find a matching transformation is compatible with the epipolar geometry is chosen to minimize overall disparity Epipolar rectification • Make the epipolar lines parallel • Dense correspondences along image scanlines • Computation of warping homographies Invitation to 3D vision

30. Epipolar rectification Rectified Image Pair Invitation to 3D vision

31. Epipolar rectification Rectified Image Pair Invitation to 3D vision

32. Dense Matching • Establish dense correspondences along scan-lines • Standard stereo configuration • Constraints to guide the search 1. ordering constraint 2. disparity constraint – limit on disparity 3. uniqueness constraint – each point has a unique match in the second view Invitation to 3D vision

33. Dense Matching Invitation to 3D vision

34. Dense Reconstruction Invitation to 3D vision

35. Texture mapping, hole filling Invitation to 3D vision

36. Texture mapping Invitation to 3D vision