Feature Tracking and Triangulation Strategies for 3D Reconstruction

1. KLT tracker & triangulationClass 6 Read Shi and Tomasi’s paper on good features to track http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf Optional: Lucas-Kanade 20 Years Onhttp://www.ri.cmu.edu/projects/project_515.html

2. Feature matching vs. tracking Image-to-image correspondences are key to passive triangulation-based 3D reconstruction Extract features independently and then match by comparing descriptors Extract features in first images and then try to find same feature back in next view What is a good feature?

3. Feature point extraction • Approximate SSD for small displacement Δ • Find points for which the following is maximum maximize smallest eigenvalue of M

4. SIFT features • Scale-space DoG maxima • Verify minimum contrast and “cornerness” • Orientation from dominant gradient • Descriptor based on gradient distributions

5. Feature tracking • Identify features and track them over video • Small difference between frames • potential large difference overall • Standard approach: KLT (Kanade-Lukas-Tomasi)

6. Intermezzo: optical flow • Brightness constancy assumption (small motion) • 1D example possibility for iterative refinement

7. Intermezzo: optical flow • Brightness constancy assumption (small motion) • 2D example the “aperture” problem (1 constraint) ? (2 unknowns) isophote I(t+1)=I isophote I(t)=I

8. Intermezzo: optical flow • How to deal with aperture problem? (3 constraints if color gradients are different) Assume neighbors have same displacement

9. Lucas-Kanade Assume neighbors have same displacement least-squares:

10. Alternative derivation • Compute translation assuming it is small differentiate: • Affine is also possible, but a bit harder (6x6 in stead of 2x2)

11. Revisiting the small motion assumption • Is this motion small enough? • Probably not—it’s much larger than one pixel (2nd order terms dominate) • How might we solve this problem? * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

12. Reduce the resolution! * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

13. u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image It-1 image It-1 image I image I Gaussian pyramid of image It-1 Gaussian pyramid of image I Coarse-to-fine optical flow estimation slides from Bradsky and Thrun

14. warp & upsample run iterative L-K . . . image J image It-1 image I image I Gaussian pyramid of image It-1 Gaussian pyramid of image I Coarse-to-fine optical flow estimation slides from Bradsky and Thrun run iterative L-K

15. Good feature to track • Tracking • Use same window in feature selection as for tracking itself maximize minimal eigenvalue of M Strategy: • Look for strong well distributed features, typically few hundreds • initialize and then track, renew feature when too many are lost

16. Example Simple displacement is sufficient between consecutive frames, but not to compare to reference template

17. Example

18. Synthetic example

19. Good features to keep tracking Perform affine alignment between first and last frame Stop tracking features with too large errors

20. Live demo • OpenCV (try it out!) LKdemo

21. L2 M m2 C2 Triangulation m1 C1 L1 Triangulation • calibration • correspondences

22. Triangulation • Backprojection • Triangulation Iterative least-squares • Maximum Likelihood Triangulation

23. Backprojection • Represent point as intersection of row and column • Condition for solution? Useful presentation for deriving and understanding multiple view geometry (notice 3D planes are linear in 2D point coordinates)

24. Next class: epipolar geometry