Feature Detection and Matching in Images with Radial Distortion

1. Feature Detection and Matching in Images with Radial Distortion Miguel Lourenço, João P. Barreto, Abed Malti Institute for Systems and Robotics, Faculty of Science and Technology University of Coimbra, Portugal

2. Presentation Outline • SIFT Features – brief overview • RD problems in keypoint detection and matching • Theoretical reasoning • Experimental validation • Improvement to the SIFT algorithm to enhance it with RD • Real experiments – a comparison study • Motion estimation and 3D reconstruction in endoscopic images

3. Motivation for keypoint detection and matching • Point correspondence across multiple views • Camera calibration • Sparse 3D reconstruction • Recover camera/robot motion • Visual Slam • Representation of image content • Image retrieval applications • Recognition tasks (e.g. Voc-tree) • Image compression Partioning of the descriptor space

4. SIFT Features (Lowe, IJCV 2004 – 6725 citations on google scholar) • SIFT is probably the most broadly used algorithm for keypoint detection and matching • How does SIFT work ? • Image salient points detected in a scale space framework (x,y,sigma) Increase scale Gaussian pyramid DoG pyramid

5. SIFT Features (Lowe, IJCV 2004 – 6725 citations on Scholar google) • SIFT is probably the most broadly used algorithm for keypoint detection and matching • How does SIFT work ? • Image salient points detected in a scale space framework • SIFT descriptor is computed based on local image gradient on a scale and rotation normalized patch

6. Problem statement (1/2) Q • What is radial distortion? • Bending of the light rays pulling image points towards the center along radial direction X Z Mini-lens Boroscope Fish-eye lens • Cameras with radial lens distortion are often used in computer and robotic vision applications O O'

7. Problem statement (2/2) • SIFT is invariant to rotation and scale but it is not invariant to RD 421 correct matches 336 correct matches • Our Contribution: Modifications to the original SIFT for invariance to image RD • Assumptions: • RD can be fairly described by the division model (Fitzgibbon, CVPR 2001) • RD is roughly known ( e.g. line stretching ) (Barreto, CVIU 2006)

8. Tracking RD effects in SIFT • How does RD affect the SIFT algorithm? • Study using images with artificially added distortion • Isolate the RD effect in SIFT detection and matching • Reliable ground truth RD = 55% RD = 35% RD = 0% RD = 15% • Improvement to the SIFT algorithm to handle RD issues • Results on real imagery

9. How does RD affect keypoint detection? • Repeatability of keypoint detection decreases with increasing distortion • Small features (fine scale) tend to disappear during the blurring process • Coarse features tend to be detected at finer levels of scale • Flat regions (e.g. edges) start gain to strong gradient variations Filtering bounds Regular DoG pyramid ‘Distorted’ DoG pyramid

10. Proposed Solution: Adaptive smoothing • Radial distortion must be removed before the Gaussian smoothing • We can avoid the reconstruction artifacts by using an adaptive filter Rectification (~ 1.5 seconds in Matlab)

11. Standard vs Adaptive Gaussian smoothing • Inherent properties of the standard Gaussian filter • Decouple the convolution mask in X and Y directions • Simplification of the adaptive filter • Advantages of the Simplified Adaptive Filter • Shape only depends on the radius of the convolution window • Isotropic filter that can be decoupled for each image radius

12. Detection repeatability (synthetic adding of RD) Repeatability Computational time Error in calibration • Better Repeatability results for keypoint detection • Lower computational time than image rectification • More robust to calibration errors

13. How does RD affect matching? • RD modifies the local structures in the image and by consequence the gradients are affected • Changes in local gradients of the image deteriorates SIFT descriptor performance • Proposed solution: Compute gradients in the distorted image and perform implicit correction using the jacobian matrix of the distortion function

14. Matching evaluation (synthetic adding of RD) Implicit gradient correction SIFT in Rectified Images SIFT in RD Images • Compressive effect adds new contributions to the descriptor that do not occur in undistorted views • The matching performance can be improved by correcting image gradients before building the descriptor • Implicit gradient correction outperforms explicit image rectification for distortion amounts up to 25%.

15. Experiments with Real Images • Planar scenes for repeatability test and scenes with depth variation for motion estimation • Firewire camera with regular lens (~ 10% ) of distortion • Dragonfly camera with mini lens (~ 25% ) of distortion • Firewire camera with fish-eye lens (~ 45% ) of distortion

16. Planar images SIFT in RD Images SIFT in Rectified Images Our method 10% distortion 176 matches 294 matches 364 matches 25% distortion 201 matches 310 matches 401matches 45% distortion 326 matches 112 matches 253 matches

17. Motion recovery / Sparse 3D reconstruction • Scenes with depth variation where wrong matches are discarded using epipolar geometric constraints Main Scene 10% distortion 25% distortion 45% distortion

18. Experimental evaluation Main Scene Number of Inliers Inliers Distribution RMS rotation angle 3D reconstruction

19. Conclusions / Future Work • We proposed a set of modifications to the original SIFT algorithm (RD-SIFT) for achieving invariance to radial distortion. The additional computational overhead is minimum • RD – SIFT proved to be superior to explicit image correction • Better repeatability and retrieval performance • Less computational overhead • Increased robustness to calibration errors • Future Work • Extend the approach to other keypoint detectors (e.g. MSER and SURF) • Real-time implementation using GPGPU (to make available to the community) • Get rid of calibration dependence

21. THANKS FOR COMING 

22. Detection after explicit RD correction • Correct the radial distortion via image rectification Rectification (~ 1.5 seconds in Matlab) • Drawbacks of this approach • Signal reconstruction introduces artifacts affecting SIFT performance Image 1.5x Image (Bilinear) 1.5x Image (Bicubic)