Feature Based Image Mosaicing

1. Feature Based Image Mosaicing Satya Prakash Mallick

2. Introduction Mosaicing methods can be classified broadly into • Direct Method: • Uses information from all pixels. • Iteratively updates an estimate of homography so that a particular cost function is minimized. • Sometimes Phase-Correlation is used to estimate the a few parameters of the homography. • Feature Based Method: • A few corresponding points are selected on the two images and homography is estimated using these reliable points only.

3. My Choice? • Feature Based Method: Because • They are in general more accurate. • Can Handle large disparities. • Convergence: Direct methods, may not converge to the optimal solution is the presence of local minima. • For reliable performance direct methods rely on feature based initialization.

4. Problems in Image Mosaicing • Global Alignment:essentially means recovery of underlying homography. • Local Alignment:Correct the local mismatches left after global alignment. • Image Blending:a decision has to be made as to what color the overlapping region should take. • Image Warping:After calculation of homography, a decision has to be made as to how to warp one image w.r.t the other.

5. Problems in Image Mosaicing • Automatic selection of images to blend. • Auto-exposure compensation • Camera error compensation. I am looking at the first four problems.

6. The Algorithm ( Overview ) • Detect corners in both the images • Solve for correspondence. • Use RANSAC to estimate homography. • Refine the estimate of homography using a non-linear method. • Make a decision on how to blend the images. • Warp the one image with respect to the other taking the blending decision into consideration

7. Solving for correspondences • Corners were detected in both the images using Harris corner detector. • Correspondences are solved using a version of Zhang’s relaxation algorithm: • Matching Through Correlation: • Disambiguating Matches Through Relaxation:

8. Solving for correspondences • Matching through correlation:

9. Solving for correspondences • Disambiguating matches through relaxation: • Show correspondence results

10. Estimation of homography • RANSAC was implemented for estimating the homography relating the two scenes. Results show a comparison between using RANSAC and Least squares. • The error function being minimized is:

11. Image Blending • Weighted Image Blending:

12. Image Blending • Lightest Path Cut: The idea is to take the difference of the images in the overlapping region and cut the region along that curve of minimum intensity. The implementation was done using dynamic programming. • Note: Optimal implementation of the algorithm is non-trivial and can be a good problem to look into.

13. Results of Blending Try to find the curve along which the image overlap was cut! ALL THE BEST!

14. Homography Refinement • In the Weighted image blending example we saw, there was some blurring at the edges of the overlap. • The obvious question is: Are we sure that the homography estimate is good enough or can it be further refined?

15. Homography Refinement • The Homography was refined using Newton’s non-linear optimization technique. It’s “safe” to use Newton’s method because we are already close to the solution. The vector X shown is made of inliers got using RANSAC

16. Homography Refinement • Results: The blurring near the edge of overlapping region is almost gone.

17. Limitations and Mistakes • My biggest mistake: I assumed that, to make a mosaic with many images, it is sufficient to keep on adding images to the main mosaic. The mosaic doesn’t do well, if we try to blend more than four images. So there should have been a global refinement on different homographies

18. Conclusions • Lessons to be learnt: • Non-Linear optimization of homography is not an overkill. After RANSAC, it is usually very fast ( Most of the time, my results converged after 2-3 iterations only) • It’s a good idea to blend images along the lightest curve along the difference image. However, the image quilting algorithm cannot be directly used in that. • To blend a large number of images, there is “something” more to be done than finding pair-wise homography of images.

19. Conclusion • Image warping should always be done backward with bilinear interpolation.

20. References • [Hartley] Hartley, R. & Zisserman, A. (2000) Multiple View Geometry{Cambridge University Press, UK. • [Shum] Shum, H. & Szeliski, R. (1998) Construction and refinement of panoramic mosaics with global and local alignment. IEEE Int'l Conf. Computer Vision, pp. 953-958. • [Faugeras] Zoghlami, I. & Faugeras,O. & Deriche,R. (1997) Using geometric corners to build a 2d mosaic from as set of images.Computer Vision and Pattern Recognition, pp 421-425. • [Zhang] Zhang, Z.& Deriche, R. & Faugeras, O & Luong, Q. A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry (1995) Artificial Intelligence Journal, 78:87-119, October 1995 • [Harris] Harris, C. & Stephens, M. A combined corner and edge detector.(1998) Proc. of 4th Alvey Vision Conf.,147-151. • [Szeliski] Szeliski, R. Image Mosaicing for Tele-Reality Applications.(1994). Digital Equipment Corporation, Cambridge, USA. • [Davis] Davis, J. Mosaics of scenes with moving objects.(1998).Computer Vision and Pattern Recognition • [Capel] Capel,D & Zisserman,A. Automated mosaicing with super-resolutionzoom.(1998).Computer Vision and Pattern Recognition

21. THANKS !