Recognising Panoramas

1. Recognising Panoramas M. Brown and D. Lowe, University of British Columbia

2. Introduction • Are you getting the whole picture? • Compact Camera FOV = 50 x 35°

3. Introduction • Are you getting the whole picture? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135°

4. Introduction • Are you getting the whole picture? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° • Panoramic Mosaic = 360 x 180°

5. Why “Recognising Panoramas”?

6. Why “Recognising Panoramas”? • 1D Rotations (q) • Ordering  matching images

7. Why “Recognising Panoramas”? • 1D Rotations (q) • Ordering  matching images

8. Why “Recognising Panoramas”? • 1D Rotations (q) • Ordering  matching images

9. 2D Rotations (q, f) • Ordering  matching images Why “Recognising Panoramas”? • 1D Rotations (q) • Ordering  matching images

10. 2D Rotations (q, f) • Ordering  matching images Why “Recognising Panoramas”? • 1D Rotations (q) • Ordering  matching images

11. 2D Rotations (q, f) • Ordering  matching images Why “Recognising Panoramas”? • 1D Rotations (q) • Ordering  matching images

12. Why “Recognising Panoramas”?

13. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

14. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

15. Overview • Feature Matching • SIFT Features • Nearest Neighbour Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

16. Overview • Feature Matching • SIFT Features • Nearest Neighbour Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

17. Invariant Features • Schmid & Mohr 1997, Lowe 1999, Baumberg 2000, Tuytelaars & Van Gool 2000, Mikolajczyk & Schmid 2001, Brown & Lowe 2002, Matas et. al. 2002, Schaffalitzky & Zisserman 2002

18. SIFT Features • Invariant Features • Establish invariant frame • Maxima/minima of scale-space DOG  x, y, s • Maximum of distribution of local gradients q • Form descriptor vector • Histogram of smoothed local gradients • 128 dimensions • SIFT features are… • Geometrically invariant to similarity transforms, • some robustness to affine change • Photometrically invariant to affine changes in intensity

19. Overview • Feature Matching • SIFT Features • Nearest Neighbour Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

20. Nearest Neighbour Matching • Find k-NN for each feature • k  number of overlapping images (we use k = 4) • Use k-d tree • k-d tree recursively bi-partitions data at mean in the dimension of maximum variance • Approximate nearest neighbours found in O(nlogn)

21. Overview • Feature Matching • SIFT Features • Nearest Neighbour Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

22. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

23. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

24. Overview • Feature Matching • Image Matching • RANSAC for Homography • Probabilistic model for verification • Bundle Adjustment • Multi-band Blending • Results • Conclusions

25. Overview • Feature Matching • Image Matching • RANSAC for Homography • Probabilistic model for verification • Bundle Adjustment • Multi-band Blending • Results • Conclusions

26. RANSAC for Homography

27. RANSAC for Homography

28. RANSAC for Homography

29. Overview • Feature Matching • Image Matching • RANSAC for Homography • Probabilistic model for verification • Bundle Adjustment • Multi-band Blending • Results • Conclusions

30. Probabilistic model for verification

31. Probabilistic model for verification • Compare probability that this set of RANSAC inliers/outliers was generated by a correct/false image match • ni = #inliers, nf = #features • p1 = p(inlier | match), p0 = p(inlier | ~match) • pmin = acceptance probability • Choosing values for p1, p0 and pmin

32. Finding the panoramas

33. Finding the panoramas

34. Finding the panoramas

35. Finding the panoramas

36. Overview • Feature Matching • Image Matching • RANSAC for Homography • Probabilistic model for verification • Bundle Adjustment • Multi-band Blending • Results • Conclusions

37. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

38. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

39. Overview • Feature Matching • Image Matching • Bundle Adjustment • Error function • Multi-band Blending • Results • Conclusions

40. Overview • Feature Matching • Image Matching • Bundle Adjustment • Error function • Multi-band Blending • Results • Conclusions

41. Error function • Sum of squared projection errors • n = #images • I(i) = set of image matches to image i • F(i, j) = set of feature matches between images i,j • rijk = residual of kth feature match between images i,j • Robust error function

42. Homography for Rotation • Parameterise each camera by rotation and focal length • This gives pairwise homographies

43. Bundle Adjustment • New images initialised with rotation, focal length of best matching image

44. Bundle Adjustment • New images initialised with rotation, focal length of best matching image

45. Overview • Feature Matching • Image Matching • Bundle Adjustment • Error function • Multi-band Blending • Results • Conclusions

46. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

47. Overview • Feature Matching • Image Matching • Bundle Adjustment • Multi-band Blending • Results • Conclusions

48. Multi-band Blending • Burt & Adelson 1983 • Blend frequency bands over range l

49. 2-band Blending Low frequency (l > 2 pixels) High frequency (l < 2 pixels)

50. Linear Blending