Image Stitching

1. Image Stitching

2. Panoramic Image Mosaics Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html 2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html Image Stitching

3. Gigapixel panoramas & images Mapping / Tourism / WWT Medical Imaging Image Stitching

4. Image Mosaics • + + … + = Goal: Stitch together several images into a seamless composite Image Stitching

5. Today’s lecture • Image alignment and stitching • motion models • image warping • point-based alignment • complete mosaics (global alignment) • compositing and blending • ghost and parallax removal Image Stitching

6. Readings • Szeliski, CVAA: • Chapter 3.6: Image warping • Chapter 5.1: Feature-based alignment (in preparation) • Chapter 9.1: Motion models • Chapter 9.2: Global alignment • Chapter 9.3: Compositing • Recognizing Panoramas, Brown & Lowe, ICCV’2003 • Szeliski & Shum, SIGGRAPH'97 Image Stitching

7. Motion models

8. Motion models • What happens when we take two images with a camera and try to align them? • translation? • rotation? • scale? • affine? • perspective? • VideoMosaic Image Stitching

9. Image Warping

10. f f f h h x x x f x Image Warping • image filtering: change range of image • g(x) = h(f(x)) • image warping: change domain of image • g(x) = f(h(x)) Image Stitching

11. h h Image Warping • image filtering: change range of image • g(x) = h(f(x)) • image warping: change domain of image • g(x) = f(h(x)) f g f g Image Stitching

12. Parametric (global) warping • Examples of parametric warps: aspect rotation translation perspective cylindrical affine Image Stitching

13. 2D coordinate transformations Image Stitching

14. 2D coordinate transformations • Translation • Rotation + translation • Similarity • Affine Image Stitching

15. p’ H p Homography • A: Projective – mapping between any two PPs with the same center of projection • rectangle should map to arbitrary quadrilateral • parallel lines aren’t • but must preserve straight lines • same as: project, rotate, reproject • called Homography PP2 PP1 • To apply a homography H • Compute p’ = Hp (regular matrix multiply) • Convert p’ from homogeneous to image coordinates

16. 2D coordinate transformations • Projective transformation or Homography • Normalized homogeneous Homogeneous, only defined up to a scale Image Stitching

17. 2D coordinate transformations Image Stitching

18. Image Warping • Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’)=f(h(x))? h(x) x x’ f(x) g(x’) Image Stitching

19. Forward Warping • Send each pixel f(x) to its corresponding location x’=h(x) in g(x’) • What if pixel lands “between” two pixels? h(x) x x’ f(x) g(x’) Image Stitching

20. Forward Warping • Send each pixel f(x) to its corresponding location x’=h(x) in g(x’) • What if pixel lands “between” two pixels? h(x) x x’ f(x) g(x’) Image Stitching

21. Forward Warping • Non-integer x’ • Nearest integer coordinate • Aliasing • Jumping • Splatting • Distribute to 4 neighbors (weighted) and normalize later • Both aliasing and blur • Cracks and holes • Filling hole with nearby neighbors • Further aliasing and blurring

22. Inverse Warping • Get each pixel g(x’) from its corresponding location x’=h(x) in f(x) • What if pixel comes from “between” two pixels? h(x) x x’ f(x) g(x’) Image Stitching

23. Inverse Warping • Get each pixel g(x’) from its corresponding location x’=h(x) in f(x) • What if pixel comes from “between” two pixels? • Answer: resample color value from interpolated (prefiltered) source image x x’ f(x) g(x’) Image Stitching

24. Interpolation • Possible interpolation filters: • nearest neighbor • bilinear • bicubic (interpolating) • sinc / FIR • Needed to prevent “jaggies” and “texture crawl” Image Stitching

25. Interpolation Bicubic • Bilinear ɑ: the derivative at x=1 Image Stitching

26. Interpolation • Window sinc functions Image Stitching

27. Image Interpolation All perform high-accuracy low-pass filtering while preserve details and avoid aliasing Bilinear for speed Bicubic and windowed sinc for better visual quality Image Stitching

28. Decimation • Interpolation: increase the image resolution • Decimation (downsampling): reduce resolution Samples are convolved with a low-pass filter before downsampled to avoid aliasing Image Stitching

29. Decimation • Sample rate : same kernel as interpolation Commonly use r=2 Image Stitching

30. Decimation Frequency response Image Stitching

31. Decimation • Linear filter gives a relatively poor response • Binomial filter cuts off a lot of frequencies but is useful for computer vision analysis pyramid • Cubic filter: a=-1 filter has a shaper fall-off than the a=-0.5 filter • Cosine-windowed sinc function • QMF-9 filter for wavelet denoising and aliases a fair amount • 9/7 analysis filter form JPEG2000 Image Stitching

32. Prefiltering • Essential for downsampling (decimation) to prevent aliasing • MIP-mapping [Williams’83]: • build pyramid (but what decimation filter?): • block averaging • Burt & Adelson (5-tap binomial) • 7-tap wavelet-based filter (better) • trilinear interpolation • bilinear within each 2 adjacent levels • linear blend between levels (determined by pixel size) Image Stitching

33. Prefiltering • Essential for downsampling (decimation) to prevent aliasing • Other possibilities: • summed area tables • elliptically weighted Gaussians (EWA) [Heckbert’86] Image Stitching

34. Resmaple on interpolated signal • To avoid blurring or aliasing Image Stitching

35. Motion models (reprise)

36. Affine Perspective 3D rotation Translation 2 unknowns 6 unknowns 8 unknowns 3 unknowns Motion models Image Stitching

37. Planar perspective motion • Simplest case: • Overlapping photographic prints, scanned pieces • Translation and rotation are usually adequate Four unknowns Image Stitching

38. Planar perspective motion • Planar Homography Image Stitching

39. Planar Perspective Motion • Extract features from images • Match features using RANSAC and get inliers • Estimate a homography H given pairs of corresponding features • Least squares fitting Image Stitching

40. Solving for homography p’ = Hp Can set scale factor i=1. So, there are 8 unknowns. Set up a system of linear equations: Ah = b where vector of unknowns h = [a,b,c,d,e,f,g,h]T Need at least 8 eqs, but the more the better… Solve for h. If overconstrained, solve using least-squares Image Stitching

41. Image warping with homographies image plane in front image plane below Image Stitching

42. Plane perspective mosaics • 8-parameter generalization of affine motion • works for pure rotation or planar surfaces • Limitations: • local minima • slow convergence • difficult to control interactively Image Stitching

43. Rotational mosaics • Directly optimize rotation and focal length • Advantages: • ability to build full-view panoramas • easier to control interactively • more stable and accurate estimates GrandCanyon panorama Image Stitching

44. (Xc,Yc,Zc) uc f u 3D → 2D Perspective Projection Image Stitching

45. Rotational panoramas • Setting With known f0, f1, solve R10 Image Stitching

46. Rotational panoramas • More stable for large-scale image stitching algorithms than homography • How to solve the unknowns? • Update R10 with incremental rotation matrix R(w) to current estimate R10 Image Stitching

47. Rotations and quaternions • How do we represent rotation matrices? • Axis / angle (n,θ)R = I + sinθ [n] + (1- cosθ) [n]2(Rodriguez Formula), with[n] = cross product matrix (see paper) • Unit quaternions [Shoemake SIGG’85]q = (nsinθ/2, cosθ/2) = (w,s)quaternion multiplication (division is easy)q0 q1 = (s1w0+ s0w1, s0 s1-w0·w1) Image Stitching

48. Incremental rotation update • Small angle approximationΔR = I + sinθ [n] + (1- cosθ) [n]2 ≈ θ [n] = [ω]linear inω • Update original R matrixR ← RΔR Image Stitching

49. Rotational panoramas • How to solve the unknowns of R10? • Compute the incremental rotation vector w, Image Stitching

50. Rotational mosaic • Projection equations • Project from image to 3D ray • (x0,y0,z0) = (u0-uc,v0-vc,f) • Rotate the ray by camera motion • (x1,y1,z1) = R01 (x0,y0,z0) • Project back into new (source) image • (u1,v1) = (fx1/z1+uc,fy1/z1+vc) Image Stitching