Image Alignment

1. Image Alignment Samuel Cheng Slide credit: Noah Snavely, James Thompkin

2. Remark on HW2 • Don’t use function blindly. Sometimes it is better to do things from scratch. Read “help” document carefully • Imfuse • Imfilter • Check result. Be alert if result does not look right • Most of you probably will get better hybrid image with a stronger low-pass filter • Take advantage of piazza, please sign up and post any questions

3. Image alignment Why don’t these image line up exactly?

4. What is the geometric relationship between these two images? ? Answer: Similarity transformation (translation, rotation, uniform scale)

5. What is the geometric relationship between these two images? ?

6. What is the geometric relationship between these two images? Very important for creating mosaics!

7. f f g h h x x x g x Image Warping • image filtering: change range of image • g(x) = h(f(x)) • image warping: change domain of image • g(x) = f(h(x)) Image Stitching

8. h h Image Warping • image filtering: change range of image • g(x) = h(f(x)) • image warping: change domain of image • g(x) = f(h(x)) f g f g Image Stitching

9. Parametric (global) warping • Examples of parametric warps: aspect rotation translation Image Stitching

10. T Parametric (global) warping • Transformation T is a coordinate-changing machine: p’ = T(p) • What does it mean that T is global? • Is the same for any point p • can be described by just a few numbers (parameters) • Let’s consider linear xforms (can be represented by a 2D matrix): p = (x,y) p’ = (x’,y’)

11. Common linear transformations • Uniform scaling by s: (0,0) (0,0) What is the inverse?

12. Common linear transformations • Rotation by angle θ(about the origin) θ (0,0) (0,0) What is the inverse? For rotations:

13. 2x2 Matrices • What types of transformations can be represented with a 2x2 matrix? 2D mirror about Y axis? 2D mirror across line y = x?

14. 2x2 Matrices • What types of transformations can be represented with a 2x2 matrix? 2D Translation? NO! Translation is not a linear operation on 2D coordinates

15. All 2D Linear Transformations • Linear transformations are combinations of … • Scale, • Rotation, • Shear, and • Mirror • Properties of linear transformations: • Origin maps to origin • Lines map to lines • Parallel lines remain parallel • Ratios are preserved • Closed under composition

16. Homogeneous coordinates (x, y, w) w Trick: add one more coordinate: homogeneous plane (x/w, y/w, 1) w = 1 x homogeneous image coordinates y Converting from homogeneous coordinates

17. Translation • Solution: homogeneous coordinates to the rescue

18. Affine transformations any transformation with last row [ 0 0 1 ] we call an affine transformation

19. Basic affine transformations Translate Scale 2D in-plane rotation Shear

20. Affine Transformations • Affine transformations are combinations of … • Linear transformations, and • Translations • Properties of affine transformations: • Origin does not necessarily map to origin • Lines map to lines • Parallel lines remain parallel • Ratios are preserved • Closed under composition

21. Is this an affine transformation?

22. Where do we go from here? what happens when we mess with this row? affine transformation

23. Projective Transformations aka Homographies aka Planar Perspective Maps Called a homography (or planar perspective map)

24. Homographies • Example on board

25. black area where no pixel maps to Image warping with homographies image plane in front image plane below

26. Homographies

27. Projective Transformations • Projective transformations … • Affine transformations, and • Projective warps • Properties of projective transformations: • Origin does not necessarily map to origin • Lines map to lines • Parallel lines do not necessarily remain parallel • Ratios are not preserved • Closed under composition

28. 2D image transformations • These transformations are a nested set of groups • Closed under composition and inverse is a member

29. Computing transformations • Given a set of matches between images A and B • How can we compute the transform T from A to B? • Find transform T that best “agrees” with the matches

30. Computing transformations ?

31. Simple case: translations How do we solve for ?

32. Simple case: translations Displacement of match i = Mean displacement =

33. Another view • System of linear equations • What are the knowns? Unknowns? • How many unknowns? How many equations (per match)?

34. Another view • Problem: more equations than unknowns • “Overdetermined” system of equations • We will find the least squares solution

35. Least squares formulation • For each point • we define the residuals as

36. Least squares formulation • Goal: minimize sum of squared residuals • “Least squares” solution • For translations, is equal to mean displacement

37. Least squares formulation • Can also write as a matrix equation 2n x 2 2x 1 2n x 1

38. Least squares • Find t that minimizes

39. Least squares • Find t that minimizes

40. Least squares • Find t that minimizes

41. Affine transformations • How many unknowns? • How many equations per match? • How many matches do we need?

42. Affine transformations • Residuals: • Cost function:

43. Affine transformations • Matrix form 6x 1 2n x 1 2n x 6

44. Homographies p’ p • To unwarp (rectify) an image • solve for homography H given p and p’ • solve equations of the form: wp’ = Hp • linear in unknowns: w and coefficients of H • H is defined up to an arbitrary scale factor • how many points are necessary to solve for H?

45. Solving for homographies Not linear!

46. Solving for homographies

47. Solving for homographies 2n × 9 2n 9 Defines a least squares problem:

48. Solving for homographies

49. Questions?

50. Image Alignment Algorithm Given images A and B • Compute image features for A and B • Match features between A and B • Compute homography between A and B using least squares on set of matches What could go wrong?