CS 585 Computational Photography

1. CS 585 Computational Photography Nathan Jacobs

2. Today’s Agenda • parametric transformations • image warping • seam carving

3. Image by Andrew Davidhazy (http://people.rit.edu/andpph/)

4. http://www.sportingworld.co.uk

8. topics in context • matlab programming • image formation and camera fundamentals • pinhole and thin-lens model • intensity and color • shutter, aperture, focal length • depth of field • radiometric and geometric calibration • auto-focus, auto-exposure, auto-color balance • image processing fundamentals • point processing • spatial filtering • morphological operations • frequency space representations • image warping • advanced topics • retargeting • matting, morphing, compositing • warping and alignment • panorama construction • extending dynamic range • manipulating depth of field • manipulating lighting (e.g. structured lighting and flash/no-flash) • light fields • coded apertures • compressive sensing • something that excites you!

9. f f f T T x x x f x Image Transformations • image filtering: change range of image • g(x) = T(f(x)) image warping: change domain of image g(x) = f(T(x))

10. T T Image Transformations • image filtering: change range of image • g(x) = T(f(x)) f g image warping: change domain of image g(x) = f(T(x)) f g

11. Parametric (global) warping • Examples of parametric warps: aspect rotation translation perspective cylindrical affine

12. Digression: Image Warping in Biology • D'Arcy Thompson • http://www-groups.dcs.st-and.ac.uk/~history/Miscellaneous/darcy.html • http://en.wikipedia.org/wiki/D'Arcy_Thompson • Importance of shape and structure in evolution Slide by Durand and Freeman

13. T Parametric (global) warping Transformation T is a coordinate-changing machine: p’ = T(p) What does it mean that T is global? • Is the same for any point p • can be described by just a few numbers (parameters) Let’s represent T as a matrix: p’ = Mp p = (x,y) p’ = (x’,y’)

14. Scaling • Scaling a coordinate means multiplying each of its components by a scalar • Uniform scaling means this scalar is the same for all components: 2

15. X  2,Y  0.5 Scaling • Non-uniform scaling: different scalars per component:

16. Scaling • Scaling operation: • Or, in matrix form: scaling matrix S What’s inverse of S?

17. (x’, y’) (x, y)  2-D Rotation x’ = x cos() - y sin() y’ = x sin() + y cos()

18. 2-D Rotation • This is easy to capture in matrix form: • Even though sin(q) and cos(q) are nonlinear functions of q, • x’ is a linear combination of x and y • y’ is a linear combination of x and y • What is the inverse transformation? • Rotation by –q • For rotation matrices R

19. 2x2 Matrices • What types of transformations can be represented with a 2x2 matrix? 2D Identity? 2D Scale around (0,0)?

20. 2x2 Matrices • What types of transformations can be represented with a 2x2 matrix? 2D Rotate around (0,0)? 2D Shear?

21. 2x2 Matrices • What types of transformations can be represented with a 2x2 matrix? 2D Mirror about Y axis? 2D Mirror over (0,0)?

22. 2x2 Matrices • What types of transformations can be represented with a 2x2 matrix? 2D Translation? NO! Only linear 2D transformations can be represented with a 2x2 matrix

23. All 2D Linear Transformations • Linear transformations are combinations of … • Scale, • Rotation, • Shear, and • Mirror • Properties of linear transformations: • Origin maps to origin • Lines map to lines • Parallel lines remain parallel • Ratios are preserved • Closed under composition

24. Homogeneous Coordinates • Q: How can we represent translation as a 3x3 matrix?

25. Homogeneous Coordinates • Homogeneous coordinates • represent coordinates in 2 dimensions with a 3-vector homogenouscoords

26. y 2 (2,1,1) or (4,2,2) or (6,3,3) 1 x 2 1 Homogeneous Coordinates • Add a 3rd coordinate to every 2D point • (x, y, w) represents a point at location (x/w, y/w) • (x, y, 0) represents a point at infinity • (0, 0, 0) is not allowed Convenient coordinate system to represent many useful transformations

27. Homogeneous Coordinates • Q: How can we represent translation as a 3x3 matrix? • A: Using the rightmost column:

28. Translation • Example of translation Homogeneous Coordinates tx = 2ty= 1

29. Basic 2D Transformations • Basic 2D transformations as 3x3 matrices Translate Scale Rotate Shear

30. Matrix Composition • Transformations can be combined by matrix multiplication p’ = T(tx,ty) R(Q) S(sx,sy) p

31. Affine Transformations • Affine transformations are combinations of … • Linear transformations, and • Translations • Properties of affine transformations: • Origin does not necessarily map to origin • Lines map to lines • Parallel lines remain parallel • Ratios are preserved • Closed under composition • Models change of basis • Will the last coordinate w always be 1?

32. Projective Transformations • Projective transformations … • Affine transformations, and • Projective warps • Properties of projective transformations: • Origin does not necessarily map to origin • Lines map to lines • Parallel lines do not necessarily remain parallel • Ratios are not preserved • Closed under composition • Models change of basis

33. 2D global image transformations • These transformations are a nested set of groups • Closed under composition and inverse is a member

34. Image Transformations in Matlab • maketform: make a transform from parameters • cpselect: GUI for defining corresponding points • cpcorr: refine selected points using normxcorr2 • cp2tform: make a transform from corresponding points • imtransform : do the transformation

35. Recovering Transformations • What if we know f and g and want to recover the transform T? • e.g. better align images from Project 1 • willing to let user provide correspondences • How many do we need? ? T(x,y) y y’ x x’ f(x,y) g(x’,y’)

36. Translation: # correspondences? • How many correspondences needed for translation? • How many Degrees of Freedom? ? T(x,y) y y’ x x’

37. Euclidian: # correspondences? • How many correspondences needed for translation and rotation? • How many DOF? ? T(x,y) y y’ x x’

38. Affine: # correspondences? • How many correspondences needed for affine? • How many DOF? ? T(x,y) y y’ x x’

39. Projective: # correspondences? • How many correspondences needed for projective? • How many DOF? ? T(x,y) y y’ x x’

40. Image warping • Given a coordinate transform (x’,y’)=T(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’)=f(T(x,y))? T(x,y) y y’ x x’ f(x,y) g(x’,y’)

41. Forward warping • Send each pixel f(x,y) to its corresponding location (x’,y’)=T(x,y) in the second image T(x,y) y y’ x x’ f(x,y) g(x’,y’) Q: what if pixel lands “between” two pixels?

42. Forward warping T(x,y) y y’ x x’ f(x,y) g(x’,y’) • Send each pixel f(x,y) to its corresponding location (x’,y’)=T(x,y) in the second image Q: what if pixel lands “between” two pixels? • A: distribute color among neighboring pixels (x’,y’) • Known as “splatting”

43. T-1(x,y) Inverse warping Get each pixel g(x’,y’) from its corresponding location (x,y)=T-1(x’,y’) in the first image y y’ x x x’ f(x,y) g(x’,y’) Q: what if pixel comes from “between” two pixels?

44. T-1(x,y) Inverse warping y y’ x x x’ f(x,y) g(x’,y’) Get each pixel g(x’,y’) from its corresponding location (x,y)=T-1(x’,y’) in the first image Q: what if pixel comes from “between” two pixels? • A: Interpolate color value from neighbors • nearest neighbor, bilinear, Gaussian, bicubic • Check out interp2 in Matlab

45. Forward vs. inverse warping • Q: which is better? • A: usually inverse—eliminates holes • however, it requires an invertible warp function—not always possible...

46. Limitations of 2x2 of 3x3 warps • people, cloth, non-rigid stuff… • can be can be more general • has its limits Zhang, Collins, Liu

47. Project 2: Image Resizing by Seam Carving • http://www.youtube.com/watch?v=6NcIJXTlugc http://cs.uky.edu/~jacobs/classes/2011_photo/projects/02_retargeting/index.html

48. Summary • Global parametric coordinate transformations • 2x2 matrices • 3x3 matrices • Image warping • forward vs. backward/inverse • a non-parametric warp: seam carving

49. next week • compositing and matting • gradient domain editing