CS 691 Computational Photography

1. CS 691 Computational Photography Instructor: Gianfranco Doretto Image Filtering

2. What is an image? • We can think of an image as a function, f, from R2 to R: • f( x, y) gives the intensity at position ( x, y) • Realistically, we expect the image only to be defined over a rectangle, with a finite range: • f: [a,b]x[c,d] [0,1] • A color image is just three functions pasted together. We can write this as a “vector-valued” function:

3. Images as functions

4. f f f h h x x x f x Image Processing • image filtering: change range of image • g(x) =h(f(x)) • image warping: change domain of image • g(x) = f(h(x))

5. h h Image Processing • image filtering: change range of image • g(x) = h(f(x)) • image warping: change domain of image • g(x) = f(h(x))

6. Point Processing • The simplest kind of range transformations are these independent of position x,y: • g = t(f) • This is called point processing. • Important: every pixel for himself – spatial information completely lost!

7. Negative

8. Contrast Stretching

9. Image Histograms Cumulative Histograms s = T(r)

10. Histogram Equalization

11. Image filtering • Image filtering: compute function of local neighborhood at each position • Really important! • Enhance images • Denoise, resize, increase contrast, etc. • Extract information from images • Texture, edges, distinctive points, etc. • Detect patterns • Template matching

12. 8 coefficient 0 original Pixel offset 1D Smoothing examples impulse 2.4 0.3 filtered

13. coefficient 0 Pixel offset 1D Smoothing examples 8 8 edge 4 4 0.3 filtered original

14. Example: Box filter 1 1 1 1 1 1 1 1 1

15. Image filtering 1 1 1 1 1 1 1 1 1

16. Image filtering 1 1 1 1 1 1 1 1 1

17. Image filtering 1 1 1 1 1 1 1 1 1

18. Image filtering 1 1 1 1 1 1 1 1 1

19. Image filtering 1 1 1 1 1 1 1 1 1

20. Image filtering 1 1 1 1 1 1 1 1 1 ?

21. Image filtering 1 1 1 1 1 1 1 1 1 ?

22. Image filtering 1 1 1 1 1 1 1 1 1

23. Box Filter 1 1 1 1 1 1 1 1 1 • What does it do? • Replaces each pixel with an average of its neighborhood • Achieve smoothing effect (remove sharp features)

24. Smoothing with box filter

25. Cross-correlation filtering • Let’s write the box filter down as an equation. Assume the averaging window is (2k+1)x(2k+1): • We can generalize this idea by allowing different weights for different neighboring pixels: • This is called a cross-correlation operation and written: • H is called the “filter,” “kernel,” or “mask.”

26. Convolution • Cross-correlation: • A convolution operation is a cross-correlation where the filter is flipped both horizontally and vertically before being applied to the image: • It is written: • Supposehis the mean kernel (or box filter). How does convolution differ from cross-correlation?

27. Convolution is nice! • Notation: • Convolution is a multiplication-like operation • commutative • associative • distributes over addition • scalars factor out • identity: unit impulse e = […, 0, 0, 1, 0, 0, …] • Conceptually no distinction between filter and signal • Usefulness of associativity • often apply several filters one after another: (((a * b1) * b2) * b3) • this is equivalent to applying one filter: a * (b1 * b2 * b3)

28. 0 0 0 0 1 0 0 0 0 Practice with linear filters ? Original

29. 0 0 0 0 1 0 0 0 0 Practice with linear filters Original Filtered (no change)

30. 0 0 0 0 0 1 0 0 0 Practice with linear filters ? Original

31. 0 0 0 0 0 1 0 0 0 Practice with linear filters Original Shifted left By 1 pixel

32. 0 1 0 1 1 0 1 0 1 2 1 0 1 0 1 0 1 0 Practice with linear filters - ? (Note that filter sums to 1) Original

33. 1 0 0 1 1 0 1 0 1 2 0 1 0 1 1 0 1 0 Practice with linear filters - Original • Sharpening filter • Accentuates differenceswith local average

34. 1D sharpening example 1.7 11.2 8 8 coefficient -0.25 -0.3 original Sharpened (differences are accentuated; constant areas are left untouched).

35. Sharpening

36. 1 0 -1 2 0 -2 1 0 -1 Other filters Sobel Vertical Edge (absolute value)

37. Q? 1 2 1 0 0 0 -1 -2 -1 Other filters Sobel Horizontal Edge (absolute value)

38. Important filter: Gaussian • Weight contributions of neighboring pixels by nearness 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x5

39. Smoothing with Gaussian filter

40. Smoothing with box filter

41. Gaussian filters • Remove “high-frequency” components from the image (low-pass filter) • Images become more smooth • Convolution with self is another Gaussian • So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have • Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2 • Separable kernel • Factors into product of two 1D Gaussians

42. Separability of the Gaussian filter • The 2D Gaussian can be expressed as the product of two functions, one a function of x and the other a function of y • In this case, the two functions are the (identical) 1D Gaussian

43. * = = * Separability example 2D convolution(center location only) The filter factorsinto a product of 1Dfilters: Perform convolutionalong rows: Followed by convolutionalong the remaining column:

44. Separability • Why is separability useful in practice?

45. Some practical matters

46. Practical matters How big should the filter be? • Values at edges should be near zero • Rule of thumb for Gaussian: set filter half-width to about 3 σ

47. Practical matters • What is the size of the output? • MATLAB: filter2(h, f, shape) • shape = ‘full’: output size is sum of sizes of f andh • shape = ‘same’: output size is same as f • shape = ‘valid’: output size is difference of sizes of f andh full same valid h h h h f f f h h h h h h h h

48. Practical matters • What about near the edge? • the filter window falls off the edge of the image • need to extrapolate • methods: • clip filter (black) • wrap around • copy edge • reflect across edge

49. Practical matters • methods (MATLAB): • clip filter (black): imfilter(f,h, 0) • wrap around: imfilter(f,h, ‘circular’) • copy edge: imfilter(f,h, ‘replicate’) • reflect across edge: imfilter(f,h, ‘symmetric’)

50. Questions?