Image Processing IB Paper 8 – Part A

1. Image ProcessingIB Paper 8 – Part A Ognjen Arandjelovićhttp://mi.eng.cam.ac.uk/~oa214/

2. – Image Essentials, Sampling –

3. Image Processing Paradigm We can think of image processing as a black box that takes an input image and produces an output image. Feature extraction, for example, does not fall under the umbrella of image processing.

4. Image Processing Image processing deals with computer-based manipulation of digital images. These include: • geometric operations (various morphs) • brightness and contrast correction • colour enhancement, • segmentation, • denoising.

5. Image Processing Applications Fingerprint enhancement for analysis and recognition: Original fingerprint image After denoising and contrast enhancement After binarization and morphological processing

6. Image Processing Applications Enhancement of CCTV footage: Still from original CCTV footage After brightness and contrast enhancement After denoising

7. Image Processing Applications Achieving illumination and pose invariance for automatic face recognition: After face detection and cropping After adaptive high-pass filtering Still from original authentication video

8. Discretization An inherent cause of information loss when representing physical signals on a computer: Face geometry • Two types: • Level / numerical It is impossible to represent infinite numbers (e.g. Pi) using a finite amount of storage space • Temporal or spatial It is impossible to sample a signal at infinite frequency or at infinite number of locations

9. Discretization Under a Magnifying Glass An example on an image of a face: Pixels = spatial discretization Values = numerical discretization

10. Numerical Quantization The luminance of each pixel can be represented only with finite precision. This loss of information is called quantization noise. Face geometry We can compute the average quantization noise energy per pixel: Minimal luminance difference that can be represented

11. Numerical Quantization – An Example

12. Spatial Discretization – An Example

13. 1D Sampling In 1D, we model sampling of a continuous function as multiplication by a train of delta functions:

14. 1D Sampling – Frequency Domain Multiplication in the spatial domain goes into convolution in the frequency domain: A train of delta functions Fourier spectrum of the original 1D signal Fourier spectrum of the sampled 1D signal

15. 2D Sampling Much like in 1D, in 2D we model sampling of a continuous function as multiplication by a comb of delta functions: A comb of delta functions Image as a 2D function / surface

16. 2D Sampling – Frequency Domain Much like in 1D, in 2D we model sampling of a continuous function as multiplication by a comb of delta functions: A comb of delta functions Fourier spectrum of the sampled 2D signal Fourier spectrum of a 2D signal

17. Geometric Transformations Geometric transformations warp images without (in principle) changing the value of the corresponding 2D signal. Examples include: • Scaling • Cropping • Rotation • Arbitrary morphs

18. Geometric Transformations – Apps Image registration for face recognition (correction of mild pose variations):

19. Geometric Transformations – Apps Image warping for mosaicing of photographs: How to perform seamless concatenation of images?

20. Geometric Transformations – Main Idea All geometric transformations of images at their core concern the problem of resampling. As the original (not sampled) 2D signal cannot be accessed, we need to first reconstruct it from the set of samples we have i.e. pixels.

21. Signal Reconstruction Remember that in the case of a band-limited signal, we can obtain a perfect reconstruction using the ideal low-pass filter: Frequency spectrum of the sampled signal Ideal low-pass filter Perfectly reconstructed signal What is the problem with this?

22. The Ideal Low-Pass Filter Recall the Fourier transform of the ideal low-pass filter (i.e. the pulse function): Fourier transform Ideal low-pass filter The sinc function N.B.

23. The Ideal Low-Pass Filter in 2D Recall the Fourier transform of the ideal low-pass filter (i.e. the pulse function): Fourier transform Ideal 2D low-pass filter The 2D sinc function

24. Reconstruction in Spatial Domain Using duality, we can see that the original signal can be reconstructed by convolving its sampled version with a sinc function:

25. The Ideal LPF in the Spatial Domain The reason why we cannot use the ideal LPF in image processing is of computational nature: This value, for example depends on all samples! The idea is to use something more manageable, by making the new sample dependent only on its neighbourhood.

26. Linear Interpolation In linear interpolation, the signal between two samples is approximated by a straight line: Clearly not perfect We can thus say that the constraint is that the line should pass through the two sample points – there are 2 DOFs (a point on the line and the slope).

27. Linear Interpolation We can then write an expression for the value of the function at an arbitrary location between two original samples: Value at the new sample Value of “pixel a” Value of “pixel b”

28. Bi-Linear Interpolation Bi-linear interpolation is simply a 2D extension of linear interpolation: Original sampling locations What is the value at this location?

29. Bi-Linear Interpolation One way of thinking about bi-linear interpolation is as fitting of a plane between neighbouring non-collinear sampling points: Original sampling locations What is the value at this point?

30. Bi-Linear Interpolation Alternatively, you can think of bi-linear interpolation as weighted average of the new sample’s neighbours: Original sampling locations What is the value at this point?

31. Cubic Interpolation We have seen that linear interpolation is far from perfect. Cubic interpolation does better: Much better!

32. Cubic Interpolation Constraints In cubic interpolation we have the following constraints for segment between x2 and x3, given 4 consecutive samples x1, x2, x3, x4: • C0 Continuity:Value at x2 and x3 should be exact(as with linear interpolation) • C1 Continuity:Gradient at x2 should be (x3-x1) / 2and at x3 should be (x4-x2) / 2

33. The Gradient Constraint The see why the gradient at say x2 should be(x3-x1) / 2, consider 1st order Taylor series expansions around x2: Now subtract (1) from (2):

34. Cubic Interpolation in 1D The expression for cubic interpolation is thus more complicated:

35. Bi-Cubic Interpolation The expression for bi-cubic interpolation gets very messy, so we do not give it here. The principle, however, is the same – you can think of it as cubic interpolation in the x direction, followed by cubic interpolation in the y direction.

36. Bi-Linear vs. Bi-Cubic Interpolation A summary of key differences: • Cubic produces less smoothing of the signal than linear • Cubic is about 4 times more computationally demanding and is hence seldom used for resampling of large images

37. Impulse Response of Linear Interp. We can analyse the performance of the two interpolators by looking at their impulse responses: An impulse Impulse response of the linear interpolation process

38. Impulse Response of Linear Interp. By taking the Fourier transform of impulse responses of the two interpolators, we can see how they affect different frequencies: Frequency response of cubic interpolation is much flatter at higher frequencies – closer to the ideal LPF.

39. Image Rescaling Now that we know how to reconstruct the original signal from a discrete set of samples, we can easily manipulate images in various ways. New sampling locations(use interpolation) Original sampling locations Image resizing as resampling – the original 5 х5 image is resampled to 6 х 6.

40. Image Rescaling Example To examine the effects of resampling, let us look at a small rectangular patch extracted from CCTV footage: Magnified patch – 50 х90 pixels

41. Image Rescaling Example Let us now compare the results of bi-linear and bi-cubic interpolation-based magnification for a factor of 2: Bi-cubic Bi-linear Note that bi-linear magnification produces a much smoother result.

42. Rescaling Comparison – An Example Quantitative insight can be gained by looking at the image difference of two results: As expected from theory, the difference is mainly in the high-frequency content.

43. Downsampling Caveats Remember that the original signal can be reconstructed from a set of samples using a LPF only if it is band-limited: A train of delta functions Fourier spectrum of the original 1D signal Fourier spectrum of the sampled 1D signal No overlap of spectrum ‘replicas’ (this overlap is called aliasing) It is crucial to ensure that the signal is band-limited before downsampling.

44. Downsampling Caveats – Aliasing Let’s take a look at what happens with downsampling without considering the aliasing problem: Downsampling Original image (184 х184 pixels) Resulting image (40 х 40 pixels) Note strange downsampling artefacts.

45. Downsampling Caveats – Aliasing To ensure that high frequencies are suppressed, the original image should first be LP filtered: Gaussian Smoothing Original image (184 х184 pixels) Low-pass smoothed image

46. Downsampling Caveats – Aliasing Now we can downsample as before, without worrying about aliasing: Downsampling Low-pass smoothed image Resulting image (40 х 40 pixels)

47. Downsampling Caveats – Aliasing Compare the results (or try switching anti-aliasing off in your Acrobat Reader!): Without LP filtering With LP filtering

48. Image Rotation Now that we know how to reconstruct the original signal from a discrete set of samples, we can easily manipulate images in various ways. New sampling locations(use interpolation) Original sampling locations Image rotation as resampling – the original 5 х5 image is rotated ~30 degrees clockwise.

49. Image Rotation – New Samples The main difference to rescaling is that the locations of new samples are now slightly more difficult to compute. Recall that these are related to the original locations by the rotation matrix: Where θ is the angle of rotation.

50. Image Rotation – Image Size Note that the size of the resulting, rotated image is in general different than the size of the original image. If the input image is of size H х W pixels, then the output image is of size H' х W' pixels, where: The ceiling operator