ISTANBUL TECHNICAL UNIVERSITY FACULTY OF AERONAUTICS & ASTRONAUTICS DEPARTMENT OF METEOROLOGICAL ENGINEERING

1. ISTANBUL TECHNICAL UNIVERSITY FACULTY OF AERONAUTICS & ASTRONAUTICSDEPARTMENT OF METEOROLOGICAL ENGINEERING Digital Image Processing SEZEL KARAYUSUFOĞLU 110030226

2. DIGITAL IMAGE PROCESSING . There are severaladvantages connected with using digital images: - Digital softcopy workstations running on standard computers can be used rather than expensive plotting devices requiring special hardware,  - Data transfer is simpler with regard to plotting results because they can be post-processed on the same computer,  - Digital image processing techniques can be used for image enhancement, - Digital image processing techniques render possible the automation of photogrammetric measurement tasks (=> Research Goals).  However, there are problems connected with image acquisition. For many applications the format of CCD-sensors which are commercially available is too small or the sensors are too expensive. In this case or if high precision is required and metric cameras have to be used, one will first get analogue photographs. Those images have to be scanned offline using photogrammetric scanners, which turns out to be a very time-consuming process. In order to achieve precisions comparable to those achieved in analogue photogrammetry, point localization has to be done with an accuracy better than 1 pixel (Subpixel Accuracy).

3. Digital Photogrammetry-Data Aquisatation

4. Digital Photogrammetry - SCANNING Scanning of a one-dimensional function The original image function f(x) is a continuous one regarding location and intensity. 1.Scanning:Discretization of the location by defining picture elements (pixel) 2. Quantization:Discretization of the intensity; the intensity range IG is part of the natural numbers including zero (e.g. IG = 0, 1, 2, … 255 = 8 bit quantization)  The result of this procedures is a digitized (“digital”) image.

5. Digital SCANNING The sampling procedure can be interpreted as convolution of the image function f(x,y) with the response function r(x,y) of the sensor area Δx Δy. 1 |x| ≤ Δx/2 |y| ≤ Δy/2 r(x,y) = 0 |x| > Δx/2 |y| > Δy/2 Two-dimensional convolution S(x,y) = f(x,y) * r(x,y) = f(m,n) g(x-m, y-n)dm dn Discrete two-dimensional convolution S(x,y) = The sampling theorem states, that the spatial frequencies fx and fy, which can be recovered from the data, are a function of the sampling intervals Δx and Δy.

6. IMAGE CORRELATION The aim of image correlation is to determine identical (homoloque) points in digital image data. For this purpose a reference matrix A is defined in the first image and a (larger) search matrix B in the second one. Then the correlation coefficients r are calculated for each possible position (n,m) of the reference matrix within the search matrix. A Referencematrix B Search matrix

7. IMAGE CORRELATION The maximum of r defines the most probable position of A within B. Least-squares method of image correlation 1. Definition of two submatricesg1, g2 2. Approximate valuesImage coordinate differences ∆x, ∆y (<2 pixel) Parameters of the affine transformation r0 = 0 r1 = 1  Number of iterations k  3.For each pixel Radiometric adjustment

8. IMAGE CORRELATION 5.Determination of intensity gradients out of g2 6.Establishing the error equations 7.Esstablishment and solution of normal equations N= ATA  Solution yields the unknowns 8.Correction of approximate values

9. IMAGE CORRELATION 10.Standard error of unit weight Number of pixelsn Standard error of unknowns

10. Image Processing Image processing topics are related to applications directly performed by human vision. The main subjects of image processing are image restoration and image enhancement.Image restoration topics contain the techniques, which are used to repair a deformed image by using known or predicted statistical or geometrical degradation models. Image restoration is used in the applications, which require quality images such as in photogrammetry, in publication, in photography etc. Repair of the distorted images and filtering of the noises are example of restoration applications. Image enhancement topics are usually used to have better appearance of images to human vision. For example, contrast stretching is an enhancement application. There is no general enhancement techniques, which are used to enhance every kind of images, without any modification. For example, if an enhancement technique, which is applied to a satellite image, is applied to a medical image, the desired result may not be obtained

11. SAMPLING of DIGITAL IMAGES Conversion process of video signals to digital images is called digitization. To process an image by a computer, it should be converted to digital form. Because video signals are continuous or analog functions. A typical video signal may be considered as having windows or frames which contain video information. Every window or frame comprises fully one screen visual information. Every frame is subdivided to fields and fields contain line information. In fig-1, one frame video information has shown. Figure-1: Frames and lines In fig-1, solid lines show fields and dashed lines correspond line information.

12. voltage one line information time one line one pixel SAMPLING of DIGITAL IMAGES In fig-2 a video signal corresponds to a line video signal has shown. In fig-2 if it is noticed that, at the beginning and end of the signal, impulses are seen. These indicate the imaging system hardware, a new line data is sent. Analog video signals are converted to digital form by sampling with equal time intervals.

13. one line information impulse signals SAMPLING of DIGITAL IMAGES In fig-3 an analog signal and its sampling have been shown. As seen in fig-3, a line video signal is sampled in equal time intervals and corresponding voltage values are recorded as integers. These integers correspond to pixel brightness at t time or t point. After a frame had been digitized, this frame can be stored and processed as digital image on computers. Digitized image may be considered as two dimensional array (matrix). Every element of the array corresponds to one pixel in the image. In general f(x,y) notation is used for digital images. Here f(x,y) is the brightness value of (x,y) point and x, y are row and column number of image respectively Figure-3: Sampling of analog signal (digitization)

14. After an image had been digitized, image data can be processed with a software. Processing operations can be shown hierarchically in figure-4, [Umbaugh, 1998]. feature extraction, pattern recognition feature object transformations, image segmentation, edge detection spectral segmentation, edge/line preprocessing neighboring, segmentation raw pixel data pixel Low level SAMPLING of DIGITAL IMAGES After an image had been digitized, image data can be processed with a software. Processing operations can be shown hierarchically in figure-4 In the lowest level of the pyramid, raw image data is seen. At this level, some preprocessing operations can be performed directly to pixels. At the second level, by using pixel neighborhood, preprocessing operations may be done. And at the higher levels, more complex operations are performed. For example edge extraction and object recognition etc. High level Figure-4: Image Operations Figure-4: Image Operations

15. max brightness 0 space RESOLUTION of IMAGES Resolution may be described as measure of the power that how two neighbor pixels are separated. If two adjacent pixels are seen as one pixel, they cannot be distinguished from each other. In other words, if two adjacent pixels belong to two different objects, and if they seem as one pixel, then these two different objects will be seem as one. And this result causes mistakes in some applications. One can say that, the higher resolution, more much object details on an image. Spatial Frequency Resolution In fig-5, to express these spatial frequencies, a square wave has been shown. Frequency shows that how often the signal changes in the space. The signal in the figure has two values, zero and maximum. Figure-5: a.) Square wave used to form 5.b. image, b.) Low frequency image (f=2).

17. RESOLUTION of IMAGES If the square signal in fig-5 is considered to be used to represent one line of an image, and if it repeats as same down to whole image rows, then an image is obtained which is formed by vertical strips as in fig-5.b. If the frequency of the wave is increased, vertical strips get close to each other and they also get narrow. In this case, whole image will have more details. But in this case, if someone looks at the image from a far distance, that eye very hard may resolve time vertical strips and in a limit distance they will not be distinguished. If so, one can say that, spatial frequency is also related to the distance of the location of an observer from the image. To define this relation (relation between spatial frequency and observation distance), a sight angle and vertical period is needed. Here the period is exact change of a signal. For example a period for a square wave corresponds to one minimum and one maximum point. As seen, to define a period, at least two pixels are needed. One for maximum, and one for minimum point. When more much periods are used, degrees correspond to sight angle i.e., when some get away image, to provide same resolution, proportional to distance, wider periods should be used. With this thought, a metric measure of resolution is obtained for the human vision system.

18. REPRESENTATION of IMAGES on COMPUTERS Images are considered as 2-D functions and shown with f(x,y) notation. Brightness of a pixel at (x,y) location corresponds to f(x,y) data. In linear algebra 2-D arrays contain f(x,y) data and 2-D image data f(x,y) is expressed in matrix form. One row or column of a matrix is called vector. The images may be binary, gray-level or multi color.  4.1. Binary Images Binary images are the simplest form. In these images, pixels may only have two different values, for example 0=black, 1=white. Binary images are often used to represent printed documents or to represent the basic shapes of interested objects. To transmit printed documents by FAX, generally binary images are used. This images may also be used for optical character recognition. To have a binary image, commonly thresholding techniques are used. For example if a gray-level image is thresholded at a chosen level, the values equal or exceeds the threshold are assigned 1 and the lower values are assigned zero. Thus the final thresholded image is binary.

19. RESOLUTION of IMAGES Gray-Level Images Gray level images bear tone information. Bit number used to represent a pixel corresponds to a different brightness value. For example 8 bop data corresponds 256 different gray level values between 0-256. Color Images Color images may be considered as a collection of three different gray-level bands. In the color images, real information belongs to three separate spectral bands are stored. The most common type of color images is RGB images. If each color band requires 8-bit, that time an RGB image is formed with 24 bits. In fig-6, typical RGB image structure has shown. 6.a. Typical RGB images may be considered as 3 different images fR(x,y), fG(x,y) fB(x,y) 6.a. Typical RGB images may be considered as 3 different images fR(x,y), fG(x,y) fB(x,y)

20. RESOLUTION of IMAGES In many applications, RGB color information is transformed to a mathematical space and it is converted to brightness value to be used. After this operation the image contains 1-D brightness information or 2-D color space. 2-D color spaces don’t contain brightness values. In stead of this it contains typical color information related to different color amounts. Transformation of color images in to mathematical spaces mentioned above, provides to be described the colors easily by humans Another color transformation based on geometrical coordinates is spherical transform. With this transformation brightness values are obtained from color information. Spherical coordinate transform can be used in color segmentation algorithms. The relations between RGB components and spherical coordinates may be expressed by the following equations; Here L is length of RGB vector. A is the angle between B axis and RG plane and B is the angle between R and G axis. Here L bears brightness information while A and B angles bear color information. In fig-7, HSL space has shown.

21. White Green Hue saturation Blue Red Black RESOLUTION of IMAGES Figure-7: HSL color space Nearby in fig-8, RGB to HSL transformation has shown.

22. RESOLUTION of IMAGES One problem related to the color spaces described above is that they are not perceptually uniform. This means that a color described in a space may be percepted as a different color in the other space even if they are at the same location. So, there is no metric criterion to measure the resemblance of the colors in terms of human perception. In computer vision to recognize objects, it’s necessary to compare the colors. And to do this we should find a method for this comparison. For this purpose in computer vision systems uniform color spaces may be used. Color and human vision perception subjects have been studied extensively in many scientific researches. Especially, Commission Internationale de L’Eclariage (CIE) has defined some color standards on this subject. One of these standards is chromatic coordinates. Chromatic coordinates for RGB color space is defined as;

23. IMAGE ANALYSIS Image analysis is used in computer imaging, to find how to perform processing operations in an image. These analysis techniques constitute a part of wide range image processing topics. Usually, image analysis is performed with iterative steps, and these are used to find the answers of some special questions. For example, the questions; Is color information needed?, Is it necessary to transform image to frequency domain?, Is segmentation required for extraction of information about objects in the image?, What are the important details?, are answered with image analysis. Image analysis process may be subdivided into three groups as in fig-9; preprocessing, data reduction and feature analysis. Preprocessing operations are used to filter noises, and to eliminate unnecessary or unwanted details or data. Furthermore, preprocessing contains gray-level or spatial quantisation processes. Such as reducing number of bits per pixel or the image size. Second step is data reduction. In this step, the data is reduced either in spatial domain or transformed to frequency domain, and next feature analysis step is performed

24. IMAGE ANALYSIS In computer vision, final production is commonly high level analysis or processing information. For example, the geometrical or other spatial information to move a robot arm, is higher level information obtained after higher level analysis. In medical imaging, the information describes a tumor is also an example of high level information, and these information is obtained with image analysis. In image processing applications, image analysis methods are also too useful to decide that which processing techniques should be used to reach aimed purpose. For example, in image restoration, finding the degrading model to repair image is defined by image analysis. Figure- 9: Image Analysis Operations.

25. IMAGE ANALYSIS The mathematical models for image analysis are constructed either in pixel geometry (spatial domain) or in frequency domain (or spectral domain), (fig-10). Figure-10: Image analysis domains

26. SIMPLE ALGEBRAIC OPERATIONS in IMAGES Algebraic operations used for images are commonly viewed in two groups; mathematical and logical operations. Image adding, subtracting, dividing and multiplying operations constitute mathematical processings and “AND, OR, NOT” etc. operations forms logical operations. a.) Image Addition To perform mathematical operations on two images, both in two images corresponding pixels are considered. If we assume that f3 is result image of addition of f1 and f2 images, f3 is obtained by adding corresponding pixel values of f1 and f2. This may be shown as f3(x,y) = f1(x,y) + f2(x,y). In fig-11, a and b images have been added and final image 11-c has obtained. Figure-11: Image addition

27. SIMPLE ALGEBRAIC OPERATIONS in IMAGES b.) Image Subtraction Image subtraction operation is commonly used in motion images to find image differences, (fig-12). Fig-12 image has obtained by subtracting 11-a from 11-b. Figure-12:Subtracted image

28. SIMPLE ALGEBRAIC OPERATIONS in IMAGES c.) Image Multiplication and Division Original image is either multiplied or divided by a constant number. After multiplication, lighter image is obtained and darker image after division, (fig-13). Figure-13: a.) Original image, b.)Original image*2 c.)Original image /2

29. SIMPLE ALGEBRAIC OPERATIONS in IMAGES d.)Logical Operations Logical operations are bitwise operations AND, OR and NOT. With their different combinations XOR, NOR and NAND operations may also be performed easily. AND and OR operations are used to combine information of two images. They have special effects but they’re too useful in image processing. In fig-14 some logical operations is shown. Figure-14:Logical image operations. Infig-14-a, a 128 x 128 image area formed by wholly white pixels is used as mask, then this mask is used to perform AND operation with the image in 13-a, and the final AND image has obtained in, 14-b. In 14-c, not image of 13-a has shown.

30. EDGE / LINE DETECTION For detection of edge and lines in images, various detection operators are used. Most of these are applied with convolution masks and most of these are based on differential operations. Differentials for edge/line detection are used to define color or brightness changes of pixels and their change directions. If there is an abrupt change of brightness in a short interval in an image, it means that in that interval there is an edge with great probability. Some edge/line detection operators also contain dimension information in spite of brightness. Edge/line detection operators are used as primary step to define objects. Potential edges of complex objects are fixed with edge operators as first step. During this process, direction of each possible edges are defined too. In the second step, using both brightness and dimension information joins these potential edges. Thus the final edges of objects are obtained. Many of the edge operators are negatively affected from noises in noisy images. So, before edge detection noises should be filtered as possible. Instead of this, frequency domain filters may also be used for edge detection. But here spatial domain methods are being considered.

31. Brightness Brightness Spatial coordinates Spatial coordinates b.) Real edge a. Ideal edge EDGE / LINE DETECTION The size of operator mask and operator coefficients should be chosen carefully not to be effected of noises. Otherwise, noise points may be accepted as an edge point. If the mask size is chosen greater, that time operator is more effected from noises. The basic idea of edge detection operators is based on comparison of the brightness values of pixels with their neighbors. For example, if a pixel’s value is the same as its neighbors or too close, then with a great probability this pixel will not lie on an edge. If there is difference between pixel and its neighbors then it may be thought that this pixel may lie on an edge. An ideal edge appears with an abrupt gray-level change. But in practice, instead of such abrupt change, smooth changes occur. In fig-15, an ideal and corresponding practical situation of gray-level changes of an edge has shown. Figure-15: Ideal and real (practical edge)

32. EDGE / LINE DETECTION a.)Sobel Operators Sobel edge detection masks detect vertical and horizontal edges separately. And these directional edges are combined finally. Coefficients of 3 x 3 sobel masks are; horizontal mask vertical mask These two masks are convoluted with image separately. The magnitude and direction of an edge is calculated by using convolution results of two masks. These are; S= In fig-16, effect of sobel operators has shown.

33. EDGE / LINE DETECTION 16.a.Effect of horizontal Sobel mask 16.c. Sobel edge magnitude 16.b. Effect of vertical Sobel mask In fig-16, the results of sobel masks applied to original image in fig-13 has shown. 16-a is the result of horizontal sobel mask, 16-b is vertical sobel mask effect and 16-c is magnitude of sobel masks.

34. -1 0 1 1 1 1 0 0 0 -1 0 1 1 1 1 -1 0 1 EDGE / LINE DETECTION Prewitt Operators Prewitt operators are similar to sobel, but the mask coefficients are different. horizontal mask vertical mask S=. Magnitude and direction of an edge is found by; 17.a.Effect of horizontal Prewitt mask 17.b. Effect of vertical Prewitt mask

35. EDGE / LINE DETECTION Laplace Operators Laplace operators are used as the ones described above. Laplace operators are symmetric. I.e all changes in all directions effect the result. Sign of the result defines the direction of gray-level change. Below laplacian masks has shown. 0 -1 0 1 -2 1 -1 5 -1 -2 5 -2 0 -1 0 1 -2 1 In fig-20, original image in 20-a has been projected on to l1 and l2 laplacian masks. Magnitude of edges has shown in 20-d.

36. EDGE / LINE DETECTION Figure-20: a.)Original image, b.) Effect of l1 mask c.) Effect of l2 mask, d.) Edge magnitude

37. EDGE / LINE DETECTION Hough Transformation Hough Transformation is used to define known geometrical shapes such as line, circle, ellipse etc. With the edge operators, only the pixels, which are, potentially edge pixels are found. However a line is formed with the neighbor (connected) pixels arranged at the same direction. In other words, lines should express the potential edge pixels lie on the same direction. To do this, Hough transformation may be used.

38. IMAGE FILTERING With SPATIAL MASKS Spatial filters are used effectively to filter noises. These filters may be effective from 3 x 3 neighbourhood to 11 x 11. Spatial filters are subdivided in two groups; ordered filters and average filters. In ordered filters, gray-level values in the chosen neighbourhood are ordered ascending manner. From this order, the pixel which has the minimum probability to be noise is chosen and center pixel is replaced with this chosen value. In average filters, according to a defined average criterion, the average value of the neighbouring pixels are calculated and this value is put to the center pixel location. Ordered filters are usually used to filter salt and pepper noises, negative exponential noises and Rayleigh distributed noises. Average filters are used to remove Gauss distributed noises or uniform distributed noises. The most effective ordered filter is median filter. With this filter, gray-level values in a chosen neighbourhood are sorted in the ascending manner. Center pixel of the mask is replaced with the median value of the ordered gray-levels. With sorting operation, noise valued pixels will be either on the head of the array or in the end of the array with a great probability. And the median value will not be a noise point with a great probability. If there is a noise point in the center of the mask and if it’s replaced with median value, then the noise could be filtered out. In fig-23 the effect of median filter has shown.

39. IMAGE FILTERING With SPATIAL MASKS a. noisy image b.median filtered image with3*3 mask Fig 23-a image is obtained by adding 5000 salt and pepper noise to 16-a image. And this noise image is filtered with a 3 x 3 median filter mask. Result image has shown in 23-b.Some other types of ordered filters are also available. For example in some ordered filters, the average value of max. and min. values is replaced with center pixels.

40. IMAGE FILTERING With SPATIAL MASKS In average filters, the center pixel is replaced with an average value of neighboring pixel. In fig-24, an arithmetic average filter has applied to original image, and the result has shown in 24-b. a. Original image b . Effect of average (mean) filter Figure-24:Average filter.

41. HISTOGRAM ANALYSIS TECHNIQUES Global Brightness Changes In order to change brightness of an image, image histogram should be changed. To do this, either the number of gray levels of an image or the gray level values should be changed. The simplest brightness (contrast) changing method is to slide the original image histogram to right or left. This operation is called in some papers as histogram slicing. To slide the histogram to the right, a constant should be added to all gray levels, and to slide it to left the inverse operation is needed. Image multiplation operation previously explained in the previous chapter may also be used for histogram slicing. Another contrast changing method is histogram stretching and compressing. In this method either the gray level values are separated from each other when they’re assembled together, or more separated gray level values are combined

42. HISTOGRAM ANALYSIS TECHNIQUES In fig-25, an original aerial photo and its histogram has shown. In fig-26, this original image has remapped by bunching gray levels with 10 intervals. Final image in fig-26 has 25 gray levels, while original has 256 Figure-25:Original aerial photo image with 256 gray-levels and its histogram.

43. HISTOGRAM ANALYSIS TECHNIQUES Figure-26: Bunched (histogram compressed) image with bunchsize=10 and its histogram

44. HISTOGRAM ANALYSIS TECHNIQUES Thresholding This technique is used to remove the gray-level trends in an image, to make the gray-level regions more discrete. This technique is generally used to transform the original image to a binary form. For example, a threshold value say “k” is chosen. The gray-levels greater than k is set to maximum level, and levels less than k are set to minimum level of image. This technique is generally used for processing scanned text images. The same technique may be used with more threshold values for the same image also. Original image in fig- 25 is thresholded at the gray-level 155. The levels greater than or equal 155, are set to value 255 which is the maximum of range, and the levels less than 155 are set to minimum range value zero. The result thresholded image has shown in Figure 27.

45. HISTOGRAM ANALYSIS TECHNIQUES Figure-27:Thresholded binary image, ( threshold value T= 155.)

46. HISTOGRAM ANALYSIS TECHNIQUES Histogram Equalization In the original image histogram in fig-25, the gray-levels of the image are too close to each other. And this image’s density function is not uniform or near uniform. This means that the probabilities of the gray-levels are not equal. An ideal image’s density function should be uniform, i.e. the probabilities of the gray-levels should be equal to each other. If we transform the original histogram to a uniform histogram, new image will be close to ideal. Such a transformation is called histogram equalization. If the original image histogram is transformed to another histogram by using its own distribution function as transformation function, the result image histogram is equalized, (fig-28),

47. Figure-28: Histogram equalized image transformed by its original distribution function