Christophe Gauge

1. Gannon University Department of Computer and Information Science Automated Colorization of Grayscale Images • Christophe Gauge Advisor: Dr. Sreela Sasi

2. Automated Colorization of Grayscale Images Introduction Image Colorization

3. Automated Colorization of Grayscale Images Introduction (contd.) Digital Image Colorization

4. Automated Colorization of Grayscale Images Introduction (contd.) Applications of Image Colorization

5. Automated Colorization of Grayscale Images + = + = • Previous Research Image Colorization

6. Image Image New Grayscale Image Sample Image Texture-based Segmentation Texture-based Segmentation Feature Extraction Feature Extraction Automated Colorization of Grayscale Images Color Descriptors Texture Descriptors Texture Descriptors Database Texture Matching Colorization Process Current Research Process Workflow

7. Image segmentation: • Is the partitioning of an image into homogeneous regions based on a set of characteristics. • Is a key element in image analysis and computer vision. Automated Colorization of Grayscale Images Image Segmentation

8. Clustering: • Is one of the methods available for image segmentation. • Is a process which can be used for classifying pixels based on similarity according to the pixel’s color or gray-level intensity. Automated Colorization of Grayscale Images Image Segmentation (contd.)

9. Despite the substantial amount of research performed to date, the design of a robust and efficient clustering algorithm remains a very challenging problem Automated Colorization of Grayscale Images Image Segmentation (contd.)

10. Automated Colorization of Grayscale Images Color-based Image Segmentation Composite Image

11. Automated Colorization of Grayscale Images Color-based Image Segmentation Composite Imagewith salt & pepper noise added

12. Automated Colorization of Grayscale Images Texture-based Image Segmentation

13. Original Image Gabor Filters Filtered Image Filtered Image Filtered Image … Filtered Image Energy Computation Feature Image Feature Image Feature Image … Feature Image Automated Colorization of Grayscale Images Add, mean smoothing, normalization Feature Image Segmentation Blobs Workflow Process Texture-Based Image Segmentation

14. Image Segmentation Multi-Channel Filtering - Gabor Transform Automated Colorization of Grayscale Images

15. Automated Colorization of Grayscale Images Previous Research (contd.) Texture-Based Segmentation

16. Image Segmentation Normalized Sum of Gabor Responses Automated Colorization of Grayscale Images

17. Image Image New Grayscale Image Sample Image Texture-based Segmentation Texture-based Segmentation Feature Extraction Feature Extraction Automated Colorization of Grayscale Images Color Descriptors Texture Descriptors Texture Descriptors Database Texture Matching Colorization Process Current Research Process Workflow

18. Automated Colorization of Grayscale Images Previous Research (contd.) • Clustering and Feature Extraction

19. The K-means algorithm has been used for a fast and crisp “hard” segmentation. • The Fuzzy set theory has improved this process by allowing the concept of partial membership, in which an image pixel can belong to multiple clusters. • This “soft” clustering allows for a more precise computation of the cluster membership, and has been used successfully for image clustering and segmentation. Automated Colorization of Grayscale Images Previous Research

20. The Fuzzy C-means clustering (FCM) algorithm [1] is a widely used method for “soft” image clustering. • However, the FCM algorithm is computationally intensive. • It is also very sensitive to noise because it only iteratively compares the properties of each individual pixel to each cluster in the feature domain. Automated Colorization of Grayscale Images Previous Research (contd.)

21. Automated Colorization of Grayscale Images Image Segmentation Modified Fuzzy C-means Clustering

22. Automated Colorization of Grayscale Images Previous Research (contd.) Fuzzy C-means clustering (FCM) Algorithm

23. Step 1 Set the number c of clusters, the fuzzy parameter m, and the stopping condition ε Step 2 Initialize the fuzzy membership values µ Step 3 Set the loop counter b = 0 Step 4 Calculate the cluster centroid values using (3) Step 5 For each pixel, compute the membership values using (4) for each cluster Step 6 Compute the objective function A. If the value of A between consecutive iterations < ε then stop, otherwise set b=b+1 and go to step 4 Automated Colorization of Grayscale Images Previous Research (contd.) FCM Pseudo-code

24. In order to improve the tolerance to noise of the Fuzzy C-means clustering algorithm, Krinidis and Chatzis [2] have proposed a new Robust Fuzzy Local Information C-means Clustering (FLICM) algorithm by introducing the novel Gki factor. The purpose of this algorithm is to adjust the fuzzy membership of each pixel by adding local information from the membership of neighboring pixels. Automated Colorization of Grayscale Images Previous Research (contd.) Modified Fuzzy C-means clustering with Gkifactor

25. The Gkifactor is obtained by using a sliding window of predefined dimensions: Automated Colorization of Grayscale Images Sliding window of size 1 around the ith pixel Previous Research (contd.) Modified Fuzzy C-means clustering with Gkifactor

26. The Gki factor is calculated by using the following equation: Automated Colorization of Grayscale Images Previous Research (contd.) Modified Fuzzy C-means clustering with Gkifactor

27. This algorithm is further improved by including both the local spatial information from neighboring pixels and the spatial Euclidian distance of each pixel to the cluster’s center of gravity. In this research, the algorithm is also extended for clustering of color images in the Red-Green-Blue (RGB) color space. Automated Colorization of Grayscale Images Current Algorithm Modified Fuzzy C-means clustering with novel Hik factor

28. Illustration of the new Hik factor displaying the spatial Euclidian distance to the center of gravity of each cluster Automated Colorization of Grayscale Images Current Algorithm (contd.)

29. Customize Parameters Image Calculate cluster centroid Calculate cluster membership values Compute Gki Compute Hki Automated Colorization of Grayscale Images - Readjust membership values Compute objective function Defuzzification and clustering Current Algorithm (contd.) Process Workflow

30. Automated Colorization of Grayscale Images Current Algorithm (contd.) Modified Fuzzy C-means Clustering

31. Automated Colorization of Grayscale Images Simulation and Results Synthetic Grayscale Test Image

32. Natural test image Automated Colorization of Grayscale Images FCM segmentation using the modified FCM algorithm with 5 clusters, Gki window=1 and Hik FCM segmentation with 5 clusters Simulation and Results Natural Test Image

33. Synthetic 4-color test image with added salt and pepper noise Automated Colorization of Grayscale Images FCM clustering FCM clustering with Gki window=1 and with Hik FCM clustering with Gki window=5 and with Hik Simulation and Results Synthetic Grayscale Test Image

34. Synthetic 4-color test image with added salt and pepper noise Automated Colorization of Grayscale Images FCM clustering with Gki window=1 and with Hik FCM clustering FCM clustering with Gki window=5 and with Hik Simulation and Results Synthetic Color Test Image

35. Automated Colorization of Grayscale Images Image Segmentation Clustering Demo

36. In this research, the FCM with the Gki factor is modified using the Hik factor, and the algorithm is extended for the clustering of color images. • The use of the sliding window in the Gki factor improves the segmentation results by incorporating local information about neighboring pixels in the membership function of the clusters. However, this resulted in a significant increase in the number of calculations required for each iteration for each pixel, and can be given by Automated Colorization of Grayscale Images • Modified Fuzzy C-means Clustering Summary

37. By combining the Gki and the Hik factors, this modified FCM algorithm considerably reduced the number of iterations needed to achieve convergence. The tolerance to noise of the Fuzzy C-means algorithm is also greatly increased, allowing for an improved capability to obtain coherent and contiguous segments from the original image. Automated Colorization of Grayscale Images • Modified Fuzzy C-means Clustering • Summary (contd.)

38. However, because of the radial nature of the spatial Euclidean distance to the cluster’s center of gravity, this new method may not be as effective for images containing circular shapes, or for images where the cluster’s center of gravity are close to each-other. • In this research, the FCM is extended for the clustering of color images in the RGB color space. The effectiveness of this algorithm may be tested for images in other color spaces also. Automated Colorization of Grayscale Images • Modified Fuzzy C-means Clustering • Summary (contd.)

39. Image Image New Grayscale Image Sample Image Texture-based Segmentation Texture-based Segmentation Feature Extraction Feature Extraction Automated Colorization of Grayscale Images Color Descriptors Texture Descriptors Texture Descriptors Database Texture Matching Colorization Process Current Research Process Workflow

40. Sample Color Images Automated Colorization of Grayscale Images

41. Image Segmentation Normalized Sum of Gabor Responses Automated Colorization of Grayscale Images

42. Automated Colorization of Grayscale Images Image Segmentation Feature Extraction

43. Blob Filtering for color and texture extraction. Automated Colorization of Grayscale Images Image Segmentation • Feature Extraction(contd.)

44. Texture and Color database Automated Colorization of Grayscale Images • Image Segmentation • Feature Extraction(contd.)

45. Current Research Process Workflow Image Image New Grayscale Image Sample Image Texture-based Segmentation Texture-based Segmentation Feature Extraction Feature Extraction Automated Colorization of Grayscale Images Color Descriptors Texture Descriptors Texture Descriptors Database Texture Matching Colorization Process

46. Automated Colorization of Grayscale Images • Grayscale Image Processing

47. Current Research Process Workflow Image Image New Grayscale Image Sample Image Texture-based Segmentation Texture-based Segmentation Feature Extraction Feature Extraction Automated Colorization of Grayscale Images Color Descriptors Texture Descriptors Texture Descriptors Database Texture Matching Colorization Process

48. Previous Research Visual descriptors • Visual descriptors are descriptions of the visual features of the contents of images. • They describe elementary characteristics such as the shape, color, and texture. • MPEG-7 is a multimedia content description standard. It was standardized in ISO/IEC 15938 (Multimedia content description interface). • This description is associated with the content itself, to allow fast and efficient searching for material that is of interest to the user. • MPEG-7 is formally called Multimedia Content Description Interface. Thus, it is not a standard which deals with the actual encoding of moving pictures and audio, like MPEG-1, MPEG-2 and MPEG-4. It uses XML to store metadata. Automated Colorization of Grayscale Images

49. Previous Research Visual descriptors The Img(Rummager) application was developed in the Automatic Control Systems & Robotics Laboratory at the Democritus University of Thrace-Greece. The application can execute an image search based on a query image, either from XML-based index files, or directly from a folder containing image files, extracting the comparison features in real time. Automated Colorization of Grayscale Images http://chatzichristofis.info/?page_id=213

50. Automated Colorization of Grayscale Images Previous Research (contd.) • Content-Based Image Retrieval