Image Cluster ing

1. Image Clustering Emel Doğrusöz Esra Ataer Muhammet Baştan Tolga Can Information Retrieval Systems

2. Outline • What is image clustering/segmentation? • Why is it used? • Segmentation algorithms. • Effectiveness • How C3M can be applied to image segmentation? • Summary Information Retrieval Systems

3. What defines an object? "I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees." --Max Wertheimer Information Retrieval Systems

4. Segmentation and Grouping •To recognize objects – rather thandealing with too many pixels – weneed a compact/summaryrepresentation •Obtain this representation fromimage/motion sequence/set oftokens •“What is interesting and what isnot” depends on the application Information Retrieval Systems

5. Image segmentation • Segmentation = splitting an image into regions based on some criteria (intensity, color, texture, orientation energy, …). Information Retrieval Systems

6. Segmentation and Grouping • Tokens – whatever we need to group (pixels, points, surface elements, etc.) • Grouping (or clustering) – collect together tokens that “belong together” – top down segmentation • tokens belong together because they lie on the same object – bottom up segmentation • tokens belong together because they are locally coherent • Fitting – associate a model with tokens – Issues: which model? which token goes to which element? how manyelements in the model? Information Retrieval Systems

7. Image segmentation • Haralick and Shapiro • ”The regions should be uniform and homogenous with respect to some characteristic such as intensity value or texture. Region interiors should be simple and without many small holes. Adjacent regions should have significantly different values with respect to the characteristic on which they are uniform. Boundaries of each segment should be simple, not ragged, and must be spatially accurate.” Information Retrieval Systems

8. Applications • Object recognition : recognize objects from image • Optical character recognition (characters, words from document image) • Face recognition • Fingerprint recognition • Medical image processing - diagnosis • e.g., cancerous/healthy cell • Industrial automation • Content based image retrieval • e.g., searching for an object class from a video database Information Retrieval Systems

9. Segmentation Algorithms • Simple Segmentation Algorithms • Thresholding • Background Subtraction • Segmentation by Clustering • Simple Clustering Methods • K-means • Segmentation by Graph-Theoretic Clustering • Normalized Cuts Information Retrieval Systems

10. Simple Clustering Methods Two natural Algorithms: • Agglomerative clustering • attach closest to cluster it is closest to • repeat • Divisive clustering • split cluster along best boundary • repeat Information Retrieval Systems

11. Simple Clustering Methods • Point-Cluster distance • single-link clustering • complete-link clustering • group-average clustering • Dendrograms • yield a picture of output as clustering process continues Information Retrieval Systems

12. Divisive Methods • Construct a single cluster containing all points • Until the clustering is satisfactory - Split the cluster that yields the two components with the largest intercluster distance Information Retrieval Systems

13. Divisive Methods, an Example Information Retrieval Systems

14. Agglomerative Methods • Make each point a separate cluster • Until the clustering is satisfactory • Merge the two clusters with the smallest inter-cluster distance Information Retrieval Systems

15. Thresholding • Gray level thresholding is the simplest segmentation process. Multilevel thresholding (object) (background) Information Retrieval Systems

16. Thresholding • Thresholding is computationally inexpensive and fast • Correct threshold selection is crucial for successful threshold segmentation Information Retrieval Systems

17. Thresholding-example Information Retrieval Systems

18. K-means Clustering • Choose a fixed number of clusters • Choose cluster centers and point-cluster allocations to minimize error Information Retrieval Systems

19. K-means Algorithm • Choose k data points to act as cluster centers • Until the clustering is satisfactory • Assign each data point to the cluster that has the nearest cluster center • Ensure each cluster has at least one data point splitting, etc • Replace the cluster centers with the means of the elements in the clusters Information Retrieval Systems

20. Image Clusters on intensity Clusters on color K-means clustering using intensity alone and color alone Information Retrieval Systems

21. Segmentation By Graph-Theoretic Clustering • Form the graph • Cut the graph • Consider each connected part as a cluster Information Retrieval Systems

22. Forming Graph • Each pixel is treated as a node in a graph • Each node (pixel) is linked to its neighboring nodes (pixels), the strength of each link (in graph theory, a link is also called an edge) is determined by the affinity measure of the two associated nodes (pixels). For example, if two pixels have similar color, or texture, or motion, then the link between them is strong Information Retrieval Systems

23. Criterias • Affinity by Distance • Affinity by Intensity • Affinity by Color • Affinity by Texture Information Retrieval Systems

24. Example Graph Information Retrieval Systems

25. Example Graph Information Retrieval Systems

26. Cut the Graph • To separate pixels into groups, we need to “cut” some edges. Graph cut methods find the cut that minimizes the total edge weights in the cut while maximizing the edge weights in the clusters Information Retrieval Systems

27. Cutting the Graph Information Retrieval Systems

28. Information Retrieval Systems

29. Information Retrieval Systems

30. Eigenvectors and Cut • Aim : to assign each pixel to a cluster while maximizing the affinity between pixels in each cluster • We could maximize • But we have to satisfy • This is an eigenvalue problem – choose eigenvector of A with largest eigenvalue Information Retrieval Systems

31. Algorithm Information Retrieval Systems

32. Min Cut is not always the best Information Retrieval Systems

33. Normalized Cuts • A cut penalizes large segments • Fix by normalizing for size of segments • Volume( A ) = sum of costs of all edges that touch A Information Retrieval Systems

34. Example: Information Retrieval Systems

35. Background subtraction • If we know the background • Use moving average etc. to estimage backgroud • Subtract the background from the current frame • Large absolute values are interesting pixels • Applications • Traffic monitoring • Surveillance/security • User interaction Information Retrieval Systems

36. Background subtraction • Segment moving foreground from static bacground Current image Background image Foreground pixels Information Retrieval Systems

37. Some Other Methods • Watershed Segmentation • Fitting & Hough Transform • Fit line, circle, etc. • Segmentation with Expectation Maximization (EM) • New algorithms are emerging... Information Retrieval Systems

38. Quality of Segmentation • How to judge the effectiveness of the segmentation algorithm? • The Berkeley Segmentation Dataset and Benchmark • 12000 hand labeled segmentation of images from Corel data set Information Retrieval Systems

39. Quality of Segmentation Information Retrieval Systems http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/BSDS300/html/dataset/images.html

40. Quality of Segmentation • In scientific articles, the output of the segmentation algorithm is given for some example images.  judging by visual inspection Information Retrieval Systems From Trecvid 2003 News Videos K-means, K = 5

41. C3M • What should correspond to the document vectors? • Individual pixels • A group of pixels • Entire image Information Retrieval Systems

42. C3M • Individual pixel  document vector • If single band (e.g., grayscale image) • One value per pixel (if no position info is kept) • Histogram will be too sparse, only one nonzero. Ex: [ 0 0 0 ... 1 ... 0 0 0] (1 x #bins), some columns will be zero, not desired. • Multiple bands (e.g., RGB, HSV images) • 3 or more values per pixel • D-matrix : (#pixels) x (#columns) • Ex: 264 x 352 HSV image  92928 x 3 D-matrix Information Retrieval Systems

43. C3M • Group of pixels (e.g., 4x4, 8x8 grids) • Vectors can contain more data • Mean, std of pixel values, texture, etc. • Entire images (to group similar images) • Several color, texture features can be extracted from each image to form a feature vector corresponding to a document vector Information Retrieval Systems

44. C3M - Effectiveness • Will the resulting segmentation be meaningfull? • Advantage over K-means • Number of clusters can be estimated beforehand • Single pass (may be more efficient, faster) • Is it possible to apply C3M in a graph theoretic clustering algorithm (e.g., Normalized Cuts)?  better segmentation Information Retrieval Systems

45. Summary • Segmentation is still an unsolved problem in image processing & computer vision. • Some algorithms perform well on some specific domains and poorly on others. • “Normalized Cuts” is nowadays popular, usually giving the best segmentation. • Variations of the existing algorithms are emerging continuously. • C3M can be adapted to image segmentation Information Retrieval Systems