Jianping Fan CS Department UNC-Charlotte

1. Density-Based Data Clustering Algorithms: K-Means & Others Jianping Fan CS Department UNC-Charlotte http://webpages.uncc.edu/jfan/

2. What is Data Clustering? • Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups

3. Applications of Data Clustering • Understanding • Group related documents for browsing, group genes and proteins that have similar functionality, or group stocks with similar price fluctuations • Summarization • Reduce the size of large data sets Why? Clustering precipitation in Australia

4. This is the goal for visual analytics

5. Inter-cluster distances are maximized Intra-cluster distances are minimized What are key issues for data clustering? • Similarity or distance function • Inter-cluster similarity or distance • Intra-cluster similarity or distance • Number of clusters

6. What are key issues for data clustering? • Similarity or distance function x z Distance Similarity function

7. What are key issues for data clustering? • Intra-cluster similarity or distance

8. What are key issues for data clustering? • Inter-cluster similarity or distance

9. What are key issues for data clustering? • Number of clusters K

10. What are key issues data clustering? • Objective function for clustering Inter-Cluster Distance

11. What are key issues data clustering? • Objective function for clustering Intra-Cluster Distance

12. Inter-cluster distances are maximized Intra-cluster distances are minimized What are key issues data clustering? • Objective function for clustering Objective Function for Clustering

13. K-Means (K-Centers) Clustering Cluster Centers

14. K-Means Clustering • Similarity or distance function x z Distance Similarity function

15. K-Means Clustering • Similarity or distance function for K-means clustering

16. K-Means Clustering • Intra-cluster similarity or distance

17. K-Means Clustering • Inter-cluster similarity or distance

18. K-Means Clustering • Number of clusters K

19. K-Means Clustering • Objective function for clustering Inter-Cluster Distance

20. K-Means Clustering • Objective function for clustering Intra-Cluster Distance

21. Inter-cluster distances are maximized Intra-cluster distances are minimized K-Means Clustering • Objective function for clustering Objective Function for Clustering

22. Inter-cluster distances are maximized Intra-cluster distances are minimized K-Means Clustering • Objective Function for K-Means Clustering

23. K-Means Clustering: Clustering for New Sample y Test Sample Test sample is assigned into the closest cluster!

24. How many clusters? Six Clusters Two Clusters Four Clusters Notion of a Cluster can be Ambiguous Why it is ambiguous?

25. K-Means Clustering Approach • Number of K (centers) • Similarity or distance function • Assignment rules • Determining number of centers • Putting centers in dense areas • Distance as similarity • Assign into the closest one

26. K-Means Clustering • What are the potential problems for K-means clustering? • Number of centers? K • Where are the centers? Center selection • Inter-cluster overlapping • Data manifold

27. K-Means Clustering • Number of centers? K • Where are the centers? Center selection

28. K-Means Clustering • Number of centers? K • Where are the centers? Center selection

29. K-Means Clustering • Number of centers? K • Where are the centers? Center selection If K is set too big, what we can do?

30. K-Means Clustering • Number of centers? K • Where are the centers? Center selection If K is set too small, what we can do?

31. K-Means Clustering • If initial K is too big, what we can do further? Merging the overlapped clusters as a bigger one Bottom-Up Approach

32. K-Means Clustering • If initial K is too small, what we can do further? Top-Down Approach Separating diverse cluster into multiple homogeneous ones

33. K-Means Clustering • Where are the centers? Center selection How to identify dense regions?

34. K-Means Clustering • How to identify the dense regions for center selection? • Interactive selection via visualization When I see it, then I know it

35. K-Means Clustering • How to identify the dense regions for center selection? • Automatic selection via DB-Scan

36. K-Means Clustering • Inter-cluster overlapping

37. K-Means Clustering • Data Manifold Spectral Clustering

38. K-means Clustering • Partitional clustering approach • Each cluster is associated with a centroid (center point) • Each point is assigned to the cluster with the closest centroid • Number of clusters, K, must be specified • The basic algorithm is very simple Key Issues for K-Means: • K centroids • Similarity function • Objective function for optimization

39. K-means Clustering – Details • Initial centroids are often chosen randomly. • Clusters produced vary from one run to another. • The centroid is (typically) the mean of the points in the cluster. • ‘Closeness’ is measured by Euclidean distance, cosine similarity, correlation, etc. • K-means will converge for common similarity measures mentioned above. • Most of the convergence happens in the first few iterations. • Often the stopping condition is changed to ‘Until relatively few points change clusters’ • Complexity is O( n * K * I * d ) • n = number of points, K = number of clusters, I = number of iterations, d = number of attributes

40. Most common measure is Sum of Squared Error (SSE) For each point, the error is the distance to the nearest cluster To get SSE, we square these errors and sum them. x is a data point in cluster Ci and mi is the representative point for cluster Ci can show that micorresponds to the center (mean) of the cluster Given two clusters, we can choose the one with the smallest error One easy way to reduce SSE is to increase K, the number of clusters A good clustering with smaller K can have a lower SSE than a poor clustering with higher K Evaluating K-means Clusters

41. K-means Given a set of observations (x1, x2, …, xn), where each observation is a d-dimensional real vector,  k-means clustering aims to partition the n observations into k (≤ n) sets S = {S1, S2, …, Sk} so as to minimize the within-cluster sum of squares (WCSS). In other words, its objective is to find: where μi is the mean of points in Si.

42. K-means Given an initial set of k means m1(1),…,mk(1) , the algorithm proceeds by alternating between two steps: Assignment step: Assign each observation to the cluster whose mean yields the least within-cluster sum of squares (WCSS). Since the sum of squares is the squared Euclidean distance, this is intuitively the "nearest" mean.

43. K-means Update step: Calculate the new means to be the  centroids of the observations in the new clusters.

44. Optimal Clustering Sub-optimal Clustering Two different K-means Clustering Original Points

45. Importance of Choosing Initial Centroid

46. Importance of Choosing Initial Centroid

47. Importance of Choosing Initial Centroid …

48. Importance of Choosing Initial Centroid …

49. Problems with Selecting Initial Points • If there are K ‘real’ clusters then the chance of selecting one centroid from each cluster is small. • Chance is relatively small when K is large • If clusters are the same size, n, then • For example, if K = 10, then probability = 10!/1010 = 0.00036 • Sometimes the initial centroids will readjust themselves in ‘right’ way, and sometimes they don’t • Consider an example of five pairs of clusters

50. Solutions to Initial Centroid Problem • Multiple runs • Helps, but probability is not on your side • Sample and use hierarchical clustering to determine initial centroids • Select more than k initial centroids and then select among these initial centroids • Select most widely separated • Postprocessing • Bisecting K-means • Not as susceptible to initialization issues