Combinatorial clustering algorithms. Example: K-means clustering

1. Combinatorial clustering algorithms.Example: K-means clustering

2. Clustering algorithms • Goal: partition the observations into groups ("clusters") so that the pairwise dissimilarities between those assigned to the same cluster tend to be smaller than those in different clusters. • 3 types of clustering algorithms: mixture modeling, mode seekers (e.g. PRIM algorithm), and combinatorial algorithms. • We focus on the most popular combinatorial algorithms.

3. Combinatorial clustering algorithms Most popular clustering algorithms directly assign each observation to a group or cluster without regard to a probability model describing the data. Notation: Label observations by an integer “i” in {1,...,N} and clusters by an integer k in {1,...,K}. The cluster assignments can be characterized by a many to one mapping C(i) that assigns the i-th observation to the k-th cluster: C(i)=k. (aka encoder) One seeks a particular encoder C*(i) that minimizes a particular *loss* function (aka energy function).

4. Loss functions for judging clusterings One seeks a particular encoder C*(i) that minimizes a particular *loss* function (aka energy function). Example: within cluster point scatters

5. Cluster analysis by combinatorial optimization Straightforward in principle: Simply minimize W(C) over all possible assignments of the N data points to K clusters. Unfortunately such optimization by complete enumeration is feasible only for small data sets. For this reason practical clustering algorithms are able to examine only a fraction of all possible encoders C. The goal is to identify a small subset that is likely to contain the optimal one or at least a good sub-optimal partition. Feasible strategies are based on iterative greedy descent.

6. K-means clustering is a very popular iterative descent clustering methods. Setting: all variables are of the quantitative type and one uses a squared Euclidean distance. In this case Note that this can be re-expressed as 

7. Thus one can obtain the optimal C* by solving the enlarged optimization problem This can be minimized by an alternating optimization procedure given on the next slide…

8. K-means clustering algorithm leads to a local minimum 1. For a given cluster assignment C, the total cluster variance is minimized with respect to {m1,...,mk} yielding the means of the currently assigned clusters, i.e. find the cluster means. 2. Given the current set of means, TotVar is minimized by assigning each observation to the closest (current) cluster mean. That is C(i)=argmink ||xi-mk||2 3. Steps 1 and 2 are iterated until the assignments do not change.

9. Recommendations for k-means clustering • Either: Start with many different random choices of • starting means, and choose the solution having smallest value of • the objective function. • Or use another clustering method (e.g. hierarchical clustering) • to determine an initial set of cluster centers.