K-Medoid

1. K-Medoid

2. Partitional Clustering • Partition n objects into k clusters • These techniques start with K clusters (partitions) • The partitions (clusters) is decided in advance by the user.

3. k-medoid methods • There are two best-known k-medoid methods: • PAM (Partitioning Around Medoids) • CLARA (Clustering LARge Applications)

4. PAM (Partitioning Around Medoids) The Idea: • Find a single partition of the data into K clusters • Each cluster has a most representative point • a point that is the most “centrally” located point in the cluster with respect to some measure, e.g., distance. These lead us to the medoid definition…

5. 10 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10 10 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10 Medoid - definition • A medoid is an actual point in the dataset that is centrally located and is therefore representative of the cluster.

6. More precisely… • Object Oj belongs to the cluster represented by Om if: d (Oj, Om) = minOe d )Oj, Oe) • Oj is a non-selected object • Om is a (selected) medoid • d(O1,O2) denotes the dissimilarity or distance between objects O1 and O2. • minOe denotes the minimum over all medoids Oe

7. PAM – In General… To find the k-medoids… • PAM begins with an arbitrary selection of k objects. • Then, in each step, a swap between a selected object Om and a non-selected object Op is made. • As long as such a swap would result in an improvement of the quality of the clustering.

8. B A A B M A simple example for swap • Suppose there are 2 medoids:A and B • And we replace A with a new medoid M.

9. A B M Y • For all the objects Y that are originally in the cluster represented by A: • find the nearest medoid in light of the replacement.

10. B M A Y There are 2 cases: • Case 1: Y moves to the cluster represented by B, but not to the new one represented by M. • Case 2: Y moves to the new cluster represented by M, and the cluster represented by B is not affected. case2 case1

11. M • We also need to consider all the objects Z that are originally in B’s cluster. A Z B

12. M More 2 cases: • Case 3: Z either stays with B • Case 4: Z moves to the new cluster represented by M. case4 A Z case3 B

13. M A Y Z B • Om current medoidthat is to be replaced (e.g., A). • Op new medoid toreplace Om (e.g., M). • Oj other non-medoidobjects that may or may not needto be moved (e.g., Y and Z) • Oj,2 a current medoid that is nearest to Oj without A and M (e.g., B).

14. To formalize the effect of a swap between Om and Op, PAM computes costs Cjmp for all non-medoid objects Oj. • Depending on which of the following cases Oj is in, Cjmp is defined differently.

15. Case 1: • Oj currently belongs to the cluster represented by Om. • Oj be more similar to Oj,2 than to Op, i.e., d)Oj, Op) >= d)Oj, Oj,2) • Thus, Oj would belong to thecluster represented by Oj,2 • The cost of the swap is: Cjmp = d)Oj, Oj,2) – d)Oj, Om) Oj Oj,2 Om Op

16. Case 2: • Oj currently belongs to the cluster represented by Om. • Oj is less similar to Oj,2 than to Op, i.e., d)Oj, Op) < d)Oj, Oj,2) • Thus, Oj would belong to thecluster represented by Op • The cost of the swap is: Cjmp = d)Oj, Op) – d)Oj, Om) Oj Om Oj,2 Op

17. Case 3: • Oj currently belongs to a cluster Oj,2. • Oj is more similar to Oj,2 than to Op. • Then, even if Om is replaced by Op, Oj would stay in the cluster represented by Oj,2. • The cost is: Cjmp = 0 Oj Om Oj,2 Op

18. Case 4: • Oj currently belongs to the cluster represented by Oj,2. • Oj is less similar to Oj,2 than to Op. • Then, replacing Om with Op would cause Oj to jump to the cluster of Op from that of Oj. • The cost of the swap is: Cjmp = d)Oj, Op) – d)Oj, Oj,2) Oj Om Oj,2 Op

19. Total Cost • Combining the four cases , the Total Cost of replacing Om with Op is given by: TCmp =

20. Algorithm PAM • Arbitrarily choose k objects as the initial medoids • Until no change, do • (Re) assign each object to the cluster to which the nearest medoid • Randomly select a non-medoid object Op, compute the total cost, TCmp, of swapping medoid Om with Op • If TCmp < 0 then swap Om with Op to form the new set of k medoids

21. 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 1 0 0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 1 0 0 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 PAM: Example Arbitrary choose k object as initial medoids Assign each remaining object to nearest medoids K=2 Randomly select a nonmedoid object,Oramdom Do loop Until no change Compute total cost of swapping Swapping O and Oramdom If quality is improved.

22. PAM Disadvantage • Experimental results show that PAM works satisfactorily for small data sets (e.g., 100 objects in 5 clusters) .But, it is not efficient in dealing with medium and large data sets. • This is not too surprising if we perform a complexity analysis on PAM. There are altogether k(n-k) pairs.For each pair, computing TCmp requires the examination of (n - k) non-selected objects. • Thus, the complexity combined is of. • And this is the complexity of only one iteration. • Thus, it is obvious that PAM becomes too costly for large values of n and k. • This analysis motivates the development of CLARA.

23. CLARA(Clustering LARge Applications) • Designed to handle large data sets • The Idea: Instead of finding representative objects for the entire data set, CLARA draws a sample of the data set, applies PAM on the sample, and finds the medoids of the sample. • The point is that, if the sample is drawn in a sufficiently random way, the medoids of the sample would approximate the medoids of the entire data set.

24. To come up with better approximations, CLARA draws multiple samples and gives the best clustering as the output. • The quality of a clustering is measured based on the average dissimilarity of all objects in the entire data set. • Experiments shows that samples of size 40 + ‏2K give satisfactory results.

25. Algorithm CLARA • For i = 1 to 5, repeat the following steps: • Draw a sample of 40 + 2k objects randomly from the entire data set, and call Algorithm PAM to find k medoids of the sample. • For each object Oj in the entire data set, determinewhich of the k medoids is the most similar to Oj. • Calculate the average dissimilarity of the clusteringobtained in the previous step.If this value is less than the current minimum, use this value as the current minimum, and retain the k medoids found in Step 2 as the best set of medoids obtained so far. • Return to Step 1 to start the next iteration.

26. Biological Application

27. The Biological Problem Some facts… • Recent advances of experimental techniques and automation in molecular and structural biology have led to the rapid increase in the determination of many protein structures. • The number of structures deposited in the Protein Data Bank (PDB) is now over 20,000 and the contents are growing rapidly.

28. Over half of all of the proteins of sequenced genomes has no inferable molecular functions. • As sequence similarity infers functional similarity, structural similarity also infers similarity in molecular function:if a hypothetical protein has a structure similar to one or more protein structures of known function, the structural similarity infers a powerful clue to the molecular function of the hypothetical protein. • Measures of structural similarity, assessed computationally or visually, between pairs of proteins are also the foundation for classifying protein structures.

29. The Goal • The goal of the method is:To find measures of structural similarity between proteins • We base our method on: distances

30. Some Biological Background…

31. Protein Structure • Amino Acid: • שרשרת פוליפפטידית:

32. Structure of the -Helix: • sheet:

33. The Method

34. The Method • We start with the distance matrix representation of protein structure. • The distance matrix of a protein structure is a square matrix consisting of the distances between all pairs of atoms in the protein.

35. When there are residues in protein p, its distance matrix is the matrix Dp is: {dp(i, j): i, j=1, . . . , } • dp(i, j) is the distance (in Å) between residues i and j.

36. We sub-divide the distance matrix of each protein structure into many overlapping sub-matrices. • The overlapping sub-matrices presenting local features involving m-residues by m-residues in the protein is the following collection : • m x m sub-matrices described by:

37. The collection of these sub-matrices over P proteins is: • We use a collection of these sub-matrices from a large number of distance matrices to extract a set of K medoid sub-matrices by medoid analysis (PAM).

38. Example: 100 Medoids One hundred medoid sub-matrices obtained from partitioning around medoids (PAM) analysis of distance matrices of 100 sampled proteins.

39. Generation of the LFF Profile • Each of the protein sub-matrices is labeled by the index of the nearest medoid sub-matrix. • The count vector summarizes the frequency distribution of local feature patterns of the protein. • Any given protein structure can be represented by a profile, a vector of a common length K, containing the frequencies of occurrence of these medoid sub-matrices in the structure.

40. We call this decoding process:profiling of the protein structure • The final feature vector , profile of protein p

41. We normalize frequency of local interaction pattern k in protein p by:

42. Normalization of the results • Because the abundance of local patterns varies considerably from one pattern to another, some normalization of the profile is necessary. • For example, the ‘‘null’’ pattern is most abundant of all, and, without normalization, such an abundant pattern will dominate when computing structural similarity or dissimilarity distances. • This is not desirable because the frequency of the void pattern contains little structural information.

43. Normalize Vector • Normalize vector X: • . • In our method:

44. Before…

45. After…

46. Calculation of Similarity/ Dissimilarity Scores • The profile of protein P: • The collection of profiles, or the protein-by-pattern matrix:

47. As a measure of structural similarity between two proteins p and q with profiles Ap and Aq in , we use their cosine. • The cosine distance is defined as 1 - cos(Ap, Aq) and used to represent structural dissimilarity or structural distance. Note that the cosine distance ranges from 0 (closest) to 1 (farthest).

48. Problem & Solution • The Problem:The profile of protein P is a vector which belong to . • The Solution:Using SVD which helps to reduce the number of dimensions.

49. Singular Value Decomposition (SVD) • The SVD of matrix A is defined as • U is an m x n matrix • V is a n x n square matrix • U,V are orthgonal so that: • . • Now we can approximating the original protein by pattern matrix by:

50. We compute the truncated SVD with k=3 to obtain approximation A3 of the protein by pattern matrix, because the first three values are significantly greater than the rest. • We can represent proteins and patterns in the same R3 space by their first three principal coordinates: • The 1st -> length of protein • The 2nd -> types of secondary structure elements • The 3rd -> parallelism, direction