A Heuristic K-means C lustering A lgorithm by K ernel PCA

1. A Heuristic K-means Clustering Algorithm by Kernel PCA Mantao Xu and Pasi Fränti UNIVERSITY OF JOENSUU DEPARTMENT OF COMPUTER SCIENCE JOENSUU, FINLAND

2. Problem Formulation Given N data samplesX={x1, x2, …, xN}, construct the codebook C ={c1, c2, …, cM} such that mean-square-error is minimized. The class membership p (i) is

3. Traditional K-Means Algorithm • Iterations of two steps: • assignment of each data vector with a class label • computation of cluster centroid by averaging all data vectors that are assigned to it • Characteristics: • Randomized initial partition or codebook • Convergence to a local minimum • Use of L2, L1 and L distance • Fast and easy implementation • Extensions: • Kernel K-means algorithm • EM algorithm • K-median algorithm

4. Motivation Investigation on a clustering algorithm that : • performs the conventional K-Means algorithm in searching a solution close to the global optimum. • estimates the initial partition close to the optimal solution • applies a dissimilarity function based on the current partition instead of L2 distance

5. Selecting initial partiton Based on kernel feature extraction approach and dynamic programming (DP): • Construct 1-D subspace by kernel PCA. • Find a suboptimal partition by DP in the 1-D subspace. • Output the partition of the DP as the initial solution to the K-means algorithm.

6. Kernel PCA vs. PCA Sovle principal component analysis (PCA) in the reproducing kernel space F, thus implicitly depicts the irregular shape of data with a nonlinear hypercurve: PCA Kernel PCA

7. Problem formulation of kernel PCA For the kernel PCA for the N data samples, X={xi}, solve the eigenvalue problem in F, which is assumed to be equivalent to: where the eigenvector V is a linear expansion by ans K is the kernel matrix with respect to data X

8. Dynamic programming in kernel component direction The optimal convex partition Qk={(qj-1,qj]| j=1,,n} in the kernel component direction wcan be obtained by dynamic promgramming in terms of either MSE distortion on one-dimensional kernel component subspace (1) or in terms of MSE distortion on original feature space (2)

9. ,G )=AddVariance Delta-MSE(x 4 2 x 2 y 1 G G y x x 2 1 4 1 2 y 3 x 3 Delta-MSE(x ,G )=RemovalVariance 4 1 Application of Delta-MSE Dissimilarity Move vector x fromcluster ito cluster j, the change of the MSE function [10] caused by this move is:

10. Pseudocodes of the heuristic K-Means

11. Four K-Means algorithms used in the experiments • K-D tree based K-Means: selects its initial cluster centroids from the k-bucket centers of a kd-tree structure that is recursively built by PCA. • PCA-based K-Means: estimate a sub-optimal initial partition by applying the dynamic programming in the PCA direction • KPCA-I: the proposed K-Means algorithm based on the dynamic programming criterion (1) • LFD-II: the proposed K-Means algorithm based on the dynamic programming criterion (2)

12. Performance comparison 1 F-ratio validity index values for UCI data sets:

13. Performance comparison 2 F-ratio validity index values for image data sets:

14. F-ratio validity index values

15. F-ratio validity index values

16. F-ratio validity index values

17. F-ratio validity index values

18. Conclusions A new approach to the k-center clustering problem by incorporating the kernel PCA and dynamic programming. The proposed approachin general is superior to thetwo other algorithms: the PCA-based and the kd-tree basedK-Means. Gain in classification performance of the proposed approach increases with the number of clustersin comparison to two others.

19. Further Work • Solving the k-center clustering problem by iteratively incorporating the kernel Fisher discriminant analysis and the dynamic programming technique • Solving the k-center clustering problem by boosting a decision function (conduct decision function f over X to obtain a scalar space f(X)).