An Unsupervised Learning Approach for Overlapping Co-clustering

1. An Unsupervised Learning Approach for Overlapping Co-clustering Machine Learning Project Presentation Rohit Gupta and Varun Chandola {rohit,chandola}@cs.umn.edu

2. Outline • Introduction to Clustering • Description of Application Domain • From Traditional Clustering to Overlapping Co-clustering • Current State of Art • A Frequent Itemsets Based Solution • An Alternate Minimization Based Solution • Application to Gene Expression Data • Experimental Results • Conclusions and Future Directions

3. Clustering • Clustering is an unsupervised machine learning technique • Uses unlabeled samples • In the simplest form – determine groups (clusters) of data objects such that the objects in one cluster are similar to each other and dissimilar to objects in other clusters • Where each data object is a set of attributes (or features) with a definite notion of proximity • Most traditional clustering algorithms • Are partitional in nature. Assign a data object to exactly one cluster • Perform clustering along one dimension

4. Application Domains • Gene Expression Data • Genes vs. Experimental Conditions • Find similar genes based on their expression values for different experimental conditions • Each cluster would represent a potential functional module in the organism • Text Documents Data • Documents vs. Words • Movie Recommendation Systems • Users vs. Movies

5. Overlapping Clustering • Also known as soft clustering, fuzzy clustering • A data object can be assigned to more than one cluster • Motivation is that many real world data sets have inherently overlapping clusters • A gene can be a part of multiple functional modules (clusters)

6. Co-clustering • Co-clustering is the problem of simultaneously clustering rows and columns of a data matrix • Also known as bi-clustering, subspace clustering, bi-dimensional clustering, simultaneous clustering, block clustering • The resulting clusters are blocks in the input data matrix • These blocks often represent more coherent and meaningful clusters • Only a subset of genes participate in any cellular process of interest that is active for only a subset of conditions

7. Overlapping Co-clustering overlaps Co-clusters Segal et al, 2003, Banerjee et al, 2005 Dhillon et al, 2003, Cho et al 2004, Banerjee et al, 2005 [Bergmann et al, 2003] Overlapping Co-clusters

8. Current State of Art • Traditional Clustering – Numerous algorithms like k-means • Overlapping Clustering – Probabilistic RelationalModel Based Approach by Segal et al and Banerjee et al • Co-clustering – Dhillon et al for gene expression data and document clustering. (Banerjee et al provided a general framework using a general class of Bregman distortion functions) • Overlapping co-clustering • Iterative Signature Algorithm (ISA) by Bergmann et al for gene expression data • Uses an Alternate Minimization technique • Involves thresholding after every iteration • We propose a more formal framework based on the co-clustering approach by Dhillon et al and another simpler frequent itemsets based solution

9. Frequent Itemsets Based Approach • Based on the concept of frequent itemsets from association analysis domain • A frequent itemset is a set of items (features) which occur together more than a specified number of times (referred to as support threshold) in the data set • The data has to be binary (only presence or absence is considered)

10. Frequent Itemsets Based Approach (2) • Application to gene expression data: • Normalization – first along columns (conditions) to remove scaling effects and then along rows (genes) • Binarization • Values above a preset threshold λ are set to 1 and the rest to 0. • Values above a preset percentile are set to 1 and the rest to 0. • Split each gene column to three components g+, g0 and g- signifying the up and down regulation of the gene's expression. This triples the number of items (or genes) • Gene expression matrix converted to transaction format data – each experiment is a transaction and contains index values for the genes that were expressed in this experiment

11. Frequent Itemsets Based Approach (3) • Algorithm: • Run closed frequent itemset algorithm to generate frequent closed itemsets with a specified support threshold σ • Post-Processing: • Prune frequent itemsets (set of genes) of length < α • For each remaining itemset, scan the transaction data to record all the transactions (experiments) in which this itemset occurs • (Note: The combination of these transactions (experiments) and the itemset (genes) will give the desired sets of genes with subsets of conditions they are most tightly co-expressed with)

12. Limitations of Frequent Itemsets Based Approach • Binarization of the gene expression matrix may lose some of the patterns in the data • Up-regulation and down-regulation of genes not directly taken into account • Setting up right support threshold incorporating the domain knowledge is not trivial • Large number of modules obtained – difficult to evaluate biologically • Traditional association analysis based approaches only considers dense blocks, noise may break the actual module in this case – Error tolerant Itemsets (ETI) offers a potential solution though

13. Alternate Minimization (AM) Based Approach • Extends the non-overlapping co-clustering approach by [Dhillon et al, 2003, Banerjee et al 2005] • Algorithm • Input: Data Matrix A (size: m x n) and k, l (# of row and column clusters) • Initialize row and column cluster mappings, X (size: m x k) and Y (size: n x l) • Random assignment of rows (or columns) to row (or column) clusters • Any traditional one dimensional clustering can be used to initialize X and Y • Objective function: ||A – Â||2, Â is matrix approximation of A computed as follows: • Each element of a co-cluster (obtained using current X and Y) is replaced by mean of co-cluster (aI,J) • Each element of a co-cluster is replaced by (ai,J + aI,j – aI,J) i.e row mean + column mean – overall mean

14. Alternate Minimization (AM) Based Approach(2) • While (converged) • Phase 1: • Compute row cluster prototypes (based on current X and matrix A) • Compute Bregman distance, dΦ(ri, Rr) - each row to each row cluster prototype • Compute probability with which each of m rows fall into each of k row clusters • Update row cluster X keeping column cluster Y same (some thresholding is required here to allow limited overlap) • Phase 2: • Compute column cluster prototypes (based on current Y and matrix A) • Compute Bregman distance, dΦ(cj, Cc) - each column to each column cluster prototype • Compute probability with which each of n columns fall into each of l column clusters • Update column cluster Y keeping row cluster X same • Compute objective function: ||A – Â||2 • Check convergence

15. Observations • Each row or column can be assigned to multiple row and column clusters respectively by certain probability based on their distances from respective cluster prototypes. This will produce overlapping co-clustering. • Maximum overlapping co-clusters that could be obtained = k x l • Initialization of X and Y can be done in multiple ways – two ways are explored in the experiments • Thresholding to control percent overlap is tricky and requires domain knowledge • Cluster Evaluation is important – internal and external • SSE, Entropy of each co-cluster • Biological evaluation using GO (Gene Ontology) for results on gene expression data

16. Experimental Results (1) • Frequent Itemsets Based Approach • A synthetic data set (40 X 40) Total Number of co-clusters detected = 3

17. Experimental Results (2) • Frequent Itemsets Based Approach • Another synthetic data set (40 X 40) Total Number of co-clusters detected = 7 All 4 blocks (in the original data set) were detected Need post-processing to eliminate unwanted co-clusters

18. Experimental Results (3) • AM Based Approach • Synthetic data sets (20 X 20) • Finds co-clusters for each case

19. Experimental Results (4) • AM Based Approach on Gene Expression Dataset • Human Lymphoma Microarray Data [Described in Cho et al, 2004] • # genes = 854 • # conditions = 96 • k = 5, l = 5, one dimensional k-means for initialization of X and Y • Total Number of co-clusters = 25 Objective Function vs. Iterations Input Data A preliminary analysis of the 25 co-clusters show that only one meaningful co-cluster is obtained

20. Conclusions • Frequent Itemsets based approach is guaranteed to find dense overlapping co-clusters • Error Tolerant Itemset Approach offers a potential solution to address the problem of noise • AM based approach is a formal algorithm to find overlapping co-clusters • Simultaneously performs clustering in both dimensions while minimizing a global objective function • Results on synthetic data prove the correctness of the algorithm • Preliminary results on gene expression data show promise and will be further evaluated • A key insight here is that application of these techniques to gene expression data requires domain knowledge for pre-processing, initialization, thresholding as well as post-processing of the co-clusters obtained

21. References • [Bergmann et al, 2003] Sven Bergmann, Jan Ihmels and Naama Barkai, Iterative signature algorithm for the analysis of large-scale gene expression data, Phys. Rev. E 67, pp 31902, 2003 • [Liu et al, 2004] Jinze Liu, Paulsen Susan, Wei Wang, Andrew Nobel and Jan Prins, Mining Approximate Frequent Itemset from Noisy Data, Proc. IEEE ICDM, pp. 463-466, 2004 • [Cho et al, 2004] H. Cho, I. S. Dhillon, Y. Guan, and S. Sra, Minimum sum-squared residue co-clustering of gene expression data. In Proceedings of SIAM Data Mining Conference, pages 114-125, 2004 • [Dhillon et al, 2003] Inderjit S. Dhillon, Subramanyam Mallela and Dharmendra S. Modha, Information-Theoretic Co-Clustering, Proc. ACM SIGKDD, pp. 89-98, 2003 • [Banerjee et al, 2004] A generalized maximum entropy approach to bregman co-clustering and matrix approximation. In KDD '04: Proceedings of the 10th ACM SIGKDD, pages 509-514, 2004