Introduction to data mining

1. Peter van der Putten Leiden Institute of Advanced Computer Science Leiden University putten@liacs.nl Transcriptomics and Proteomics in Zebrafish workshop, Leiden University March 9, 2006 Introduction to data mining

2. Presentation Outline • Objective • Present the basics of data mining • Gain understanding of the potential for applying it in the bioinformatics domain

3. Agenda Today • Data mining definitions • Before Starting to Mine…. • Descriptive Data Mining • Dimension Reduction & Projection • Clustering • Association rules • Predictive data mining concepts • Classification and regression • Bioinformatics applications • Predictive data mining techniques • Logistic Regression • Nearest Neighbor • Decision Trees • Naive Bayes • Neural Networks • Evaluating predictive models • Demonstration (optional)

4. The Promise…. . . . .

5. The Promise…. . . . .

6. The Promise…. . . . .

7. The Solution…. • NCBI Tools for data mining: • Nucleotide sequence analysis • Proteine sequence analysis • Structures • Genome analysis • Gene expression • Data mining or not?.

8. What is data mining?

9. Sources of (artificial) intelligence • Reasoning versus learning • Learning from data • Patient data • Genomics, protemics • Customer records • Stock prices • Piano music • Criminal mug shots • Websites • Robot perceptions • Etc.

10. Some working definitions…. • ‘Data Mining’ and ‘Knowledge Discovery in Databases’ (KDD) are used interchangeably • Data mining = • The process of discovery of interesting, meaningful and actionable patterns hidden in large amounts of data • Multidisciplinary field originating from artificial intelligence, pattern recognition, statistics, machine learning, bioinformatics, econometrics, ….

11. Some working definitions…. • Bioinformatics = • Bioinformatics is the research, development, or application of computational tools and approaches for expanding the use of biological, medical, behavioral or health data, including those to acquire, store, organize, archive, analyze, or visualize such data [http://www.bisti.nih.gov/]. • Or more pragmatic: Bioinformatics or computational biology is the use of techniques from applied mathematics, informatics, statistics, and computer science to solve biological problems [Wikipedia Nov 2005]

12. Bio informatics and data mining • From sequence to structure to function • Genomics (DNA), Transcriptomics (RNA), Proteomics (proteins), Metabolomics (metabolites) • Pattern matching and search • Sequence matching and alignment • Structure prediction • Predicting structure from sequence • Protein secondary structure prediction • Function prediction • Predicting function from structure • Protein localization • Expression analysis • Genes: micro array data analysis etc. • Proteins • Regulation analysis

13. Bio informatics and data mining • Classical medical and clinical studies • Medical decision support tools • Text mining on medical research literature (MEDLINE) • Spectrometry, Imaging • Systems biology and modeling biological systems • Population biology & simulation • Spin Off: Biological inspired computational learning • Evolutionary algorithms, neural networks, artificial immune systems

14. Genomic Microarrays – Case Study • Problem: • Leukemia (different types of Leukemia cells look very similar) • Given data for a number of samples (patients), can we • Accurately diagnose the disease? • Predict outcome for given treatment? • Recommend best treatment? • Solution • Data mining on micro-array data

15. Microarray data • 50 most important genes • Rows: genes • Columns: samples / patients

16. Example: ALL/AML data • 38 training patients, 34 test patients, ~ 7,000 patient attributes (micro array gene data) • 2 Classes: Acute Lymphoblastic Leukemia (ALL) vs Acute Myeloid Leukemia (AML) • Use train data to build diagnostic model ALL AML • Results on test data: • 33/34 correct, 1 error may be mislabeled

17. Some working definitions…. • ‘Data Mining’ and ‘Knowledge Discovery in Databases’ (KDD) are used interchangeably • Data mining = • The process of discovery of interesting, meaningful and actionable patterns hidden in large amounts of data • Multidisciplinary field originating from artificial intelligence, pattern recognition, statistics, machine learning, bioinformatics, econometrics, ….

18. The Knowledge Discovery Process

19. Some working definitions…. • Concepts: kinds of things that can be learned • Aim: intelligible and operational concept description • Example: the relation between patient characteristics and the probability to be diabetic • Instances: the individual, independent examples of a concept • Example: a patient, candidate drug etc. • Attributes: measuring aspects of an instance • Example: age, weight, lab tests, microarray data etc • Pattern or attribute space

20. Data mining tasks • Descriptive data mining • Matching & search: finding instances similar to x • Clustering: discovering groups of similar instances • Association rule extraction: if a & b then c • Summarization: summarizing group descriptions • Link detection: finding relationships • … • Predictive data mining • Classification: classify an instance into a category • Regression: estimate some continuous value

21. Before starting to mine…. • Pima Indians Diabetes Data • X = body mass index • Y = age

22. Before starting to mine….

23. Before starting to mine….

24. Before starting to mine…. • Attribute Selection • This example: InfoGain by Attribute • Keep the most important ones

25. Before starting to mine…. • Types of Attribute Selection • Uni-variate versus multivariate (sub set selection) • The fact that attribute x is a strong uni-variate predictor does not necessarily mean it will add predictive power to a set of predictors already used by a model • Filter versus wrapper • Wrapper methods involve the subsequent learner (classifier or other)

26. Dimension Reduction • Projecting high dimensional data into a lower dimension • Principal Component Analysis • Independent Component Analysis • Fisher Mapping, Sammon’s Mapping etc. • Multi Dimensional Scaling • ….

27. Data Mining Tasks: Clustering Clustering is the discovery of groups in a set of instances Groups are different, instances in a group are similar In 2 to 3 dimensional pattern space you could just visualise the data and leave the recognition to a human end user f.e. weight f.e. age

28. Data Mining Tasks: Clustering Clustering is the discovery of groups in a set of instances Groups are different, instances in a group are similar In 2 to 3 dimensional pattern space you could just visualise the data and leave the recognition to a human end user In >3 dimensions this is not possible f.e. weight f.e. age

29. Clustering Techniques • Hierarchical algorithms • Agglomerative • Divisive • Partition based clustering • K-Means • Self Organizing Maps / Kohonen Networks • Probabilistic Model based • Expectation Maximization / Mixture Models

30. Hierarchical clustering • Agglomerative / Bottom up • Start with single-instance clusters • At each step, join the two closest clusters • Method to compute distance between cluster x and y: single linkage (distance between closest point in cluster x and y), average linkage (average distance between all points), complete linkage (distance between furthest points), centroid • Distance measure: Euclidean, Correlation etc. • Divisive / Top Down • Start with all data in one cluster • Split into two clusters based on distance measure / split utility • Proceed recursively on each subset • Both methods produce a dendrogram

31. Levels of Clustering Agglomerative Divisive Dunham, 2003

32. Hierarchical Clustering Example • Clustering Microarray Gene Expression Data • Gene expression measured using microarrays studied under variety of conditions • On budding yeast Saccharomyces cerevisiae • Groups together efficiently genes of known similar function, • Data taken from: Cluster analysis and display of genome-wide expression patterns. Eisen, M., Spellman, P., Brown, P., and Botstein, D. (1998). PNAS, 95:14863-14868; Picture generated with J-Express Pro

33. Hierarchical Clustering Example • Method • Genes are the instances, samples the attributes! • Agglomerative • Distance measure = correlation • Data taken from: Cluster analysis and display of genome-wide expression patterns. Eisen, M., Spellman, P., Brown, P., and Botstein, D. (1998). PNAS, 95:14863-14868; Picture generated with J-Express Pro

34. Simple Clustering: K-means • Pick a number (k) of cluster centers (at random) • Cluster centers are sometimes called codes, and the k codes a codebook • Assign every item to its nearest cluster center • F.i. Euclidean distance • Move each cluster center to the mean of its assigned items • Repeat until convergence • change in cluster assignments less than a threshold KDnuggets

35. K-means example, step 1 Y k1 k2 k3 X Initially distribute codes randomly in pattern space KDnuggets

36. K-means example, step 2 Y k1 k2 k3 X Assign each point to the closest code KDnuggets

37. K-means example, step3 Y k1 k2 k2 k1 k3 k3 X Move each code to the mean of all its assigned points KDnuggets

38. K-means example, step 2 Y k1 k2 k3 X Repeat the process – reassign the data points to the codes Q: Which points are reassigned? KDnuggets

39. K-means example Y k1 k3 k2 X Repeat the process – reassign the data points to the codes Q: Which points are reassigned? KDnuggets

40. K-means example Y k1 k3 k2 X re-compute cluster means KDnuggets

41. K-means example Y k2 k1 k3 X move cluster centers to cluster means KDnuggets

42. Advantages Simple, understandable items automatically assigned to clusters Disadvantages Must pick number of clusters before hand All items forced into a cluster Sensitive to outliers K-means clustering summary Extensions • Adaptive k-means • K-mediods (based on median instead of mean) • 1,2,3,4,100  average 22, median 3

43. Biological Example • Clustering of yeast cell images • Two clusters are found • Left cluster primarily cells with thick capsule, right cluster thin capsule • caused by media, proxy for sick vs healthy

44. Self Organizing Maps(Kohonen Maps) • Claim to fame • Simplified models of cortical maps in the brain • Things that are near in the outside world link to areas near in the cortex • For a variety of modalities: touch, motor, …. up to echolocation • Nice visualization • From a data mining perspective: • SOMs are simple extensions of k-means clustering • Codes are connected in a lattice • In each iteration codes neighboring winning code in the lattice are also allowed to move

45. SOM 10x10 SOM Gaussian Distribution

46. SOM

47. SOM

48. SOM

49. SOM example

50. Famous example:Phonetic Typewriter • SOM lattice below left is trained on spoken letters, after convergence codes are labeled • Creates a ‘phonotopic’ map • Spoken word creates a sequence of labels