Download
1 / 33
330 likes | 536 Views
Bios 760R, Lecture 1 Overview. Overview of the course Classification The “ curse of dimensionality ” Reminder of some background knowledge. Tianwei Yu Room 334 Tianwei.yu@emory.edu. Overview. Focus of the course: Classification Clustering Dimension reduction. Overview.
E N D
Bios 760R, Lecture 1Overview • Overview of the course • Classification • The “curse of dimensionality” • Reminder of some background knowledge Tianwei Yu Room 334 Tianwei.yu@emory.edu
Overview Focus of the course: Classification Clustering Dimension reduction
Overview References: Textbook: The elements of statistical learning. Hastie, Tibshirani & Friedman. Other references: Pattern classification. Duda, Hart & Stork. Data clustering: theory, algorithms and application. Gan, Ma & Wu. Applied multivariate statistical analysis. Johnson & Wichern. Evaluation: Three projects (30% each) Class participation (10%)
Overview • Classification • Estimation • Prediction Supervised learning ”direct data mining” Machine Learning /Data mining • Clustering • Association rules • Description, dimension reduction and visualization Unsupervised learning ”indirect data mining” Semi-supervised learning Modified from Figure 1.1 from <Data Clustering> by Gan, Ma and Wu
Overview In supervised learning, the problem is well-defined: Given a set of observations {xi, yi}, estimate the density Pr(Y, X) Usually the goal is to find the model/parameters to minimize a loss, A common loss is Expected Prediction Error: It is minimized at Objective criteria exists to measure the success of a supervised learning mechanism. 5
Overview In unsupervised learning, there is no output variable, all we observe is a set {xi}. The goal is to infer Pr(X) and/or some of its properties. When the dimension is low, nonparametric density estimation is possible; When the dimension is high, may need to find simple properties without density estimation, or apply strong assumptions to estimate the density. There is no objective criteria to evaluate the outcome; Heuristic arguments are used to motivate the methods; Reasonable explanation of the outcome is expected from the subjective field of study. 6
Classification The general scheme. An example.
Classification In most cases, a single feature is not enough to generate a good classifier.
Classification Two extremes: overly rigid and overly flexible classifiers.
Classification Goal: an optimal trade-off between model simplicity and training set performance. This is similar to the AIC / BIC / …… model selection in regression.
Classification An example of the overall scheme involving classification:
Classification A classification project: a systematic view.
Curse of Dimensionality Bellman R.E., 1961. In p-dimensions, to get a hypercube with volume r, the edge length needed is r1/p. In 10 dimensions, to capture 1% of the data to get a local average, we need 63% of the range of each input variable.
Curse of Dimensionality In other words, To get a “dense” sample, if we need N=100 samples in 1 dimension, then we need N=10010 samples in 10 dimensions. In high-dimension, the data is always sparse and do not support density estimation. More data points are closer to the boundary, rather than to any other data point prediction is much harder near the edge of the training sample.
Curse of Dimensionality Estimating a 1D density with 40 data points. Standard normal distribution.
Curse of Dimensionality Estimating a 2D density with 40 data points. 2D normal distribution; zero mean; variance matrix is identity matrix.
Curse of Dimensionality Another example – the EPE of the nearest neighbor predictor. Tofind E(Y|X=x), take the average of data points close to a given x, i.e. the top k nearest neighbors of x Assumes f(x) is well-approximated by a locally constant function When N is large, the neighborhood is small, the prediction is accurate.
Curse of Dimensionality Just a reminder, the expected prediction error contains variance and bias components. Under model: Y=f(X)+ε
Curse of Dimensionality We have talked about the curse of dimensionality in the sense of density estimation. In a classification problem, we do not necessarily need density estimation. Generative model --- care about class density function. Discriminative model --- care about boundary. Example: Classifying belt fish and carp. Looking at the length/width ratio is enough. Why should we care how many teeth each kind of fish have, or what shape fins they have?
High-throughput biological data We talk about “curse of dimensionality” when N is not >>>p. In bioinformatics, usually N<100, and p>1000. How to deal with this N<<p issue? It is assumed most predictors do not contain information to predict Y. Dramatically reduce p before model-building. Filter genes based on: variation. normal/disease test statistic. projection. functional groups/network modules. …… For the most part of this course, we will pretend the N<<p issue doesn’t exist. Some methods are robust against large numbers of nuisance variables.
Reminder of some results for random vectors The multivariate Gaussian distribution: The covariance matrix, not limited to Gaussian. * Gaussian is fully defined by mean vector and covariance matrix (first and second moments).
Reminder of some results for random vectors The correlation matrix: Relationship with covariance matrix:
Reminder of some results for random vectors Quadratic form: A linear combination of the elements of a random vector: 2-D example:
Reminder of some results for random vectors A “new” random vector generated from linear combinations of a random vector:
Reminder of some results for random vectors Let A be a kxk square symmetrix matrix, then it has k pairs of eigenvalues and eigenvectors. A can be decomposed as: Positive-definite matrix:
Reminder of some results for random vectors Inverse of a positive-definite matrix: Square root matrix of a positive-definite matrix:
Reminder of some results for random vectors Proof of the first (and second) point of the previous slide.
Reminder of some results for random vectors With a sample of the random vector:
Reminder of some results for random vectors To estimate mean vector and covariance matrix: 30
