450 likes | 976 Views
Covariance Matrix Applications. Dimensionality Reduction. Outline. What is the covariance matrix? Example Properties of the covariance matrix Spectral Decomposition Principal Component Analysis. Covariance Matrix.
E N D
Covariance Matrix Applications Dimensionality Reduction
Outline • What is the covariance matrix? • Example • Properties of the covariance matrix • Spectral Decomposition • Principal Component Analysis
Covariance Matrix • Covariance matrix captures the variance and linear correlation in multivariate/multidimensional data. • If data is an N x D matrix, the Covariance Matrix is a d x d square matrix • .Think of N as the number of data instances (rows) and D the number of attributes (columns).
Covariance Formula • Let Data = N x D matrix. • The Cov(Data)
Example COV(R)
Dimensionality Reduction • If you work in “data analytics” it is common these days to be handed a data set which has lots of variables (dimensions). • The information in these variables is often redundant – there are only a few sources of genuine information. • Question: How can be identify these sources automatically?
Hidden Sources of Variance X1 X2 H1 X3 H2 X4 Model: Hidden Sources are Linear Combinations of Original Variables
Hidden Sources • If the information that the known variables provided was different then the covariance matrix between the variables should be a diagonal matrix – i.e, the non-zero entries only appear on the diagonal. • In particular, if Hi and Hj are independent then E(Hi-i)(Hj-j)=0.
Hidden Sources • So the question is what should be the hidden sources. • It turns out that the “best” hidden sources are the eigenvectors of the covariance matrix. • If A is a d x d matrix, then <, x> is an eigenvalue-eigenvector pair if • Ax = x
Explanation a We have two axis, X1 and X2. We want to project the data along the direction of maximum variance.
Covariance Matrix Properties • The Covariance matrix is symmetric. • Non-negative eigenvalues. • 0 ·1·2d • Corresponding eigenvectors • u1,u2,,ud
Principal Component Analysis • Also known as • Singular Value Decomposition • Latent Semantic Indexing • Technique for data reduction. Essentially reduce the number of columns while losing minimal information • Also think in terms of lossy compression.
Motivation • Bulk of data has a time component • For example, retail transactions, stock prices • Data set can be organized as N x M table • N customers and the price of the calls they made in 365 days • M << N
Objective • Compress the data matrix X into Xc, such that • The compression ratio is high and the average error between the original and the compressed matrix is low • N could be in the order of millions and M in the order of hundreds
Decision Support Queries • What was the amount of sales to GHI on July 11? • Find the total sales to business customers for the week ending July 12th?
Intuition behind SVD y x’ y’ x Customer are 2-D points
SVD Definition • An N x M matrix X can be expressed as Lambda is a diagonal r x r matrix.
SVD Definition • More importantly X can be written as Where the eigenvalues are in decreasing order. k,<r
Compression Where k <=r <= M