1 / 11

Numerical Methods for Empirical Covariance Matrix Analysis

18.338 Course Project. Numerical Methods for Empirical Covariance Matrix Analysis. Miriam Huntley SEAS, Harvard University May 15, 2013. Real World Data. RMT. “When it comes to RMT in the real world, we know close to nothing.”. -Prof. Alan Edelman , last week.

beryl
Download Presentation

Numerical Methods for Empirical Covariance Matrix Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 18.338 Course Project Numerical Methods for Empirical Covariance Matrix Analysis Miriam Huntley SEAS, Harvard University May 15, 2013

  2. Real World Data RMT “When it comes to RMT in the real world, we know close to nothing.” -Prof. Alan Edelman, last week

  3. Who Cares about Covariance Matrices? • Basic assumption in many areas of data analysis: multivariate data • You get , want to find • can be a very bad estimator if finite • Current standard using PCA (=SVD): distinguish from null model • In RMT language: any eigenvalues which lie very far away from the distribution expected for a white Wishart matrix should be considered signal

  4. Who Cares about Covariance Matrices? Data from: AlizadehA, et al. (2000) Distinct types of diffuse large B-cell lymphoma identifedby gene expression profiling. Nature 403:503-511.

  5. Why adventure beyond white Wishart? • Null model not particularly sophisticated. Can we do better? • Noise with structure • Example: Financial data • What if there is no right edge? • Known , how many samples do we need before we recover it from empirical data?

  6. Approach: General MP Law • Data matrix where and define • Y entries are iid (real or complex) and • Let Hp be the spectral distribution of and assume Hpconverges weakly to H∞ • Let FPbe the spectral distribution of (empirical) and its Stieltjestransform • Then: nxp pxp nxp See: Silverstein, J. W. and Bai, Z. D. (1995). On the empirical distribution of eigenvalues of a class of large-dimensional random matrices. J. Multivariate Anal. 54, 2,175–192. El Karoui, N., Spectrum estimation for large dimensional covariance matrices using random matrix theory, Ann. Statist. 36 (2008), 2757–2790

  7. Numerical Solutions of General MP Single, True Covariance Matrix Empirical Spectral Distribution Discretize in z Numerically Solve True Covariance Matrix Spectral Distribution Live Demos…

  8. Inverse Solutions of General MP? Single, True Covariance Matrix Empirical Spectral Distribution Discretize in z Numerically Solve True Covariance Matrix Spectral Distribution

  9. Toy Example:Block Covariance Matrix ? Warning: Don’t try this at home

  10. Toy Example:Block Covariance Matrix

  11. Thanks!This was fun. • Colwell LJ, Qin Y, Manta A and Brenner MP (2013). Signal identification from Sample Covariance Matrices with Correlated Noise. Under Review • El Karoui, N., Spectrum estimation for large dimensional covariance matrices using random matrix theory, Ann. Statist. 36 (2008), 2757–2790 • MARCENKO , V. A. and PASTUR, L. A. (1967). Distribution of eigenvalues in certain sets of random matrices. Mat. Sb. (N.S.) 72 507–536. • Silverstein, J. W. and Bai, Z. D. (1995). On the empirical distribution of eigenvalues of a class of large-dimensional random matrices. J. Multivariate Anal. 54, 2,175–192.

More Related