1 / 22

Singular Value Decomposition

Singular Value Decomposition. Jonathan P. Bernick Department of Computer Science Coastal Carolina University. Outline. Derivation Properties of the SVD Applications Research Directions. Matrix Decompositions.

burtonmary
Download Presentation

Singular Value Decomposition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Singular Value Decomposition Jonathan P. Bernick Department of Computer Science Coastal Carolina University

  2. Outline • Derivation • Properties of the SVD • Applications • Research Directions

  3. Matrix Decompositions • Definition: The factorization of a matrix M into two or more matrices M1, M2,…, Mn, such that M = M1M2…Mn. • Many decompositions exist… • QR Decomposition • LU Decomposition • LDU Decomposition • Etc. • One is special…

  4. Theorem One • [Will] For an m by n matrix A:nm and any orthonormal basis {a1,...,an} of n, define (1) si = ||Aai|| (2) Then…

  5. Theorem One (continued) Proof:

  6. Theorem Two • [Will] For an m by n matrix A, there is an orthonormal basis {a1,...,an} of n such that for all i j, Aai  Aaj = 0 • Proof: Since ATA is symmetric, the existence of {a1,...,an} is guaranteed by the Spectral Theorem. • Put Theorems One and Two together, and we obtain…

  7. Singular Value Decomposition • [Strang]: Any m by n matrix A may be factored such that A = UVT • U: m by m, orthogonal, columns are the eigenvectors of AAT • V: n by n, orthogonal, columns are the eigenvectors of ATA • : m by n, diagonal, r singular values are the square roots of the eigenvalues of both AAT and ATA

  8. SVD Example • From [Strang]:

  9. SVD Properties • U, V give us orthonormal bases for the subspaces of A: • 1st r columns of U:Column space of A • Last m - r columns of U: Left nullspace of A • 1st r columns of V: Row space of A • 1st n - r columns of V: Nullspace of A • IMPLICATION: Rank(A) = r

  10. Application: Pseudoinverse • Given y = Ax, x = A+y • For square A, A+ = A-1 • For any A… A+ = V-1UT • A+ is called the pseudoinverse of A. • x = A+y is the least-squares solution of y = Ax.

  11. Given an m by n matrix A:nm with singular values {s1,...,sr} and SVD A = UVT, define U = {u1| u2| ... |um} V = {v1| v2| ... |vn}T Then… Rank One Decomposition Amay be expressed as the sum ofr rank one matrices

  12. Matrix Approximation • Let A be an m by n matrix such that Rank(A) = r • If s1s2 ... sr are the singular values of A, then B, rank q approximation of A that minimizes ||A - B||F, is Proof: S. J. Leon, Linear Algebra with Applications, 5th Edition, p. 414 [Will]

  13. Application: Image Compression • Uncompressed m by n pixel image: m×n numbers • Rank q approximation of image: • q singular values • The first q columns of U (m-vectors) • The first q columns of V (n-vectors) • Total: q× (m + n + 1) numbers

  14. Example: Yogi (Uncompressed) • Source: [Will] • Yogi: Rock photographed by Sojourner Mars mission. • 256 × 264 grayscale bitmap  256 × 264 matrix M • Pixel values  [0,1] • ~ 67584 numbers

  15. Example: Yogi (Compressed) • M has 256 singular values • Rank 81 approximation of M: • 81 × (256 + 264 + 1) = ~ 42201 numbers

  16. Example: Yogi (Both)

  17. Application: Noise Filtering • Data compression: Image degraded to reduce size • Noise Filtering: Lower-rank approximation used to improve data. • Noise effects primarily manifest in terms corresponding to smaller singular values. • Setting these singular values to zero removes noise effects.

  18. Example: Microarrays • Source: [Holter] • Expression profiles for yeast cell cycle data from characteristic nodes (singular values). • 14 characteristic nodes • Left to right: Microarrays for 1, 2, 3, 4, 5, all characteristic nodes, respectively.

  19. Research Directions • Latent Semantic Indexing [Berry] • SVD used to approximate document retrieval matrices. • Pseudoinverse • Applications to bioinformatics via Support Vector Machines and microarrays.

  20. References • [Berry]: Michael W. Berry, et. al., “Using Linear Algebra for Intelligent Information Retrieval,” CS 94-270, Department of Computer Science, University of Tennessee, 1994. Submitted to SIAM Review. • [Holter]: Neal S. Holter, et. al., “Fundamental patterns underlying gene expression profiles: Simplicity from complexity,” Proc. Natl. Acad. Sci. USA, 10.1073/pnas. 150242097, 2000 (preprint). Available online at www.pnas.org/doi/10.1073/pnas.150242097

  21. References (continued) • [Strang]: Gilbert Strang, Linear Algebra and Its Applications, 3rd edition, Academic Press, Inc., New York, 1988. • [Will]: Todd Will, “Introduction to the Singular Value Decomposition,” Davidson College, http://www.davidson.edu/math/will/svd/index.html

  22. Full Presentation Text http://www.coastal.edu/~jbernick/ Comments Welcome!

More Related