Lecture 21 SVD and Latent Semantic Indexing and Dimensional Reduction

1. Lecture 21SVD and Latent Semantic Indexing and Dimensional Reduction Shang-Hua Teng

2. Singular Value Decomposition • where • u1 …ur are the r orthonormal vectors that are basis of C(A) and • v1 …vr are the r orthonormal vectors that are basis of C(AT )

3. Low Rank Approximation and Reduction

4. · · The Singular Value Decomposition 0 A U S VT = 0 m x n m x m m x n n x n 0 A U S VT 0 = m x n m x r r x r r x n

5. 0 A U S VT 0 = m x n m x r r x r r x n · · The Singular Value Reduction Ak Uk S VkT = m x n m x k k x k k x n

6. How Much Information Lost?

7. Distance between Two Matrices • Frobenius Norm of a matrix A. • Distance between two matrices A and B

8. How Much Information Lost?

9. Approximation Theorem • [Schmidt 1907; Eckart and Young 1939] Among all m by n matrices B of rank at most k, Ak is the one that minimizes

10. Application: Image Compression • Uncompressed m by n pixel image: m×n numbers • Rank k approximation of image: • k singular values • The first k columns of U (m-vectors) • The first k columns of V (n-vectors) • Total: k× (m + n + 1) numbers

11. Example: Yogi (Uncompressed) • Source: [Will] • Yogi: Rock photographed by Sojourner Mars mission. • 256 × 264 grayscale bitmap  256 × 264 matrix M • Pixel values  [0,1] • ~ 67584 numbers

12. Example: Yogi (Compressed) • M has 256 singular values • Rank 81 approximation of M: • 81 × (256 + 264 + 1) = ~ 42201 numbers

13. Example: Yogi (Both)

14. Patented by MIT Utilizes two dimentional, global grayscale images Face is mapped to numbers Create an image subspace(face space) which best discriminated between faces Can be used in properly lit and only frontal images Eigenface

15. The Face Database • Set of normalized face images • Used ORL Face DB

16. Two-dimensional Embedding

17. EigenFaces • Eigenface (PCA)

18. Latent Semantic Analysis (LSA) • Latent Semantic Indexing (LSI) • Principal Component Analysis (PCA)

19. Term-Document Matrix • Index each document (by human or by computer) • fij counts, frequencies, weights, etc • Each document can be regarded as a point in m dimensions

20. Document-Term Matrix • Index each document (by human or by computer) • fij counts, frequencies, weights, etc • Each document can be regarded as a point in n dimensions

21. Term Occurrence Matrix

22. c1 c2 c3 c4 c5 m1 m2 m3 m4 human 1 0 0 1 0 0 0 0 0 interface 1 0 1 0 0 0 0 0 0 computer 1 1 0 0 0 0 0 0 0 user 0 1 1 0 1 0 0 0 0 system 0 1 1 2 0 0 0 0 0 response 0 1 0 0 1 0 0 0 0 time 0 1 0 0 1 0 0 0 0 EPS 0 0 1 1 0 0 0 0 0 survey 0 1 0 0 0 0 0 0 1 trees 0 0 0 0 0 1 1 1 0 graph 0 0 0 0 0 0 1 1 1 minors 0 0 0 0 0 0 0 1 1

23. Another Example

24. Term Document Matrix

25. D T LSI using k=2… “applications& algorithms” LSI Factor 2 LSI Factor 1 “differentialequations” Each term’s coordinates specified in first K valuesof its row. Each doc’s coordinates specified in first K valuesof its column.

26. Positive Definite Matrices and Quadratic Shapes

27. Positive Definite Matrices and Quadratic Shapes For any m x n matrix A, all eigenvalues of AAT and ATA are non-negative Symmetric matrices that have positive eigenvalues are called Positive Definite matrices Symmetric matrices that have non-negative eigenvalues are called Positive semi-definite matrices