Extremum Properties of Orthogonal Quotients Matrices By Achiya Dax

1. Extremum Properties of Orthogonal Quotients Matrices By Achiya Dax Hydrological Service, Jerusalem , Israel e-mail: dax20@water.gov.il

2. The Eckart–Young Theorem (1936) says that the “truncated SVD” matrix Ak= UkDkVkT solves the least norm problem minimize F(B)=||A- B||F2 subject to rank(B) £ k ( Also called the Schmidt-Mirsky Theorem. )

3. Ky Fan’s Maximum Principle (1949) Sa symmetric positive semi-definite n x n matrix Ykthe set of orthogonal n x k matrices l1+…+lk=max{trace(YkTSYk )|YkÎYk} Solution is obtained for the Spectral matrix Vk=[v1 , v2 , … , vk]. giving the sum ofthe largest k eigenvalues.

4. Outline *TheEckart–Young Theorem and Orthogonal Quotients matrices. * The Orthogonal Quotients Equality. * The Symmetric Quotients Equality and Ky Fan’s extremum principles. *An Extended Extremum Principle.

5. The Singular Value Decomposition A=USVT S= diag {s1,s2,…,sp} , p = min{m,n} U = [u1 , u2 , … , up] , UTU = I V = [v1 , v2 , … , vp] , VTV = I AV = US ATU = VS Avj=sjuj , ATuj=sjvjj=1, … , p .

6. Low - Rank Approximations Ak= Uk Dk VkT Dk= diag {s1,s2,…,sk} , Uk= [u1 , u2 , … , uk] , UkT Uk= I Vk= [v1 , v2 , … , vk] , VkT Vk= I

7. The Eckart–Young Theorem (1936) says that the “truncated SVD” matrix Ak= UkDkVkT solves the least norm problem minimize F(B)=||A- B||F2 subject to rank(B) £ k ( Also called the Schmidt-Mirsky Theorem. )

8. Rank - k Matrices B= Xk Rk YkT where Rk is a k x k matrix Xk= [x1 , x2 , … , xk] , XkT Xk= I Yk= [y1 , y2 , … , yk] , YkT Yk=I

9. The Eckart –Young Problem can be rewritten as minimize F(B)=||A- XkRkYkT||F2 subject to XkTXk=I and YkTYk=I .

10. Theorem 1:Given a pair of orthogonal matrices, Xk and Yk , the related “OrthogonalQuotients Matrix” XkTAYk=( xiTAyj) solves the problem minimize F(Rk)=||A- XkRkYkT||F2

11. Notation: Xk - denotes the set of all real m x k orthogonal matrices Xk , Xk= [x1 , x2 , … , xk] , XkT Xk= I Yk-denotes the set of all real n x k orthogonal matrices Yk , Yk= [y1 , y2 , … , yk] , YkT Yk=I

12. Corollary 1 : The Eckart–Young Problem can be rewritten as minimize F(Xk,Yk)=||A- XkRkYkT||F2 subject to XkÎXkand YkÎYk , whereRkis the“OrthogonalQuotientsMatrix” Rk = XkTAYk .

13. The Orthogonal Quotients Equality For any pair of orthogonal matrices, XkÎXkand YkÎYk , ||A - Xk RkYkT||F2=||A||F2-||Rk||F2 where Rk is the orthogonal quotients matrix Rk = XkTAYk .

14. Corollary : The Eckart–Young Problem minimize F(Xk,Yk)=||A- XkRkYkT||F2 subject to XkÎXkand YkÎYk. is equivalent to maximize ||XkTAYk||F2 subject to XkÎXkand YkÎYk, and theSVDmatrices Uk andVksolves both problems, giving the optimal value of s12+s22+…+sk2 .

15. Question:Istherelatedminimumproblem minimize ||(Xk)TAYk||F2 subject to XkÎXkand YkÎYk, solvable ?

16. Using the Orthogonal Quotients Equality the last problem takes the form maximize F(Xk,Yk)=||A- XkRkYkT||F2 subject to XkÎXkand YkÎYk.

17. Note that the more general problem maximize F(B)=||A- B||F2 subject to rank(B) £ k is not solvable .

18. Returning to symmetric matrices How we extend the Orthogonal Quotients Equality tosymmetric matrices?

19. The Spectral Decomposition S = ( Sij)a symmetric positive semi-definite n x n matrix With eigenvalues l1 ³ l2 ³ ... ³ ln ³ 0 and eigenvectors v1 , v2 , … , vn S vj =ljvj, j = 1, … , n . S V = V D V = [v1 , v2 , … , vn] , VTV = V VT = I D = diag {l1,l2,…,ln} S = V D VT = SljvjvjT

20. Recall that Rayleigh Quotient Matrices Sk=YkTSYk play important role in Ky Fan’s Extremum Principles .

21. Ky Fan’s Maximum Principle Sa symmetric positive semi-definite n x n matrix Ykthe set of orthogonal n x k matrices l1+…+lk=max{trace(YkTSYk ) |YkÎYk} Solution is obtained for the Spectral matrix Vk=[v1 , v2 , … , vk]. which is related tothe largest k eigenvalues.

22. Ky Fan’s Minimum Principle Sa symmetric n x n matrix . Ykthe set of orthogonal n x k matrices . ln-k+1+…+ln=min{trace(YkTSYk)|YkÎYk} Solution is obtained for the Spectral matrix Vk=[vn-k+1 , … , vn] , which is related to the smallest k eigenvalues.

23. Question : Can we formulate the Symmetric Quotients Equality in terms of trace(Sk)=trace(YkTSYk)=SyjTSyj

24. The Symmetric Quotients Equality S a symmetric n x n matrix YkÎYk an orthogonal n x k matrix Sk=YkTSYkthe related“Rayleigh quotient matrix” trace(S-YkSkYkT)=trace(S)-trace(Sk)

25. Corollary 1: Ky Fan’s maximum problem maximize trace(YkSYkT ) subject to YkÎYk, is equivalent to minimize trace(S-YkSkYkT) subject to YkÎYk. The Spectral matrix Vk=[v1,v2, … ,vk] solves both problems, giving the optimal value of l1+…+lk.

26. Recall that: The Eckart–Young Problem minimize F(Xk,Yk)=||A- XkRkYkT||F2 subject to XkÎXkand YkÎYk. is equivalent to maximize ||XkTAYk||F2 subject to XkÎXkand YkÎYk, and the SVD matrices Uk andVksolves both problems, giving the optimal value of s12+s22+…+sk2 .

27. Corollary 2: Ky Fan’sminimum problem minimize trace(YkSYkT ) subject to YkÎYk, is equivalent to maximize trace(S-YkTSkYk) subject to YkÎYk. The matrix Vk=[vn-k+1 , … , vn] solves both problems, giving the optimal value of ln-k+1 + … + ln .

28. ExtendedExremumPrinciples Can we extend these extremums fromeigenvaluesofsymmetricmatrices tosingularvaluesof rectangularmatrices?

29. Notation: 1£m*£ m , 1£ n*£ n , Xm* - denotes the set of all real mxm* orthogonal matrices Xm* , Xm*= [x1 , x2 , … , xm*] , Xm*T Xm*= I Yn*-denotes the set of all real nxn* orthogonal matrices Yn* , Yn*= [y1 , y2 , … , yn*] , Yn*T Y*=I

30. Notations : Given Xm*ÎXm*andYn*ÎYm*, the m*xn* matrix (Xm*)TAYn* =( xiTAyj) is called “Orthogonal Quotients Matrix” .

31. Notations : Thesingular valuesof the Orthogonal Quotients Matrix (Xm*)TAYn* =( xiTAyj) are denoted as h1³h2³…³hk³ 0 , where k = min { m*, n* } .

32. Questions: Which choice of orthogonal matrices Xm*ÎXm*andYn*ÎYm*, maximizes (or minimizes ) the sum (h1)p+(h2)p+…+ (hk)p wherep >0is a given positive constant.

33. An Extended Maximum Principle: The SVD matrices Um*= [u1 , u2 , … , um*] and Vn*= [v1 , v2 , … , vn*] solve the problem maximize F(Xm*,Yn*)=(h1)p+(h2)p+…+ (hk)p subject toXm*ÎXm*and Yn*ÎYn*, for any positive power p > 0 , giving the optimal value of(s1)p+(s2)p+…+ (sk)p .

34. An Extended Maximum Principle That is, for any positive power p > 0 , (s1)p+(s2)p+…+ (sk)p= max{(h1)p+(h2)p+…+ (hk)p | Xm*ÎXm* and Yn*ÎYn*}, and the maximal value is attained for the matrices Um*=[u1,u2, … ,um*] and Vn*=[v1,v2 , … ,vn*] .

35. The proof is based on “rectangular” versions of CauchyInterlaceTheorem and Poincare Separation Theorem .

36. Corollary 1:When p=1 the SVD matrices Um*= [u1 , u2 , … , um*] and Vn*= [v1 , v2 , … , vn*] solve the “Rectangular Ky Fan problem” maximize F(Xm*,Yn*)= h1 +h2+…+hk subject toXm*ÎXm*and Yn*ÎYn*, giving the optimal value of s1 +s2+…+sk .

37. Corollary 2:When p=1 and m*=n*=k the SVD matrices Uk= [u1 , u2 , … , uk] and Vk= [v1 , v2 , … , vk] solve the maximum trace problem maximize F(Xk,Yk)= trace ( (Xk)TAYk) subject toXkÎXkand YkÎYk, giving the optimal value of s1 +s2+…+sk . * See also Horn & Johnson, “Topics in Matrix Analysis”, p. 195.

38. Corollary 3:When p=2 the SVD matrices Um*= [u1 , u2 , … , um*] and Vn*= [v1 , v2 , … , vn*] solve the “rectangularEckart–Youngproblem” maximize F(Xm*,Yn*)= ||(Xm*)TAYn*||F2 subject toXm*ÎXm*and Yn*ÎYn*, giving the optimal value of (s1)2+(s2)2+…+ (sk)2 .

39. AnExtendedMinimumPrinciple Question:Can we prove a similar minimumprinciple ? Answer: Yes,butthe solutionmatrices aremorecomplicated.

40. An Extended Minimum Principle: Here we consider the problem minimizeF(Xm*,Yn*)=(h1)p+(h2)p+…+ (hk)p subject toXm* ÎXm* and Yn* ÎYn*, for any positive power p>0 . The solution matrices are obtained by deleting some columns from the SVD matrices U = [u1 , u2 , … , um] and V = [v1 , v2 , … , vn] .

41. Summary *TheEckart–Young Theorem and Orthogonal Quotients matrices. * The Orthogonal Quotients Equality. * The Symmetric Quotients Equality and Ky Fan’s extremum principles. *An Extended Extremum Principle.

42. The END Thank You