ENGG2012B Lecture 6 Rank and Matrix multiplication

1. ENGG2012BLecture 6Rank and Matrix multiplication Kenneth Shum ENGG2012B

2. CALCULATION OF RANK BY RREF ENGG2012B

3. Last lecture Theorem: If one of the vectors v1, v2, …, vm is the zero vector, then v1, v2, …, vm are linearly dependent. Theorem: v1, v2, …, vm are linearly dependent if and only if one of v1, v2, …, vm is a linear combination of the others. Definition: Given a matrix M, the row-rank of M is defined as the maximum number of linearly independentrows in M. ENGG2012B

4. Calculation of rank by definition • We need to test all possible combinations of rows, and check linear independence in all cases. The circled sets of vectors are linear independent (1,1,0), (0,0,0) (0,0,0), (1,2,1), (3,4,1) (1,1,0) (1,1,0), (1,2,0) (1,1,0), (1,2,0), (3,4,1) (0,0,0) (1,1,0), (3,4,1) (1,1,0), (0,0,0), (1,2,1), (3,4,1) (0,0,0), (1,2,0) (1,1,0), (0,0,0), (3,4,1) (1,2,0) (0,0,0), (3,4,1) (1,1,0), (0,0,0), (1,2,0) Rank = 2 (3,4,1) (1,2,0), (3,4,1) ENGG2012B

5. Observation • If a matrix is given in reduced row echelon form, then the rank of this matrix is the same as the number of nonzero rows. Rank = 3 Rank = 2 ENGG2012B

6. A more practical method • Theorem: Elementary row operations does not alter the rank. • In other words, the rank of a matrix is invariant after any sequence of elementary row operations. • Method: Given a matrix M, apply elementary row operations and reduces it to rref. The rank of M is the number of non-zero rows. ENGG2012B

7. Another observation • Elementary row operations are reversible. • We can ``undo’’ all elementary row operations. • The reverse operation of an elementary row operation is also an elementary row operation. • The reverse operation of exchanging row i and row j is itself, i.e., of exchanging row i and row j. • The reverse operation of multiplying a row by a factor of k (for nonzero k), is multiplying the same row by 1/k. • The reverse operation of adding c times row i to row j is adding (-c) times row i to row j. ENGG2012B

8. Correctness of the method • To show that the rank of matrix is invariant under any elementary row operation, we only need to show that: Elementary row operations do not decrease the rank of a matrix. Elementary row operation Matrix A Matrix A’ Rank >= k Rank >= k There exist k rows in Awhich are linearly independent There exist k rows in A’which are linearly independent ENGG2012B

9. The first type of row operation Exchanging two rows merely changes the order of writing the rows. Whether a set of vectors is linear independent or not is independent of the order. Therefore the maximal number of independent rows does not change after we exchange a pair of rows. ENGG2012B

10. The second type of row operation Let A be an nm matrix. Let the i-th row of a matrix A be ri. • Suppose that the rank of A is at least k. • By using the first type of elementary row operations, we may assume without loss of generality that the first k rows r1, r2, …, rk are linearly independent. • We want to see the effect of multiplying the p-th row by a nonzero constant c. ENGG2012B

11. The second type of row operation Recall that linear independence means: a1r1+a2r2 +…+ akrk= 0 a1, a2, …, akare all zero. • Case 1: 1  p  k Replace rp by crp, and consider the linear combinationb1r1+b2r2 +…+ bp (crp)+ …+bkrk= 0 • b1r1+b2r2 +…+ (bpc) rp+ …+bkrk= 0 • b1= b2= … = bpc = … = bk = 0 (by linear independence of ri’s) • All bi’s are zero. Therefore, r1, r2 ,…, crp, …, rk are linearly independent. ENGG2012B

12. The second type of row operation • Case 2: p > k Rows r1, r2 ,…, rk have no change, and hence are linearly independent. In both cases, we can find k linearly independent rows after multiplying the p-th row by a nonzero constant c. This implies that the rank is at least k after an elementary row operation of the second type is applied. ENGG2012B

13. The third type of row operation • Suppose r1, r2 …, rkare linearly independent. • We add d times row 2 to row 1. • Want to check whether (r1+dr2), r2 , r3 , …,rkare linearly independent. Suppose f1(r1+dr2)+f2r2 +f3r3 +… …+ fkrk= 0, for some constants f1, f2, …, fk. Then, f1r1+(f1d+f2)r2 +f3r3 +… …+ fkrk= 0. But this is possible only if f1= (f1d+f2)= …= fk= 0. • All coefficients f1, f2 …, fk are 0 • (r1+dr2), r2 ,r3,rkare linearly independent. f2=0 ENGG2012B

14. Algorithm • M is any rectangular matrix. • We can calculate the rank of M by reducing it to rref. Mj in rref M1 M3 M M2 …. Rank of M= number of nonzero rows in Mj Rank of Mj = number of nonzero rows in Mj ENGG2012B

15. Summary • For mxn matrix, computing the row-rank of a matrix using the original definition requires checking 2m-1 combinations. This is not feasible even if the number of rows, m, is moderately large. • By reduction to RREF, the running times is of the order m3. This is polynomial time in m. • However, in Matlab, the computation of rank is done bysingular value decomposition (SVD),because SVD is more stable numerically. (type “doc rank” in Matlab for more details) ENGG2012B

16. Row-rank vs Column-rank • Row-rank : maximal number of linearly independent rows. • Column-rank: maximal number of linearly independent columns. • Column-rank can be calculated by elementary column operations. • An amazing fact is that the row-rank of a matrix is always equal to the column-rank. • The rank of a matrix can be defined either as the row-rank or the column-rank. ENGG2012B

17. Example: Row-rank = Column-rank The vectors (2,-1,3), (8,0,7) and (6,1,4) lie on a plane. The vectors (2,8,6), (-1,0,1) and (3,7,4) lie on another plane. Row-rank = 2 Column-rank = 2 The rows are linearly dependent (2,-1,3) = (8,0,7) - (6,1,4) The columns are linearly dependent 7*(2,8,6) = 10*(-1,0,1) +8*(3,7,4) ENGG2012B

18. MATRIX MULTIPLICATION ENGG2012B

19. Matrix multiplication is an action • Matrix-vector product is a function from a vector space to another vector space. Multiply by M v M v ENGG2012B

20. Review of function in mathematics • A function consists of • Domain: a set • Range: another set • An association between the elements. Range Domain f(x) x ENGG2012B

21. Tea Time at Coffee corner Domain Range Alice Bob Cathy Dianna Edward Tea set 1 Tea set 2 Tea set 3 Tea set 3 Tea set 4 Tea set 5 ENGG2012B

22. Example 1 Reflection • Domain: • Range: • Define ENGG2012B

23. Example 2 Rotation by 90 degrees • Domain: • Range: • Define ENGG2012B

24. Example 3 Projection • Domain: • Range: • Define No. ofinput varaibles No. of outputvariables ENGG2012B

25. Example 4 • Domain: • Range: • Define a function ENGG2012B

26. Cascading two functions Example: multiply by  3 Rotate 90 degrees and scale up by a factor of 3. ENGG2012B

27. Function composition • Can we compose the functions in example 3 and example 4 and do the computation in one step? multiply by multiply by multiply by ? ENGG2012B

28. More generally… • Repeat for matrices and multiply by multiply by multiply by ? ENGG2012B

29. Even more generally multiply by B u multiply by A w v A is m x n, B is n x p multiply by u w ? What goes in hereis the matrix product A B ENGG2012B

30. Main points • Matrix-vector multiplication is an action. • It is useful in computer graphics and geometry. • “Matrix times matrix” is function composition with domain and range in vector spaces. • The definition of the product of two matrices follows naturally from this perspective. ENGG2012B