550 likes | 637 Views
Introduction to Matrices Douglas N. Greve dgreve@mgh.harvard.edu. Why Matrices?. Simplifies notation Simplifies concepts Grounding for general linear model (GLM) Simplifies implementation (eg, matlab, octave) Optimization. 2. 4. y. p. 3.3i. -7.4. What is the matrix?.
E N D
Introduction to Matrices Douglas N. Greve dgreve@mgh.harvard.edu
Why Matrices? • Simplifies notation • Simplifies concepts • Grounding for general linear model (GLM) • Simplifies implementation (eg, matlab, octave) • Optimization
2 4 y p 3.3i -7.4 What is the matrix? • A matrix is a table of numbers, like a spread sheet • Each entry is called an “element” • There is a value in each element (no empty cells) • Elements can be whole numbers, positive, negative, fractions, real, imaginary, symbolic variables M M = M
Parts of Matrices • Size/Dimensions • Elements • Groupings of Elements • Diagonals • Triangles • Vectors
2 4 y p 3.3i -7.4 Matrix Size/Dimension • Number of Rows • Number of Columns • Rows-by-Columns (RowsxCols) 2 4 y 1x3 2x3 2 p p 2x1 1x1
Scalar versus Matrix • Scalar is just a number or value • No dimension or size g = -1 No brackets No dimension Not a 1x1 g = [-1] 1x1
2 4 y p 3.3i -7.4 Matrix Element • Indicated by its row and column number • Scalar value M(row,col) M = M(i,j) 2x3 Mij (1,2) (1,1) (1,3) M(2,3) = -7.4 (2,3) (2,1) (2,2)
Diagonals 2 4 6 1 3 5 Row = Column + Offset 7 8 9 3x3 2 4 6 1 3 5 Row = Column - 2 (Second Upper Diagonal) } Row = Column - 1 (First Upper Diagonal) 7 8 9 Row = Column + 0 (Main Diagonal) Row = Column - 1 (First Lower Diagonal) } Off-Diagonals Row = Column - 2 (Second Lower Diagonal)
Triangles 2 4 6 1 3 5 Upper Triangle 7 8 9 3x3 Lower Triangle
Vector • Special Type of Matrix • Number of Rows or Columns = 1 • Often given lower-case variable names 2 4 y 1x3 Row Vector 2 p 2x1 Column Vector
2 2 2 4 4 4 x y x p p p 3.3i 3.3i 3.3i -7.4 -7.4 -7.4 A matrix consists of vectors 2x3 Two 1x3 row vectors Three 2x1 column vectors
Special Matrices • Square • Diagonal • Identity • Triangular • Symmetric • Toeplitz
Square Matrix • Rows = Columns 2 4 p 1x1 p 3.3i 2x2
Diagonal Matrix • All off-diagonal elements = 0 • M(i,j)=0 if i != j 2 0 0 0 3 0 0 0 9 3x3
Identity Matrix • Main diagonal has all 1s • All off-diagonal elements = 0 • Square • Symbol “I” 1 0 0 I= 0 1 0 0 0 1 3x3 • Any matrix multiplied by the identity is itself
2 0 0 1 3 0 2 4 6 7 8 9 0 3 5 3x3 0 0 9 3x3 Triangular Matrices Upper Triangular – all values below the main diagonal are 0 Lower Triangular – all values above the main diagonal are 0
Symmetric Matrix • M(row,col) = M(col,row) • Reflect across main diagonal • Square 2 4 6 4 3 5 6 5 9 3x3
Toeplitz Matrix All elements on a diagonal are the same 2 4 6 2 4 1 2 4 1 2 7 1 2 3x3 7 1 3x2 A causal, time invariant linear system is represented by an upper triangular Toeplitz matrix
Ones Matrix • All elements are 1 • Any number of rows or columns 1 1 1 1 1 1 1x3 1 1 1 2x3 1 1 1 2x1 1x1
Zeros Matrix • All elements are 0 • Any number of rows or columns 0 0 0 0 0 0 1x3 0 0 0 2x3 0 0 0 2x1 1x1
Matrix Operations • Addition/Subtraction • Vector Multiplication • Matrix Multiplication • Multiplication by scalar • Transpose • Trace • Inverse • Pseudo Inverse
Matrix Addition/Subtraction • Element-by-Element Addition/Subtraction • Size of two matrices must be the same • C(row,col) = A(row,col) +/- B(row,col) C = A + B (your first matrix equation!) 2 4 6 9 0 7 A = B = 1 3 5 1 -2 3.3 2x3 2x3 2 11 15 C = 2 1 8.3 2x3
0 1 7 Vector Multiplication • One Row Vector and One Column Vector • Same length, Results in 1x1 • Inner Product, Dot Product, “Scalar” Product • Like a correlation • “Multiply and accumulate” z = x.y= x*y = S xi*yi 2 4 6 x = 1x3 2 4 6 * * * 0 1 7 0+4+42 = 46 y = 3x1
Matrix Multiplication • Series of vector multiplications • Same “inner dimension” 0 2 8 4 2 4 6 A = 1 6 2 5 B = 1 3 5 3 9 3 7 2x3 3x4 c11 c12 c13 c14 cij=Arowi*Bcolj C=A*B = c21 c22 c23 c24 2x4 Dimensions: (2x3) *(3x4)=(2x4)
0 2 4 6 1 3 Matrix Multiplication • Series of vector multiplications • Same “inner dimension” 0 2 8 4 2 4 6 A = 1 6 2 5 B = 1 3 5 3 9 3 7 2x3 3x4 c11 c12 c13 c14 c11= C=A*B = c21 c22 c23 c24 2x4 2*0+4*1+6*3= 22
4 2 4 6 5 7 Matrix Multiplication • Series of vector multiplications • Same “inner dimension” 0 2 8 4 2 4 6 A = 1 6 2 5 B = 1 3 5 3 9 3 7 2x3 3x4 22 c12 c13 c14 c12= C=A*B = c21 c22 c23 c24 2x4 2*4+4*5+6*7= 70
2 2 4 6 6 9 Matrix Multiplication • Series of vector multiplications • Same “inner dimension” 0 2 8 4 2 4 6 A = 1 6 2 5 B = 1 3 5 3 9 3 7 2x3 3x4 22 70 c14 c13 c13= C=A*B = c21 c22 c23 c24 2x4 2*2+4*6+6*9= 82
8 2 4 6 2 3 Matrix Multiplication • Series of vector multiplications • Same “inner dimension” 0 2 8 4 2 4 6 A = 1 6 2 5 B = 1 3 5 3 9 3 7 2x3 3x4 22 70 c14 82 c14= C=A*B = c21 c22 c23 c24 2x4 2*8+4*2+6*3= 42
0 1 3 5 1 3 Matrix Multiplication • Series of vector multiplications • Same “inner dimension” 0 2 8 4 2 4 6 A = 1 6 2 5 B = 1 3 5 3 9 3 7 2x3 3x4 22 70 42 82 c21= C=A*B = c21 c22 c23 c24 2x4 1*0+4*1+5*3= 18
4 1 3 5 5 7 Matrix Multiplication • Series of vector multiplications • Same “inner dimension” 0 2 8 4 2 4 6 A = 1 6 2 5 B = 1 3 5 3 9 3 7 2x3 3x4 22 70 42 82 c22= C=A*B = 18 c22 c23 c24 2x4 1*4+4*5+5*7= 54
Outer Product (Vector Multiplication) 0 0 1 1 7 7 • One Row Vector one Column Vector • Same length (N) • Result is NxN matrix • Special case of Matrix Multiplication z = y*x 2 4 6 2 4 6 x = 1x3 1x3 0 0 0 y = 2 4 6 14 28 42 3x1 3x3 3x1
Multiplication with a scalar • Multiply each element by scalar value 2 4 6 g = -1 A = No brackets No dimension Not a 1x1 1 3 5 2x3 -2 -4 -6 gA = -1 -3 -5 2x3
Matrix Transpose • “Reflect” across diagonal • B(row,col) = A(col,row) 2 1 2 4 6 4 3 A = B=AT =At =A’= 1 3 5 6 5 2x3 3x2 If A=A’, then A is symmetric (must be square)
Trace Sum of diagonal Elements Trace = S M(i,i) 2 4 6 1 3 5 M = 7 8 9 Trace = 2 + 3 + 9 = 14
Determinant • Property of a square matrix • Complicated in general • Simple for a 2x2 c11 c12 D = c11*c22 - c12* c21 C = c21 c22 2x2
1 D Matrix Inverse • C*A = I, then A=C-1 andC=A-1 • Must be square • Complicated in general • Simple for a 2x2 c11 c12 c22 -c12 C = C-1 = c21 c22 -c21 c11 2x2 2x2 D = c11*c22 - c12* c21
1.0 2.0 1.0 -2.0 C = 0.5 1.0 C-1 = -0.5 1.0 2x2 2x2 1 0 Invertibility IMPORTANT!!! • Not all matrices are invertible • D=0 • “Singular” D = 1.0*1.0 - 2.0*0.5 = 1-1 = 0
Singularity and “Ill-Conditioned” 1.0 2.0 C = 0.5 1.0 2x2 • Column 2 = twice Column 1 • Linear Dependence Ill-Conditioned: D is “close” to 0 Relates to efficiency of a GLM. D = 1.0*1.0 - 2.0*0.5 = 1-1 = 0
Pseudo-Inverse • Can be used for non-square • Same rules on invertibility apply • Least-mean-square (LMS) for over-determined system If X has more columns than rows X+ = (X’*X)-1*X’ Check: X+*X = (X’*X)-1*X’*X=I If X has more rows than columns X+ = X’ *(X*X’)-1* Check: X*X+ = X*X’ *(X*X’)-1=I
1 1 x = y = 1 2 3 1 3x1 3x1 Sum/Length/Average of a List x’ = [1 1 1]1x3 “Ones vector” sum(y) = x’*y = 1*1 + 1*2 + 1*3 = 6 1x3 3x1 length(y) = x’*x = 1*1 + 1*1 + 1*1 = 3 Averagem = (x’*x)-1(x’*y) [Note: GLM] = (3-1)*6 = 6/3 = 2
1 0 x = y = 1 2 3 1 3x1 3x1 Selective Average x’ = [0 1 1]1x3 “Ones vector” sum(y) = x’*y = 0*1 + 1*2 + 1*3 = 5 1x3 3x1 length(y) = x’*x = 0*0+ 1*1 + 1*1 = 2 Averagem = (x’*x)-1(x’*y) = (2-1)*5 = 5/2 = 2.5
1 y = 2 3 3x1 Sum of the Squares of a List y’ = [1 2 3]1x3 SS =S(y2) SS(y) = y’*y = 1*1 + 2*2 + 3*3 = 14
1 1 y = x = 2 1 3 1 3x1 3x1 Variance of a List s2 =S(yi-m)2 (N-1) Residual e = y – m*x s2 = sqrt(e’*e)/DOF DOF = x’*x - 1 = #RowsX - #ColsX
2 3 .2 Orthogonality • Vector product is 0 • Property of two vectors • Uncorrelated/Independence - statistical property z = x*y = S xi*yi 1 -1 5 x = 1x3 1 * 2 + -1 * 3 + 5 * .2 = [0] y = 1x1 3x1
y1 y2 x1 x2 Fitting a Line Thickness Dependent Variable, Measurement Subject 1 Subject 2 HRF Amplitude IQ, Height, Weight Age Independent Variable
m= (y2-y1) (x2-x1) Intercept: b y1 Slope: m y2 x1 x2 Age b= (x2*y1-x1*y2) (x2-x1) Linear Model System of Linear Equations y1 = 1*b + x1*m y2 = 1*b + x2*m Two Eqns, Two Unknowns • Solve 1st Eq for b • Substitute into 2nd Eq • Solve 2nd equation for m • Substitute m back into Eq for b “Intercept” = “Offset”
Intercept: b y1 Slope: m b m y1 y2 1 x1 1 x2 y2 = * x1 x2 Age Matrix Formulation/GLM y1 = 1*b + x1*m y2 = 1*b + x2*m 2x1 2x1 2x2 y = X*b • - parameter vector X - design matrix
Intercept: b y1 Slope: m b m y1 y2 1 x1 1 x2 y2 = * x1 x2 Age Matrix Formulation/GLM y = X*b 1 x1 • # of Cols of X = # Parameters • Each parameter has a column • Each column means something • Parameter meaning from column X = 1 x2
Intercept: b y1 Slope: m b m y1 y2 1 x1 1 x2 y2 = * x1 x2 Age GLM Solution y = X*b Multiply both sides by X-1 b=X-1*y 1 x1 x2 -x1 1 D X-1 = X = 1 x2 -1 1 • = x2-x1 Non-invertible if x1=x2 Ill-conditioned if x1 near x2 -- Sensitive to noise