Multivariate Statistics

Multivariate Statistics Matrix Algebra II W. M. van der Veld University of Amsterdam

Overview • The determinant of a matrix • The matrix inverse • System of equations

The determinant of a matrix • The determinant of a matrix is a scalar and is denoted as |A| or det(A). Det(A) only exists when A is a square matrix. • It has very important mathematical properties, but it is very difficult to provide a substantive definition. • The determinant is necessary to compute the inverse of a matrix (A-1). • The inverse of a matrix is needed for solving systems of linear equations; multivariate statistics often comes down to this. • When the determinant is zero, there exists no solution to a system of linear equations. • Let’s see how the value of the determinant is found.

Cofactors The determinant of a matrix • How to do it? The most simple case, a 2 by 2 matrix . • Det(A)=|A|=?

The determinant of a matrix • One step further, a 3 by 3 matrix. • Det(A)=|A|=? Cofactor

The determinant of a matrix • You should have noted that for matrices larger than first order, computation of the determinant is a recursive process. This process stops each time a 1 by 1 determinant is encountered, and involves multiplication by the cofactors.

The determinant of a matrix • Let A be a matrix of order n x n. If we omit one or more rows or columns from A, we obtain a matrix of smaller order, called a minor of the matrix. • Similarly, we have minors of a determinant, and in particular, if we omit from the determinant the ith row and the jth column, the resulting minor will be square and its determinant will be symbolized |Mij|. This determinant is called a cofactor (cij) if we give it a sign equal to (-1)i+j, so that: cij = (-1)i+j |Mij|. Using this notation we can write a formula for the expansion of a determinant of order n: In this version the determinant is expanded according to it’s ith row.

The determinant of a matrix • The following rules are important for determinants, and can help you sometimes to simplify calculations: • The determinant of A has the same value as the determinant of A’. • The value of the determinant changes sign if one row (column) is interchanged with another row (column). • If a determinant has two equal rows (columns), its value is zero. • If a determinant has two rows (columns) with proportional elements, its value is zero. • If all elements in a row (column) are multiplied by a constant, the value of the determinant is multiplied by that constant. • If a determinant has a row (column) in which all elements are zero, the value of the determinant is zero. • The value of the determinant remains unchanged if one row (column) is added to or subtracted form another row (column). Moreover, if a row (column) is multiplied by a constant and then added to or subtracted from another row (column) the value remains unchanged.

The determinant of a matrix • What is the determinant of:

The matrix inverse • Let A be a square matrix. If we can find a matrix B of the same order as A such that AB=BA=I, then B is said to be the inverse of A and is symbolized A-1. A-1, if exists, can be found as follows. • Let C be the matrix of cofactors of A (i.e., cij is the cofactor obtained from the minor |Mij|); then • Where C’ is the transpose of C (or if one prefers, C’ is the matrix of cofactors of A’). It is immediately seen that the inverse is undefined if A is not square (since then there is no determinant |A|), and also if |A| is equal to zero.

The matrix inverse • Illustration that AA-1 = A-1A = I.

Compute determinant The matrix inverse • How did I get A-1? Now Compute C C transpose => C’ Calculate A-1

The matrix inverse • Another way to calculate A-1. This way introduces you to solving systems of equations.

The matrix inverse • Rules for algebra with inverse matrices: • AA-1 = A-1A = I • (AB)-1 = B-1A-1 • (ABC)-1 = C-1B-1A-1 • Proof that(AB)-1 = B-1A-1.

System of equations • In the introduction I already mentioned that the basic linear equation y=bx will be very important for multivariate methods. • Here we will discuss how to solve systems of such linear equations.

System of equations • Illustration. Suppose we have the following set of equations:-3=1x1+4x2 1=3x1+2x2 • The basic way to think about this problem set is finding the intersection, i.e. for which unknowns are the equations satisfied. • This can be solved in a simple way (old style). • The solution is basically the intersection of the lines represented by the equation. • You won’t be surprised that there is a more general way to solve systems of linear equations, using matrix algebra.

System of equations • Solution for m equations with n unknowns: m=n. • What to do? Normally you divide by A so that you obtain a solution for x (give example: 15=3x). • Matrix division is defined as multiplication by the inverse, so:

System of equations • Example. Suppose we have the following set of equations:-3=1x1+4x21=3x1+2x2 • We already solved this one, resulting in x1=1 and x2=-1. • The set of equations can be written as a matrix operation.

System of equations • Thus, we have to find the inverse of: A => A-1 = C’/|A| • We have to take the transpose of C

System of equations • We have to divide by |A|. • Thus the inverse matrix is.

System of equations • Thus a solution for: -3=1x1+4x2 1=3x1+2x2 is found via

System of equations Exercise, solve: x1 + 2x2 = 0 3x1 + 7x2 = 1 • So if Ax = k solve via x = A-1k. • .... But it is not always so simple … A-1Ax = Ix = x = A-1k

System of equations • Sometimes, the requirement that m=n seems to be fulfilled, so that there should exist a solution. • But consider the following cases. (Row 2 = 2 x Row 1) (Row 3 = Row 1 + Row 2) (Column 3 = Column 1 + Column 2), etc.

System of equations • These situations are called linear dependence: • Given vectors: x1, x2,…, xn-1 • Another vector xn is linearly dependent if there exists constants α1, α2,…, αn-1 such that:xn= α1x1+α2x2+ …+αn-1xn-1 • Otherwise the vector xn is linearly independent. • In case of linear dependence; |A|= 0. • And then the inverse is not defined: A-1=C’/|A|. • And when the inverse is not defined we cannot find a solution via: A-1k=x.

System of equations • Generally a unique solution exists only if m=n, and |A|≠0 • When are there ‘problems’? • If m<n there are many solutions, the problem is underdetermined. 8x1+10x2+14x3=94x1+12x2+16x3=10 • if m>n there are no solutions, the problem is overdetermined.8x1+10x2=94x1+12x2=104x1+10x2=2

System of equations • Using the idea of linear dependency, the rank of a matrix can be introduced. • rank(A) = number of linearly independent rows or columns. • Given an mxn matrix, with m ≥ n, then if • |A| ≠ 0  rank(A) = n full rank, solvable • |A| = 0  rank(A) < n rank deficient • We will get back to the issue of rank.

Overdetermined Systems • Find Ax “closest” to k • Least-squares distance measure • Minimization problem: • Normal equations: (A’A)x = A’k • Solution: x = (A’A)-1A’k • A’A must be nonsingular; i.e. |A’A|≠0 • (A’A)-1A’ is called the left inverse matrix

Constrained minimization problem: Underdetermined Systems • Find “smallest” x that satisfies equations • Minimum norm objective • Solution: x = A’(AA’)-1k • AA’ must be nonsingular • A’(AA’)-1 is called the right inverse

Multivariate Statistics