Least-Squares Regression for Data Analysis

1. Chapter 10Real Inner Products and Least-Square (cont.) In this handout: Section 10.5: Least-Squares

2. y x A common problem in business, science, engineering is to collect data and analyze them to predict future events. If such data are plotted, they constitute a scatter diagram which may provide useful insight into the underlying relationship between system variables. The data below appears to follow a straight line relationship. The problem is to determine the equation of the straight line that best fits the data.

3. Consider an arbitrary straight line, y = b0 + b1 x , to be fitted through these data points. For each data point, the error is the difference between the y-value of the point and the y-value obtained from the straight line approximation.

4. The least-squares straight line Definition 1: The least-squares error E is the sum of the squares of the individual errors. That is, Definition 2: The least-squares straight line is the line that minimizes the least-squares error. We want to find the equation of the least-squares straight line: y = mx + c We seek values of m and c that minimizes the least-squares error.

5. The normal equations For each of N data points, the error is We want the values for m and c that minimize This occurs when Or, upon simplifying, when The last two equations are the normal equations for a least-squares fit in two variables. Examples on the board.

6. Matrix representation of the normal equations Ideally, we would like to choose m and c so that yi = mxi + c for all data pairs (xi, yi), i=1,2,…,N. That is, we want the values for m and c that solve the system or, equivalently, the matrix equation

7. Matrix representation of the normal equations This system has the standard form Ax=b where x = [m c], b = [y1 y2 … yN], and A has two columns [x1 x2 … xN] and [1 1 … 1]. Ax=b has a solution if and only if the data falls on a straight line. If not, then the system is inconsistent, and we seek a solution that minimizes the least-square error: The least-square solution which is given by the normal equations has the following matrix form

8. The least-squares solution for any linear system The concept of linear-squares can be generalized to any linear system Ax=b. • We are primarily interested in cases where the system is inconsistent. This generally occurs when A has more rows than columns. • Measurement errors are inevitable in observational and experimental sciences. • Errors can be smoothed out by averaging over many cases, i.e., taking more measurements than are strictly necessary to determine parameters of system. • Resulting system is overdetermined (more rows than columns), so usually there is no exact solution. • The least-squares is an approximate solution to this kind of systems.

9. The least-squares solution for any linear system We seek the vector that minimizes the least-squares error defined by Theorem 1: If x has the property that Ax-b is orthogonal to the columns of A, then x minimizes the least-squares error. As a consequence to Theorem 1, x is the least-squares solution to Ax=b if and only if x is the solution to This set of normal equations is guaranteed to have a unique solution whenever the columns of A are linearly independent. The solution can be found using the techniques of previous chapters.

10. The least-squares solution of a linear system (example) • Find the least squares solution of the linear system Ax = b given by x1 – x2 = 4 3x1 + 2x2 = 1 -2x1 + 4x2 = 3 • Solution:

11. The least-squares solution of a linear system (example) Solution (cont.): We have so the normal system ATAx = ATb in this case is Solving this system yields the least squares solution x1 = 17/95, x2 = 143/285