Linear Methods for Regression

1. Linear Methods for Regression Dept. Computer Science & Engineering, Shanghai Jiao Tong University

2. Outline • The simple linear regression model • Multiple linear regression • Model selection and shrinkage —the state of the art Linear Methods for Regression

3. Preliminaries • Data • is the predictor (regressor, covariate, independent variable) • is the response (dependent variable, outcome) • We denote the regression function by • This is the conditional expectation of Y given x. • The linear regression model assumes a specific linear form for which is usually thought of as an approximation to the truth. Linear Methods for Regression

4. Fitting by least squares • Minimize: • Solutions are • ：the fitted or predicted values • are called the residuals Linear Methods for Regression

5. Gaussian Distribution • The normal distribution with arbitrary center μ, and variance σ2. Linear Methods for Regression

6. Standard errors & confidence intervals • We often assume further that • Under additional assumption of normality for the , a 95% confidence interval for is: Linear Methods for Regression

7. Fitted Line and Standard Errors Linear Methods for Regression

8. Fitted regression line with pointwise standard errors: Linear Methods for Regression

9. Multiple linear regression • Model is • Equivalently in matrix notation: • is N-vector of predicted values • is N × p matrix of regresses, with ones in the first column • is a p-vector of parameters Linear Methods for Regression

10. Estimation by least squares Linear Methods for Regression

11. Z Linear Methods for Regression

12. The Bias-variance tradeoff • A good measure of the quality of an estimator is the mean squared error. Let be the true value of f (x) at the point x. Then • This can be written as variance + bias^2. • Typically, when bias is low, variance is high and vice-versa. Choosing estimators often involves a tradeoff between bias and variance. Linear Methods for Regression

13. The Bias-variance tradeoff • If the linear model is correct for a given problem, then the least squares prediction f is unbiased, and has the lowest variance among all unbiased estimators that are linear functions of y. • But there can be (and often exist) biased estimators with smaller MSE. • Generally, by regularizing(shrinking, dampening, controlling) the estimator in some way, its variance will be reduced; if the corresponding increase in bias is small, this will be worthwhile. Linear Methods for Regression

14. The Bias-variance tradeoff • Examples of regularization: subset selection (forward, backward, all subsets); ridge regression, the lasso. • In reality models are almost never correct, so there is an additional model biasbetween the closest member of the linear model class and the truth. Linear Methods for Regression

15. Model Selection • Often we prefer a restricted estimate because of its reduced estimation variance. Linear Methods for Regression

16. Variable subset selection • We retain only a subset of the coefficients and set to zero the coefficients of the rest. • There are different strategies: • All subsets regressionfinds for each s 0, 1, 2, . . . p the subset of size s that gives smallest residual sum of squares. The question of how to choose s involves the tradeoff between bias and variance: can use cross-validation (see below) Linear Methods for Regression

17. Variable subset selection • Backward stepwise selectionstarts with the full OLS model, and sequentially deletes variables. • There are also hybrid stepwise selectionstrategies which add in the best variable and delete the least important variable, in a sequential manner. • Each procedure has one or more tuning parameters: – subset size – P-values for adding or dropping terms Linear Methods for Regression

18. Model Assessment • Objectives: 1. Choose a value of a tuning parameter for a technique 2. Estimate the prediction performance of a given model • For both of these purposes, the best approach is to run the procedure on an independent test set, if one is available • If possible one should use different test data for (1) and (2) above: a validation setfor (1) and a test setfor (2) • Often there is insufficient data to create a separate validation or test set. In this instance Cross-Validation is useful. Linear Methods for Regression

19. K-Fold Cross-Validation • Primary method for estimating a tuning parameter (such as subset size) • Divide the data into K roughly equal parts (typically K=5 or 10) Linear Methods for Regression

20. K-Fold Cross-Validation • for each k = 1, 2, . . .K, fit the model with parameter to the other K − 1 parts, giving and compute its error in predicting the kth part: This gives the cross-validation error • do this for many values of and choose the value of that makes smallest. Linear Methods for Regression

21. K-Fold Cross-Validation • In our variable subsets example, is the subset size • are the coefficients for the best subset of size , found from the training set that leaves out the k-th part of the data • is the estimated test error for this best subset. Linear Methods for Regression

22. K-Fold Cross-Validation • From the K cross-validation training sets, the K test error estimates are averaged to give: • Note that different subsets of size will (probably) be found from each of the K cross-validation training sets. Doesn’t matter: focus is on subset size, not the actual subset. Linear Methods for Regression

23. The Bootstrap approach • Bootstrap works by sampling N times with replacement from training set to form a “bootstrap” data set. Then model is estimated on bootstrap data set, and predictions are made for original training set. • This process is repeated many times and the results are averaged. • Bootstrap most useful for estimating standard errors of predictions. • Sometimes produces better estimates than cross-validation (topic for current research) Linear Methods for Regression

24. Some Key Points in the last talk • Error Decomposition • Model selection • Tradeoff between Var and Bias • Reducing Variance Approaches • Variable selection • Cross Validation / Bootstrap • Shrinkage Methods • …… Linear Methods for Regression

25. Shrinkage methods • Ridge regression • Preprocessing：Data centering • The ridge estimator is defined by Linear Methods for Regression

26. Shrinkage methods • Ridge regression • The ridge estimator is defined by • Equivalently, Linear Methods for Regression

27. Shrinkage methods • The parameter > 0 penalizes proportional to its size . Solution is • where I is the identity matrix, > 0 • Note = 0 gives the least squares estimator; if , then Linear Methods for Regression

28. Ridge regression • Ridge solution: • Singular value Decomposition: • For Ordinary Regression Linear Methods for Regression

29. Ridge regression • Ridge solution: • Singular value Decomposition: • For Ridge Regression Linear Methods for Regression

30. The Lasso • The lasso (least absolute shrinkage and selection operator)  is a shrinkage method like ridge, but acts in a nonlinear manner on the outcome y. • The lasso is defined by • No constraint on Linear Methods for Regression

31. The Lasso • Notice that ridge penalty is replaced by • This makes the solutions nonlinear in y, and a quadratic programming algorithm is used to compute them. • Because of the nature of the constraint, if t is chosen small enough then the lasso will set some coefficients exactly to zero. Thus the lasso does a kind of continuous model selection. Linear Methods for Regression

32. The Lasso • The parameter t should be adaptively chosen to minimize an estimate of expected, using say cross-validation • Ridge vs Lasso:if inputs are orthogonal, ridge multipliesleast squares coefficients by a constant < 1, lasso translatesthem towards zero by a constant, truncating at zero. Linear Methods for Regression

33. A family of shrinkage estimators • Consider the criterion • for q >=0. The contours of constant value of are shown for the case of two inputs. Linear Methods for Regression

34. Outline • The simple linear regression model • Multiple linear regression • Model selection and shrinkage —the state of the art • Principal component Regression • Partial least squares Regression Linear Methods for Regression

35. Use of derived input directions • Principal components regression • Choose a set of linear combinations of the s, and then regress the outcome on these linear combinations. • Principal components of the inputs • Uncorrelated and ordered by decreasing variance. • If is the sample covariance matrix of , then the eigenvector equations define the principal components of S. Linear Methods for Regression

36. Principal components of some input data points. The largest principal component is the direction that maximizes the variance of the projected data, and the smallest principal component minimizes that variance. • Ridge regression projects y onto these components, and then shrinks the coefficients of the low variance components more than the high-variance components. Linear Methods for Regression

37. PCA regression • Write for the ordered principal components, ordered from largest to smallest value of . • Then principal components regression computes the derived input columns and then regresses y on for some J<=p. Linear Methods for Regression

38. PCA regression • Since the s are orthogonal, this regression is just a sum of univariate regressions: • where is the univariate regression coefficient of y on . • Principal components regression is very similar to ridge regression: both operate on the principal components of the input matrix. Linear Methods for Regression

39. PCA regression • Ridge regression shrinks the coefficients of the principal components, with relatively more shrinkage applied to the smaller components than the larger; • Principal components regression discards the p-J+1 smallest eigenvalue components. Linear Methods for Regression

40. Partial least squares • This technique also constructs a set of linear combinations of the s for regression, but unlike principal components regression, it uses y (in addition to X) for this construction. • We assume that x is centered and begin by computing the univariate regression coefficient Linear Methods for Regression

41. Partial least squares • From this we construct the derived input • The first partial least squares direction. • The outcome y is regressed on , giving coefficient • Orthogonalize with respect to • Repeat the procedure Linear Methods for Regression

42. Linear Methods for Regression

43. Ridge vs PCR vs PLS vs Lasso • Recent study has shown that ridge and PCR outperform PLS in prediction, and they are simpler to understand. • Lasso outperforms ridge when there are a moderate number of sizable effects, rather than many small effects. It also produces more interpretable models. • These are still topics for ongoing research. Linear Methods for Regression

44. The End of Talk Linear Methods for Regression

45. Statistical Decision Theory • The expected prediction error • With the 0-1 loss, Linear Methods for Regression

46. Statistical Decision Theory • With the 0-1 loss, • or Linear Methods for Regression

47. Ex. 2.3 Derive equation (2.24). • The median distance from the origin to the closest data point is given by Linear Methods for Regression