Chapter 11 Multiple Linear Regression

1. Chapter 11Multiple Linear Regression Group Project AMS 572

2. Group Members Yuanhua Li From Left to Right: Yongjun Cheng, William Ho, Katy Sharpe, Renyuan Luo, Farahnaz Maroof, Shuqiang Wang, Cai Rong, Jianping Zhang, Lingling Wu.

3. Overview • 1-3 Multiple Linear Regression --William Ho • 4 Statistical Inference ---Katy Sharpe & Farahnaz Maroof • 6 Topics in Regression Modeling -- Renyuan Luo & Yongjun Cheng • 7 Variable Selection Methods & SAS ---Lingling Wu, Yuanhua Li & Shuqiang Wang • 5, 8 Regression Diagnostic and Strategy for Building a Model ---Cai Rong • Summary Jianping Zhang

4. 11.1-11.3Multiple Linear Regression Intro William Ho

5. Regression analysis is a statistical methodology to estimate the relationship of a response variable to a set of predictor variables • Multiple linear regression extends simple linear regression model to the case of two or more predictor variable Multiple Linear Regression Example: A multiple regression analysis might show us that the demand of a product varies directly with the change in demographic characteristics (age, income) of a market area. Historical Background • Francis Galton started using the term regression in his biology research • Karl Pearson and Udny Yule extended Galton’s work to statistical context • Gauss developed the method of least squares used in regression analysis

6. Probabilistic Model is the observed value of the r.v. which depends on fixed predictor values according to the following model: where are unknown parameters. the random error, , are assumed to be independent r.v.’s then the are independent r.v.’s with

7. Fitting the model • The least squares (LS) method is used to find a line that fits the equation • Specifically, LS provides estimates of the unknown parameters, which minimizes, , the sum of difference of the observed values, , and the corresponding points on the line • The LS can be found by taking partial derivatives of Q with respect to unknown variables and setting them equal to 0. The result is a set of simultaneous linear equations, usually solved by computer • The resulting solutions, are the least squares (LS) estimates of , respectively

8. Goodness of Fit of the Model • To evaluate the goodness of fit of the LS model, we use the residuals • defined by • are the fitted values: • An overall measure of the goodness of fit is the error sum of squares (SSE) • A few other definition similar to those in simple linear regression: • total sum of squares (SST)- regression sum of squares (SSR) –

9. coefficient of multiple determination: • values closer to 1 represent better fits • adding predictor variables never decreases and generally increases • multiple correlation coefficient (positive square root of ): • only positive square root is used • r is a measure of the strength of the association between the predictor variables and the one response variable

10. Multiple Regression Model in Matrix Notation • The multiple regression model can be presented in a compact form by using matrix notation Let: be the n x 1 vectors of the r.v.’s , their observed values , and random errors , respectively Let: be the n x (k + 1) matrix of the values of predictor variables (the first column corresponds to the constant term )

11. Let: and • be the (k + 1) x 1 vectors of unknown parameters and their LS estimates, respectively • The model can be rewritten as: • The simultaneous linear equations whose solutions yields the LS estimates: • If the inverse of the matrix exists, then the solution is given by:

12. 11.4Statistical Inference Katy Sharpe & Farahnaz Maroof

13. Determining the statistical significance of the predictor variables: • We test the hypotheses: vs. • If we can’t reject , then the corresponding variable is not a useful predictor of y. • Recall that each is normally distributed with mean and variance , where is the jth diagonal entry of the matrix

14. Deriving a pivotal quantity • Note that • Since the error variance is unknown, we employ its unbiased estimate, which is given by • We know that , and that and the are statistically independent. • Using and , and the , definition of the t-distribution we obtain the pivotal quantity

15. Derivation of a Confidence Interval for Note that

16. Deriving the Hypothesis Test: Hypotheses: P (Reject H0| H0 is true) =     Therefore, we reject H0 if

17. Another Hypothesis Test for all Now consider: for at least one When H0is true, • F is our pivotal quantity for this test. • Compute the p-value of the test. • Compare p to , and reject H0 if p . • If we reject H0, we know that at least one j  0, • and we refer to the previous test in this case.

18. The General Hypothesis Test • Consider the full model: (i=1,2,…n) • Now consider partial model: (i=1,2,…n) vs. • Hypotheses: for at least one • We test: • Reject H0 when

19. Predicting Future Observations • Let and let • Our pivotal quantity becomes • Using this pivotal quantity, we can derive a CI to estimate*: • Additionally, we can derive a prediction interval (PI) to predictY*:

20. 11.6.1-11.6.3Topics in Regression Modeling Renyuan Luo

21. 11.6.1 Multicollinearity • Def. The predictor variables are linearly dependent. • This can cause serious numerical and statistical difficulties in fitting the regression model unless “extra” predictor variables are deleted.

22. How does the multicollinearity cause difficulties? • If the approximate multicollinearity happens: • is nearly singular, which makes numerically unstable. This reflected in large changes in their magnitudes with small changes in data. • The matrix has very large elements. Therefore are large, which makes statistically nonsignificant.

23. Measures of Multicollinearity • The correlation matrix R.Easy but can’t reflect linear relationships between more than two variables. • Determinant of R can be used as measurement of singularity of . • Variance Inflation Factors (VIF): the diagonal elements of . VIF>10 is regarded as unacceptable.

24. 11.6.2 Polynomial Regression A special case of a linear model: Problems: • The powers of x, i.e., tend to be highly correlated. • If k is large, the magnitudes of these powers tend to vary over a rather wide range. So let k<=3 if possible, and never use k>5.

25. Solutions • Centering the x-variable:Effect: removing the non-essential multicollinearity in the data. • Further more, we can do standardizing: divided by the standard deviation of x.Effect: helping to alleviate the second problem.

26. 11.6.3 Dummy Predictor Variables What to do with the categorical predictor variables? • If we have categories of an ordinal variable, such as the prognosis of a patient (poor, average, good), just assign numerical scores to the categories. (poor=1, average=2, good=3)

27. If we have nominal variable with c>=2 categories. Use c-1 indicator variables, , called Dummy Variables, to code. for the ith category, for the cth category.

28. Why don’t we just use c indicator variables: ? If we use this, there will be a linear dependency among them: This will cause multicollinearity.

29. Example • If we have four years of quarterly sale data of a certain brand of soda cans. How can we model the time trend by fitting a multiple regression equation? Solution: We use quarter as a predictor variable x1. To model the seasonal trend, we use indicator variables x2, x3, x4, for Winter, Spring and Summer, respectively. For Fall, all three equal zero. That means: Winter-(1,0,0), Spring-(0,1,0), Summer-(0,0,1), Fall-(0,0,0). Then we have the model:

30. 11.6.4-11.6.5Topics in Regression Modeling Yongjun Cheng

31. 1938, R. A. Fisher and Frank Yates suggested the logistic transform for analyzing binary data. Logistic Regression Model

32. Why is it important ? • Logistic regressionmodel is the most popular model for binary data. • Logistic regression model is generally used for binary response variables. Y = 1 (true, success, YES, etc.) or Y = 0( false, failure, NO, etc.)

33. What is Logistic Regression Model? • Consider a response variable Y=0 or 1and a single predictor variable x. We want to model E(Y|x) =P(Y=1|x) as a function of x. The logistic regression model expresses the logistic transform of P(Y=1|x). This model may be rewritten as • Example

34. Some properties of logistic model • E(Y|x)= P(Y=1| x) *1 + P(Y=0|x) * 0 = P(Y=1|x) is bounded between 0 and 1 for all values of x .This is not true if we use model: P(Y=1|x) = • In ordinary regression, the regression coefficient has the interpretation that it is the log of the odds ratio of a success event (Y=1) for a unit change in x. • For multiple predictor variables, the logistic regression model is

35. Standardized Regression Coefficients • Why we need standardize regression coefficients? The regression equation for linear regression model: 1. The magnitudes of the can NOT be directly compared to judge the relative effects of on y. 2. Standardized regression coefficients may be used to judge the importance of different predictors

36. How to standardize regression coefficients? • Example: Industrial sales data Linear Model: The regression equation: NOTE: but thus has a much larger effect than on y .

37. Summary for general case • Standardized Transform • Standardized Regression Coefficients

38. 11.7.1Variables selection methodStepwise Regression LingLing Wu

39. (1)Why we need variable selection method? (2)How we select variables? * stepwise regression * best subsets regression Variables selection method

40. Stepwise Regression • (p-1)-variable model: • P-varaible model

42. Partial correlation coefficients

43. Stepwise regression algorithm

44. 11.7.1Variables selection method SAS Example Yuanhua Li, Jianping Zhang

45. Example 11.5 (pg. 416), 11.9 (pg. 431) Following table gives data on the heat evolved in calories during hardening of cement on a per gram basis (y) along with the percentages of four ingredients: tricalcium aluminate (x1), tricalcium silicate (x2), tetracalcium alumino ferrite (x3), and dicalcium silicate (x4).

46. SAS Program (stepwise selection is used) data example115; input x1 x2 x3 x4 y; datalines; 7 26 6 60 78.5 1 29 15 52 74.3 11 56 8 20 104.3 11 31 8 47 87.6 7 52 6 33 95.9 11 55 9 22 109.2 3 71 17 6 102.7 1 31 22 44 72.5 2 54 18 22 93.1 21 47 4 26 115.9 1 40 23 34 83.8 11 66 9 12 113.3 10 68 8 12 109.4 ; run; procregdata=example115; model y = x1 x2 x3 x4 /selection=stepwise; run;

47. Selected SAS output The SAS System 22:10 Monday, November 26, 2006 3 The REG Procedure Model: MODEL1 Dependent Variable: y Stepwise Selection: Step 4 Parameter Standard Variable Estimate Error Type II SS F Value Pr > F Intercept 52.57735 2.28617 3062.60416 528.91 <.0001 x1 1.46831 0.12130 848.43186 146.52 <.0001 x2 0.66225 0.04585 1207.78227 208.58 <.0001 Bounds on condition number: 1.0551, 4.2205 ----------------------------------------------------------------------------------------------------

48. SAS Output (cont) • All variables left in the model are significant at the 0.1500 level. • No other variable met the 0.1500 significance level for entry into the model. • Summary of Stepwise Selection • Variable Variable Number Partial Model Step Entered Removed Vars In R-Square R-Square C(p) F Value Pr > F 1 x4 1 0.6745 0.6745 138.731 22.80 0.0006 2 x1 2 0.2979 0.9725 5.4959 108.22 <.0001 3 x2 3 0.0099 0.9823 3.0182 5.03 0.0517 4 x4 2 0.0037 0.9787 2.6782 1.86 0.2054

49. 11.7.2Variables selection methodBest Subsets Regression Shuqiang Wang

50. 11.7.2 Best Subsets Regression For the stepwise regression algorithm • The final model is not guaranteed to be optimal in any specified sense. In the best subsets regression, • subset of variables is chosen from the collection of all subsets of k predictor variables) that optimizes a well-defined objective criterion