1 / 29

STATS 330: Lecture 8

STATS 330: Lecture 8. Collinearity. Collinearity. Aims of today’s lecture: Explain the idea of collinearity and its connection with estimating regression coefficients To discuss added variable plots, a graphical method for deciding if a variable should be added to a regression.

Download Presentation

STATS 330: Lecture 8

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. STATS 330: Lecture 8 Collinearity 330 lecture 8

  2. Collinearity Aims of today’s lecture: Explain the idea of collinearity and its connection with estimating regression coefficients To discuss added variable plots, a graphical method for deciding if a variable should be added to a regression 330 lecture 8

  3. Variance of regression coefficients • We saw in Lecture 6 how the standard errors of the regression coefficients depend on the error variance s2: the bigger s2, the bigger the standard errors. • We also suggested that the standard error depends on the arrangement of the x’s. • In today’s lecture, we explore this idea a bit further. 330 lecture 8

  4. Example • Suppose we have a regression relationship of the form Y=1 + 2x –w + e between a response variable Y and two explanatory variables x and w. • Consider two data sets, A and B, each following the model above. 330 lecture 8

  5. Data sets A & B: x,w data 330 lecture 8

  6. 330 lecture 8

  7. Conclusion: • The greater the correlation, the more variable the plane. • In fact, for the coefficient b of x 330 lecture 8

  8. Generalization If we have k explanatory variables, then the variance of the jth estimated coefficient is where Rj2 is the R2 if we regress variable j on the other explanatory variables 330 lecture 8

  9. Best case • If xj is orthogonal to (uncorrelated with) the other explanatory variables, then Rj2 is zero and the variance is the smallest possible i.e. 330 lecture 8

  10. Variance inflation factor The factor represents the increase in variance caused by correlation between the explanatory variables and is called the variance inflation factor (VIF) 330 lecture 8

  11. Calculating the VIF: theory To calculate the VIF for the jth explanatory variable, use the relationship using the residuals from regressing the jth explanatory variable on the other explanatory variables 330 lecture 8

  12. Calculating the VIF: example For the petrol data, calculate the VIF for t.vp (tank vapour pressure) > attach(vapour.df) > tvp.reg <- lm(t.vp~t.temp + p.temp + p.vp,data=vapour.df) > var(t.vp)/var(residuals(tvp.reg)) [1] 66.13817 Correlation increases variance by a factor of 66 330 lecture 8

  13. Calculating the VIF: quick method A useful mathematical relationship: Suppose we calculate the inverse of the correlation matrix of the explanatory variables. Then the VIF’s are the diagonal elements. > X<-vapour.df[,-5] # delete 5th column # (hc, the response) > VIF <- diag(solve(cor(X))) > VIF t.temp p.temp t.vp p.vp 11.927292 5.615662 66.138172 60.938695 330 lecture 8

  14. Pairs plot 330 lecture 8

  15. Collinearity • If one or more variables in a regression have big VIF’s, the regression is said to be collinear • Caused by one or more variables being almost linear combinations of the others • Sometimes indicated by high correlations between the independent variables • Results in imprecise estimation of regression coefficients • Standard errors are high, so t-statistics are small, variables are often non-significant ( data is insufficient to detect a difference) 330 lecture 8

  16. Non-significance • If a variable has a non-significantt, then either • The variable is not related to the response, or • The variable is related to the response, but it is not required in the regression because it is strongly related to a third variable that is in the regression, so we don’t need both. • First case: small t-value, small VIF, small correlation with response • Second case, small t-value, big VIF, big correlation with response 330 lecture 8

  17. Remedy • The usual remedy is to drop one or more variables from the model. • This “breaks” the linear relationship between the variables • This leads to the problem of “subset selection”, which subset to choose. • See Lectures 14 and 15 330 lecture 8

  18. Example: Cement data • Measurements on batches of cement • Response variable: Heat (heat emitted) • Explanatory variables • X1: amount of tricalcium aluminate (%) • X2: amount of tricalcium silicate (%) • X3: amount of tetracalcium aluminaoferrite (%) • X4: amount of dicalcium silicate (%) 330 lecture 8

  19. Example: Cement data Heat X1 X2 X3 X4 78.5 7 26 6 60 74.3 1 29 15 52 104.3 11 56 8 20 87.6 11 31 8 47 95.9 7 52 6 33 109.2 11 55 9 22 102.7 3 71 17 6 72.5 1 31 22 44 93.1 2 54 18 22 115.9 21 47 4 26 83.8 1 40 23 34 113.3 11 66 9 12 109.4 10 68 8 12 330 lecture 8

  20. Example: Cement data Estimate Std. Error t value Pr(>|t|) (Intercept) 62.4054 70.0710 0.891 0.3991 X1 1.5511 0.7448 2.083 0.0708 . X2 0.5102 0.7238 0.705 0.5009 X3 0.1019 0.7547 0.135 0.8959 X4 -0.1441 0.7091 -0.203 0.8441 Residual standard error: 2.446 on 8 degrees of freedom Multiple R-Squared: 0.9824, Adjusted R-squared: 0.9736 F-statistic: 111.5 on 4 and 8 DF, p-value: 4.756e-07 Large p-values > round(cor(cement.df),2) Heat X1 X2 X3 X4 Heat 1.00 0.73 0.82 -0.53 -0.82 X1 0.73 1.00 0.23 -0.82 -0.25 X2 0.82 0.23 1.00 -0.14 -0.97 X3 -0.53 -0.82 -0.14 1.00 0.03 X4 -0.82 -0.25 -0.97 0.03 1.00 Big R-squared Big correlation 330 lecture 8

  21. Cement data Omit Heat > diag(solve(cor(cement.df[, -1]))) X1 X2 X3 X4 38.49621 254.42317 46.86839 282.51286 Very large! Very large! cement.df$X1 + cement.df$X2 + cement.df$X3 + cement.df$X4 [1] 99 97 95 97 98 97 97 98 96 98 98 98 98 330 lecture 8

  22. Drop X4 > diag(solve(cor(cement.df[, -c(1,5)]))) X1 X2 X3 3.251068 1.063575 3.142125 VIF’s now small Estimate Std. Error t value Pr(>|t|) (Intercept) 48.19363 3.91330 12.315 6.17e-07 *** X1 1.69589 0.20458 8.290 1.66e-05 *** X2 0.65691 0.04423 14.851 1.23e-07 *** X3 0.25002 0.18471 1.354 0.209 Residual standard error: 2.312 on 9 degrees of freedom Multiple R-Squared: 0.9823, Adjusted R-squared: 0.9764 F-statistic: 166.3 on 3 and 9 DF, p-value: 3.367e-08 X1, X2 now signif 330 lecture 8

  23. Added variable plots (AVP’s) To see if a variable, say x, is needed in a regression: • Step 1: Calculate the residuals from regressing the response on all the explanatory variables except x • Step 2: calculate the residuals from regressing x on the other explanatory variables • Step 3: Plot the first set of residuals versus the second set NB: Also called partial regression plots in some books 330 lecture 8

  24. Rationale • The first set of residuals represents the variation in y not explained by the other explanatory variables • The second set of residuals represents the part of x not explained by the other explanatory variables • If there is a relationship between the two sets, there is a relationship between x and the response that is not accounted for by the other explanatory variables • Thus, if we see a relationship in the plot, x is needed in the regression!!! 330 lecture 8

  25. Example: the petrol data Let’s do an AVP for tank vapour pressure, t.vp. > rest.reg<- lm(hc~t.temp + p.temp + p.vp,data=vapour.df) > y.res<-residuals(rest.reg) > tvp.reg<-lm(t.vp~t.temp + p.temp + p.vp,data=vapour.df) > tvp.res<-residuals(tvp.reg) > plot(tvp.res,y.res, xlab = "Tank vapour pressure", ylab="Hydrocarbon emission", main = “AVP for Tank vapour pressure") 330 lecture 8

  26. Hint of a relationship: so variable required? 330 lecture 8

  27. Short cut in R There is a function added.variable.plots in R to draw the plots automatically. This is one of the functions in the R330 package which must be installed before the function can be used. Note this useful trick! > vapour.lm<-lm(hc~.,data=vapour.df) > par(mfrow=c(2,2)) # 2 x 2 array of plots > added.variable.plots(vapour.lm) 330 lecture 8

  28. Not significant in the regression (Lect 7) 330 lecture 8

  29. Some curious facts about AVP’s • Assuming a constant term in both regressions, a least squares line fitted though the AVP goes through the origin. • The slope of this line is the fitted regression coefficient for the variable in the original regression • The residuals from this line are the same as the residuals from the original regression 330 lecture 8

More Related