Organization of Lecture 9

1. QM IILectures 9 & 11: Exploration of Multiple Regression Assumptions.Edmund Malesky, Ph.D., UCSD

2. Organization of Lecture 9 • Review of Gauss-Markov Assumptions • Assumptions to calculate βhat - Multicollinearity • Assumptions to show βhat is unbiased - Omitted Variable Bias - Irrelevant Variables • Assumptions to calculate variance of βhat - Limited Degrees of Freedom - Non-Constant Variance

3. 1. 6 Classical Linear Model Assumptions(Wooldridge, p. 167) Linear in Parameters Random Sampling of n observations No perfect collinearity Zero conditional mean. The error u has an expected value of 0, given any values of the independent variables Homoskedasticity. The error has the same variance given any value of the explanatory variables. 6. Normality. The population error u is independent of the explanatory variables and is normally distributed None of the independent variables are constant and there are no exact linear relationship among independent variables

4. Categories of Assumptions • Total number of assumptions necessary depends upon how you count • Matrix vs. Scalar • Four categories of assumptions: • Assumptions to calculate βhat • Assumptions to show βhat is unbiased • Assumptions to calculate variance of βhat • Assumption for Hypotheses Testing (Normality of the Errors)

5. Categories of Assumptions • Note that these sets of assumptions follow in sequence • Each step builds on the previous results • Thus if an assumption is necessary to calculate βhat , it is also necessary to show that βhat is unbiased, etc. • If an assumption is only necessary for a later step, earlier results are unaffected

6. 2. Assumptions to calculate βhat

7. Assumptions to Calculate βhat :GM3: x’s vary • Every x takes on at least two distinct values • Recall our bivariate estimator • If x does not vary, then the denominator is 0

8. Assumptions to Calculate βhat :GM3: x’s vary • If x does not vary then our data points become a vertical line. • The slope of a vertical line is undefined • Conceptually, if we do not observe variation in x, we cannot draw conclusions about how y varies with x

9. Assumptions to Calculate βhat :GM3 New: x’s are Not Perfectly Collinear • Matrix (X’X)-1 exists • Recall our multivariate estimator • This means that (X’X) must be of full rank • No columns can be linearly related

10. Assumptions to Calculate βhat :GM3 New: x’s are Not Perfectly Collinear • If one x is a perfect linear function of another then OLS cannot distinguish among their effects • If an x has no variation independent of other x’s, OLS has no information to estimate its effect. • This is more general statement of the previous assumption – x varies.

11. Multicollinearity

12. Multicollinearity • What happens if two of variables are not perfectly related, but are still highly correlated? Excerpts from readings: • Wooldridge, p. 866: “A term that refers to correlation among the independent variables in a multiple regression model; it is usually invoked when some correlations are ‘large,’ but an actual magnitude is not well-defined.” • KKV, p. 122. “Any situation where we can perfectly predict one explanatory variable from one or more of the remaining explanatory variables.” • UCLA On-line Regression Course: “The primary concern is that as the degree of multicollinearity increases, the regression model estimates of the coefficients become unstable and the standard errors for the coefficients can get wildly inflated.” • Wooldrige, p. 144. Large standard errors can result from multicollinearity.

13. Multicollinearity: Definition • If multicollinearity is not perfect, then (X’X)-1 exists and analysis can proceed. • But multicollinearity can cause problems for hypothesis testing even if correlations are not perfect. • As the level of multicollinearity increases, the amount of independent information about the x’s decreases • The problem is insufficient information in the sample

14. Multicollinearity: Implications • Our only assumption in deriving βhat and showing it is BLUE was that (X’X)–1 exists. • Thus if multicollinearity is not perfect, then OLS is unbiased and is still BLUE. • But while βhat will have the least variance among unbiased linear estimators, its variance will increase.

15. Multicollinearity: Implications • Recall our MR equation for βhat in scalar notation: • As multicollinearity increases, the correlation between xik and xikhat increases

16. Multicollinearity: Implications • Reducing x’s variation from xhat is the same as reducing x’s variation from the mean of x in the bivariate model. • Recall the equation for the variance of βhat in the bivariate regression model: • Now, we have the analogous formula for Multiple Regression The closer x is to its predicted value, the smaller the quantity and therefore the larger the variance around Beta1_hat. (a.k.a a larger standard error).

17. Multicollinearity: Implications • Thus as the correlation between x and xhat increases, the denominator for the variance of βhat decreases – increasing the variance of βhat • Notice if x1 is uncorrelated with x2…xn, then the formula is the same as in the bivariate case.

18. Multicollinearity: Implications • In practice, x’s that are causing the same y are often correlated to some extent • If the correlation is high, it becomes difficult to distinguish the impact of x1 on y from the impact of x2…xn • OLS estimates tend to be sensitive to small changes in the data.

19. When correlation among x’s is low, OLS has lots of information to estimate βhat This gives us confidence in our estimates of βhat Illustrating Multicollinearity X1 Y X2

20. When correlation among x’s is high, OLS has very little information to estimate βhat This makes us relatively uncertain about our estimate of βhat Illustrating Multicollinearity x1 y x2

21. Multicollinearity: Causes • x’s are causally related to one another and you put them both in your model. Proximate versus ultimate causality. (i.e. legacy v. political institutions and economic reform) • Poorly constructed sampling design causes correlation among x’s. (i.e. You survey 50 districts in Indonesia, where the richest districts are all ethnically homogenous, so you cannot distinguish between ethnic tension and wealth on propensity to violence.). • Poorly constructed measures over aggregate information can make cases correlate. (i.e. Freedom House – Political Liberties, Global Competitiveness Index)

22. Multicollinearity: Causes • Statistical model specification: adding polynomial terms or trend indicators. (i.e. Time since the last independent election correlates with Eastern Europe or Former Soviet Union). • Too many variables in the model – x’s measure the same conceptual variable. (i.e. Two causal variables essentially pick-up up on the same underlying variable).

23. Multicollinearity: Warning Signs • F-test of joint-significance of several variables is significant but coefficients are not. • Coefficients are substantively large but statistically insignificant. • Standard errors of βhat ’s change when other variables included or removed, but estimated value of βhat does not.

24. Multicollinearity: Warning Signs • Multicollinearity could be a problem any time that a coefficient is not statistically significant • Should always check analyses for multicollinearity levels • If coefficients are significant, then multicollinearity is not a problem for hypothesis testing. • Its only effect is to increase the σβhat2

25. Multicollinearity: The Diagnostic • Diagnostic of multicollinearity is the auxiliary R-squared • Regress each x on all other x’s in the model • R-squared will show you linear correlation between each x and all other x’s in the model

26. Multicollinearity: The Diagnostic • There is no definitive threshold when the auxiliary R-squared is too high. • Depends on whether β is significant • Tolerance for multicollinearity depends on n • Larger n means more information • If auxiliary R-squared is close to 1 and βhat is insignificant, you should be concerned • If n is small then .7 or .8 may be too high

27. Multicollinearity: Remedies • Increase sample size to get more information • Change sampling mechanism to allow greater variation in x’s • Change unit of analysis to allow more cases and more variation in x’s (districts instead of states) • Look at bivariate correlations prior to modeling

28. Multicollinearity: Remedies • Disaggregate measures to capture independent variation • Create a composite scale or index if variables measure the same concept • Construct measures to avoid correlations

29. Multicollinearity: An Example reg approval cpi unemrate realgnp Source | SS df MS Number of obs = 148 ---------+------------------------------ F( 3, 144) = 5.38 Model | 2764.51489 3 921.504964 Prob > F = 0.0015 Residual | 24686.582 144 171.434597 R-squared = 0.1007 ---------+------------------------------ Adj R-squared = 0.0820 Total | 27451.0969 147 186.742156 Root MSE = 13.093 ------------------------------------------------------------------------------ approval | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------+-------------------------------------------------------------------- cpi | -.0519523 .1189559 -0.437 0.663 -.2870775 .1831729 unemrate | 1.38775 .9029763 1.537 0.127 -.3970503 3.172551 realgnp | -.0120317 .007462 -1.612 0.109 -.0267809 .0027176 _cons | 61.52899 5.326701 11.551 0.000 51.00037 72.05762 ------------------------------------------------------------------------------

30. Multicollinearity: An Example reg cpi unemrate realgnp Source | SS df MS Number of obs = 148 ---------+------------------------------ F( 2, 145) = 493.51 Model | 82467.9449 2 41233.9725 Prob > F = 0.0000 Residual | 12115.0951 145 83.5523803 R-squared = 0.8719 ---------+------------------------------ Adj R-squared = 0.8701 Total | 94583.0401 147 643.422041 Root MSE = 9.1407 ------------------------------------------------------------------------------ cpi | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------+-------------------------------------------------------------------- unemrate | 3.953796 .5381228 7.347 0.000 2.890218 5.017374 realgnp | .0538443 .0026727 20.146 0.000 .0485618 .0591267 _cons | -30.68007 2.708701 -11.326 0.000 -36.03371 -25.32644 ------------------------------------------------------------------------------

31. Multicollinearity: An Example alpha realgnp unemrate cpi, item gen(economy) std Test scale = mean(standardized items) item-test item-rest interitem Item | Obs Sign correlation correlation correlation alpha ---------+-------------------------------------------------------------------- realgnp | 148 + 0.9179 0.8117 0.7165 0.8348 unemrate | 148 + 0.8478 0.6720 0.9079 0.9517 cpi | 148 + 0.9621 0.9092 0.5961 0.7469 ---------+-------------------------------------------------------------------- Test 0.7402 0.8952 ---------+--------------------------------------------------------------------

32. Multicollinearity: An Example . reg approval economy Source | SS df MS Number of obs = 148 ---------+------------------------------ F( 1, 146) = 7.89 Model | 1408.13844 1 1408.13844 Prob > F = 0.0056 Residual | 26042.9585 146 178.376428 R-squared = 0.0513 ---------+------------------------------ Adj R-squared = 0.0448 Total | 27451.0969 147 186.742156 Root MSE = 13.356 ------------------------------------------------------------------------------ approval | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------+-------------------------------------------------------------------- economy | -3.403865 1.211486 -2.810 0.006 -5.798181 -1.00955 _cons | 54.68946 1.097837 49.816 0.000 52.51975 56.85916 ------------------------------------------------------------------------------

33. 3. Assumptions to Show βhat is Unbiased

34. Assumptions to Show βhat is Unbiased – GM4: x’s are fixed • Conceptually, this assumption implies that we could repeat an “experiment” in which we held x constant and observed new values for u, and y • This assumption is necessary to allow us to calculate a distribution of βhat • Knowing the distribution of βhat is essential for calculating its mean.

35. Assumptions to Show βhat is Unbiased – GM4: x’s are fixed • Without knowing the mean of βhat, we cannot know whether E(βhat )= β • In addition, without knowing the mean of βhat or its distribution, we cannot know the variance of βhat • In practical terms, we must assume that the independent variables are measured without error.

36. Assumptions to Show βhat is Unbiased – GM4: x’s are fixed • Recall our equation for βhat : • X is fixed and non-zero • Thus if

37. Assumptions to Show βhat is Unbiased – GM4: x’s are fixed • Conceptually, this assumption means that we believe that our theory - described in the equation (y=Xβ+u) accurately represents our dependent variable. • If E(u) is not equal to zero then we have the wrong equation – an omitted variable

38. Assumptions to Show βhat is Unbiased – x’s are fixed • Note that the assumption E(u)=0 implies that E(u1)=E(u2)…E(un)=0 • Therefore, the assumption E(u)=0 also implies that u is independent of the values of X • That is, E(ui|Xik)=0

39. Omitted Variable Bias

40. What is Model Specification? • Model Specification is two sets of choices: • The set of variables that we include in a model • The functional form of the relationships we specify • These are central theoretical choices • Can’t get the right answer if we ask the wrong question

41. What is the “Right” Model? • In truth, all our models are misspecified to some extent. • Our theories are always a simplification of reality, and all our measures are imperfect. • Our task is to seek models that are reasonably well specified – keeps our errors relatively modest

42. Model Specification: Causes • There are four basic types of model misspecification: • Inclusion of an irrelevant variable • Exclusion of a relevant variable • Measurement error • Erroneous functional form for the relationship

43. Omitted Variable Bias: Excluding Relevant Variables • Imagine that the true causes of y can be represented as: • But we estimate the model:

44. Omitted Variable Bias: Consequences • Clearly this model violates our assumption that E(u)=0 • E(β2 x2+u) is not 0 • We required this assumption in order to show that βhat is an unbiased estimator of β • Thus by excluding x2, we risk biasing our coefficients for x1 • If βhat is biased, then σβhat is also biased

45. Omitted Variable Bias: Consequences • Under these circumstances, what does our estimate of βhat reflect? • Recall that:

46. Omitted Variable Bias: Consequences • We retain the assumption that E(u)=0 • That is the expectation of the errors is 0 after we account for the impact of x2 • Thus our estimate of βhat1 is:

47. Omitted Variable Bias:Consequences • This equation indicates that our estimate of βhat1 will be biased by two factors • The extent of the correlation between x1 and x2 • The extent of x2’s impact on y • Often difficult to know direction of bias

48. Omitted Variable Bias: Consequences • Remember, if either of these sources of bias is 0, then the overall bias is 0 • Thus E(βhat1)= β1 if: • Correlation of x1 & x2=0; note this is a characteristic of the sample OR • β2=0; note this is not a statement about the sample, but a theoretical assertion

49. Summary of Bias in β1hat the Estimator when x2 is omitted

50. Examples of Omitted Variable Bias Sin of omission: Years of education will appear to have a substantively greater effect on salary. Sin of omission: Effect of cigarettes will appear smaller. Sin of omission: Ethnic homogeneity appears to have a greater effect on reducing civil war. Sin of omission: Leaving out size of business makes me think that land titles have a larger effect on reducing expropriation.