6. Simple Regression and OLS Estimation

6. Simple Regression and OLS Estimation Chapter 6 will expand on concepts introduced in Chapter 5 to cover the following: • Estimating parameters using Ordinary Least Squares (OLS) Estimation • Hypothesis tests of OLS coefficients • Confidence intervals of OLS coefficients • Prediction • SHAZAM use

6. Regression & OLS 6.1 The OLS Estimator and its Properties 6.2 OLS Estimators and Goodness of Fit 6.3 Confidence Intervals for Simple Regression Models 6.4 Hypothesis Testing in a Simple Regression Context 6.5 Forecasting and Prediction in Simple Regression Models 6.6 Examples of Simple Regression Models 6.7 Conclusion

6.1 The OLS Estimator and its Properties We’ve seen the true economic relationship: Yi = β1 + β2Xi + єi • Where єi and therefore Yi are random and the other terms are non-random • When this relationship is unknown, we’ve seen how to estimate the relationship Using:

6.1 Properties of the OLS Estimator • There exist a variety of methods to estimate the coefficients of our model (β1 and β2) • Why use Ordinary Least Squares (OLS)? • OLS minimizes the sum of squared errors, creating a line that fits best with the observations • With certain assumptions, OLS exhibits beneficial statistical properties. In particular, OLS is BLUE.

6.1 The OLS Estimator These OLS estimates create a straight line going through the “middle” of the estimates:

6.1.1 Fitted or Predicted Values From the above we see that often the actual data points lie above or below the estimated line. Points on the line give us ESTIMATED y values for each given x. The predicted or fitted y values are found using our x data and our estimated β’s:

6.1.1 Estimators Example Ols Estimation (From Chapter 5)

6.1.1 Estimating Errors or Residuals The estimated y values (yhat) are rarely equal to their actual values (y). The difference is the estimated error term: Since we are indifferent whether our estimates are above or below the actual, we can square these estimated errors. A higher squared error means an estimate farther from the actual

6.1.1 Estimators Example

6.1.1 Deriving OLS OLS is obtained by minimizing the sum of the square errors. This is done using the partial derivative

6.1.1 Deriving OLS These can simplify to:

6.1.1 Deriving OLS These can be expressed in their normal equations form: Notice that we have two equations with two unknowns (β2hat and β1hat). All other components come from our data set. After some math, we get the OLS estimates of β1hat and β2hat

6.1.1 Deriving OLS Finally, we should check the second derivative to confirm a minimum.

6.1.2 Statistical Properties of OLS In our model: Y, the dependent variable, is made up of two components: • β1 + β 2Xi – a non-random component that indicates the effect of X on Y. In this course, X is non-random. • Єi – a random error term representing other influences on Y.

6.1.2 Statistical Properties of OLS Error Assumptions: • E(єi) = 0; we expect no error; we assume the model is complete • Var(єi) = σ2; the error term has a constant variance • Cov(єi, єj) = 0; error terms from two different observations are uncorrelated. If the last error was positive, the next error need not be negative.

6.1.2 Statistical Properties of OLS OLS Estimators are Random Variables: • Y depends on є and is thus random. • β1hat and β2hat depend on Y… • Therefore they are random • All random variables have probability distributions, expected values, and variances • These characteristics give rise to certain OLS estimator properties.

6.1.2 OLS is BLUE We use Ordinary Least Squares estimation because, given certain assumptions, it is BLUE: B est L inear Unbiased E stimator

6.1.2 U nbiased An estimator is unbiased if it expects the true value: E(dhat) = d β2hat = ∑(Xi-Xbar)(Yi-Ybar) ---------------------- ∑(Xi-Xbar)2 β2hat = ∑(Xi-Xbar)(Yi) ------------------- ∑(Xi-Xbar)Xi By a mathematical property.

6.1.2 U nbiased β2hat = ∑(Xi-Xbar)(Yi) ------------------- ∑(Xi-Xbar)Xi E(β2hat) = ∑(Xi-Xbar)E(Yi) ------------------- ∑(Xi-Xbar)Xi Since only Yi is variable.

6.1.2 U nbiased E(β2hat) = ∑(Xi-Xbar)E(β1 + β2Xi + єi) ---------------------------------- ∑(Xi-Xbar)Xi Since Yi = β1 + β2Xi + єi E(β2hat) = ∑(Xi-Xbar)(β1 + β2Xi + 0) ---------------------------------- ∑(Xi-Xbar)Xi Since β1, β2, and Xi are non-random and E(єi)=0.

6.1.2 U nbiased E(β2hat) = β1∑(Xi-Xbar) + β2∑(Xi-Xbar)Xi ----------------------------------------- ∑(Xi-Xbar)Xi By simple algebra. E(β2hat) = β1∑(Xi-Xbar) + β2∑(Xi-Xbar)Xi ----------------- -------------------- ∑(Xi-Xbar)Xi ∑(Xi-Xbar)Xi Since there exists a common denominator.

6.1.2 U nbiased E(β2hat) = β1(0) + β2 --------------- ∑(Xi-Xbar)Xi Since the sum of the difference between an observation and its mean is zero, by definition, E(β2hat) = 0 + β2 = β2 The proof that E(β1hat)= β1 is similar.

6.1.2 U nbiased E(β2hat) = β2 This means that on average, OLS estimation will estimate the correct coefficients. Definition: If the expected value of an estimator is equal to the parameter that it is being used to estimate, the estimator is unbiased.

6.1.2 L inear The OLS estimators are linear in the dependent variable (Y): -Y’s are never raised to a power other than 1 -no non-linear operations are performed on the Y’s Note: Since X’s are squared in the β1hat and β2hat formulae, OLS is not linear in the X’s (which doesn’t matter for BLUE)

6.1.2 B est Of all linear unbiased estimators, OLS has the smallest variance. -there is a greater likelihood of obtaining an estimate close to the actual parameter Large variance => High probability of obtaining an estimate far from the center Small variance => Low probability of obtaining an estimate far from the center

6.1.2 E stimator By definition, the OLS estimator is an estimator; it estimates values for β1 and β 2.

6.1.2 Normality of Y In order to conduct hypothesis tests and construct confidence intervals from OLS, we need to know the exact distributions of β1hat and β2hat (normal, t, chi-squared, F, other, etc.) Otherwise, we can’t use statistical tables.

6.1.2 Normality of Y So far, we have assumed: • The error term, єi, is random with • E(єi)=0; no expected error • Var(єi)=σ2; constant variance • Cov(єi,єj)=0; no covariance between errors Now we add the assumption that the error term is normally distributed. Therefore: iid • Єi ~ N(0,σ2) (iid means identically and independently distributed)

6.1.2 Normality of Y If the error is normally distributed, so will be the Y term (since the randomness of Y depends on the randomness of the error term). Therefore: E(Yi) = E(β1+ β2Xi+єi)= β1+ β2Xi Var(Yi) = Var(β1+ β2Xi+єi)=Var(єi) = σ2 (Given all our previous assumptions.) Therefore: Yi ~ N(β1+ β2Xi, σ2) (Y is normally distributed with mean β1+ β2Xi and variance σ2.)

6.1.2 Normality of OLS Since β1hat and β2hat are linear functions of Y:

6.1.2 Normality of OLS If we know σ, we can construct standard normal variables (z=(x-μ)/σ):

6.1.2 Normality of OLS Since we don’t know σ2, we can estimate it: This gives us estimates of the variance of our coefficients:

6.1.2 Normality of OLS The square root of the estimated variance is referred to as the standard error (se) (as opposed to standard deviation) Using our assumptions: • (β1hat- β1)/se(β1hat) has a t distribution with N-2 degrees of freedom • (β2hat- β2)/se(β2hat) has a t distribution with N-2degrees of freedom

6.2 OLS Estimators and Goodness of Fit On average, OLS works well: • The average of the estimated errors is zero • The average of the estimated Y’s is always the average of the observed Y’s Proof: Note: This comes from our derivation of OLS as the sum of squared errors.

6.2 Measuring Goodness of Fit These conditions hold regardless of the quality of the model. Ie: You could estimate average grades as a function of locust population in Mexico. OLS would be a good estimator even though the model is useless. “Goodness of Fit” measures how well the economic model fits the data. R2 is the most common measure of goodness of fit. R2 CANNOT be compared across models.

6.2 Measuring Goodness of Fit R2 is constructed by dividing the variation of Y into two parts: • Variation in fitted Yhat terms. This is explained by the model • Variation in the estimated errors. This is NOT explained by the model.

6.2 Measuring Goodness of Fit R2 is the proportion of variation explained by the model. It is expressed as: • The ratio of explained variation to total variation in Y Or b) 1 minus the ratio of unexplained variation to total variation in Y 0<R2<1 R2=0; model has no explanatory power R2=1; model completely explains variations in Y (and generally that means you did something wrong)

6.3 Confidence Intervals for Simple Economic Models As covered previously, ordinary least squares estimation derives POINT ESTIMATES for our coefficients (β1 and β2). -These are unlikely to be perfectly accurate. Alternately, Confidence Intervals provide for us an estimate of a range for our coefficients. -We are reasonably certain that our value lies within that range.

6.3.1 Deriving a Confidence Interval Step 1: Recall Distribution We know that: (β1hat-β1)/se(β1hat) has a t distribution with N-2 degrees of freedom (β2hat- β2)/se(β2hat) has a t distribution with N-2 degrees of freedom

6.3.1 Deriving a Confidence Interval Step 2: Establish Probability: Using t-tables with N-2 degrees of freedom, we find t* such that: P(-t*<t<t*)=1-α Note that ±t* cuts off α/2 of each tail. Ie: if N=25 and α=0.10, t*=1.71

6.3.1 Deriving a Confidence Interval Step 3: Combine Steps 1 and 2 combine to give us: (1-α)% t -t* t*

6.3.1 Deriving a Confidence Interval Step 4: Rearrange for CI: OR By repeatedly calculating Confidence Intervals using OLS, 100(1- α)% of these CI’s will contain the true value of the parameter (β1).

6.3.1 Confident Example Suppose OLS Gives us the Output: If N=400, construct a 95% CI for B1: To cut off 2.5% of each tail with df=infinity, t*=1.96

6.3.1 Confident Example Suppose OLS Gives us the Output: If N=25, construct a 90% CI for B2: To cut off 5% of each tail with df=23, t*=1.71

6.3.1 Confident Example Suppose OLS Gives us the Output: In repeated samples, 95% (90%) of such confidence intervals will contain the true parameter β1 (β2). Here, we are confident that X has a positive effect on Y. We are confident that when X=0, Y is positive.

6.4 Hypothesis Testing in a Simple Regression Context As econometricians, we have questions. -Do intelligent baby toys affect baby intellect? -Do scarves have an elastic or inelastic demand? Data has answers. -Through hypothesis testing

6.4.1 Setting Up the Hypothesis Test • State null and alternate hypotheses: Ho: β2=2 Ha: β2≠2 2) Select a level of significance α=Prob(Type 1 Error) Let α=0.05 3) Determine critical t values (df=n-2) If N=25, t* = ±2.069

6.4.1 Setting Up the Hypothesis Test 4) Calculate test statistic If β2hat=6.465 and se(β2hat)=1.034, t=(β2hat- β2)/se(β2hat) =(6.465-2)/1.034) = 4.318 5) Decide (Reject and do not reject regions) Since t=4.318>t*=2.069, reject H0

6.4.1 Setting Up the Hypothesis Test 6) Interpret At a 5% level of significance, the change in Y due to a 1 unit change in X is not equal to 2.

6.4.1 Example 2 Given a sample size of 26, we estimate the formula: We want to test whether studying has any effect on grades. H0: β2=0 Ha: β2≠0

6. Simple Regression and OLS Estimation