MPP Stats Bootcamp W (12:00-1:50), Week 8

1. MPP Stats BootcampW (12:00-1:50), Week 8 Instructor: Dr. Alison Johnston (Alison.Johnston@oregonstate.edu) http://oregonstate.edu/cla/polisci/alison-johnston SOC 516 Class Tutor: Daniel Hauser (hauserd@onid.orst.edu)

2. Bootcamp Outline • Week 7: Significance testing • Outlining your hypothesis • The four steps of significance testing • For means testing • For cross-tabs • For difference in means • For regression analysis…. • Confidence Intervals

3. Falsifying Hypotheses: The Role of Significance Testing • Significance testing exists to test the validity of a hypothesis • Hypotheses are predictions. • For means, we establish hypotheses to predict what the mean is • For difference means, we establish hypotheses to predict whether the difference in means is zero • For cross-tabs, we establish hypotheses to predict whether there is association between our dependent and independent variables • For regression analysis, we establish hypotheses to predict how one variable (an independent variable) influences another (dependent variable)

4. Hypothesis testing: Just a Smoking Gun? • It is very difficult to prove whether a hypothesis is correct • Samples: We cannot be 100% sure that our sample is a perfect representation of a population • Populations: Measurement errors across units of analysis • Hypothesis testing revolves around the rejection of a hypothesis with a certain level of confidence • Null Hypothesis (H0):A default position • Alternative Hypothesis (HA): The predictive position

5. Classic Null and Alternative Hypotheses • We posit a null hypothesis AGAINST an alternative hypothesis • Both the null and alternative hypotheses must exhaust all options • One-tailed/sided test: • H0: = 0 • HA: ≤ 0 or HA: ≥ 0 • Two tailed/sided test: • H0: = 0 • HA:

6. Types of Null/Alternative Hypotheses covered in 524/516

7. Hypothesis Testing: Margin of Error(s) • In hypothesis testing, there is always some margin for error. Two types of errors exist: • Type I (false positive – “lucky drunk”): We reject a true null hypothesis • Type II (false negative – “unlucky professional”): We fail to reject a false null hypothesis. • How can we avoid both?: • Be meticulous with the collection of your data • Adopt a scrupulous decision rule (i.e. a high critical value for your test statistic with a high/low associated level of confidence/p-value) • This is where significance testing comes in….

8. Significance Testing and Empirical Methods • The general components of significance testing (via the t-test, Chi-squared test, or confidence interval method) are equivalent, regardless of whether we are looking at means, difference in means or cross-tabs • The exact formula for the test statistic/confidence interval, however, differs slightly depending on the method • (Difference in) Means Testing: Mean centric t-statistic • Cross-tabs: Pearson Chi-squared statistic of independence/non-association • Regression analysis: Zero beta centric t-statistic

9. 1. Significance Testing for Means • With descriptive statistics, our hypotheses revolve around the numerical value we predict the population mean/difference-in-means to be • One group: Arbitrary number • Two (independent) groups: Zero difference in means • We can not make claims about these values, however, without considering the variation of our data • If we have high variation (standard deviation/errors), our predictions of the population mean will be less precise. Significance testing account for this

10. Significance Testing for Means: 4 Steps • Significance testing (via the t-test method) for hypotheses of means involves 4 steps • Establishing a null and alternative hypothesis • Calculating a test-statistic which we assume is normally distributed • Sample Mean (Difference) • Sample Standard error • Comparing the test-statistic against a critical value • Degrees of freedom • Corresponding p-value • Making a conclusion about the null hypothesis

11. Means significance Testing: Steps 1 and 2

12. Means Significance Testing: Step 3 • Once you have calculated your t-statistic, you need to compare it to a critical value • Confidence level (NEVER select a confidence level below 90%) • Degrees of freedom: Sample size – number of parameters that are estimated • Critical values determine your “non-rejection region” and “rejection region” • The size of your rejection region is also known are your p-value (α), the probability of committing a Type I error (rejecting a true null hypothesis)

13. Critical Values for T-Statistic Distribution

14. Rejection Region and P-Values: One-Tailed T-test

15. Conclusions: Step 4 • After you have determined a critical value to compared your t-statistic against, you are ready to make a conclusion • Two types of conclusions: • If |t-stat| > Critical t-value: REJECT Null Hypothesis with 90/95/99% confidence • If |t-stat| < Critical t-value: FAIL TO REJECT the Null Hypothesis with 90/95/99% confidence • Contingent on level of confidence!

16. Means Testing Example: • We have calculated the mean income for a group of 200 graduate students at OSU in their first year after graduation • Mean income: $40,000 • Standard deviation: $12,467 • Note, this is NOT the standard error! • Can we reject the claim that the mean of the population of all graduate students in their first year out of school is $30,000 based on our pulled sample?

17. Means Testing Class Example: Solution • Step 1: State the null and alterative hypothesis • H0: H0: • HA: H0: • Step 2: Calculate the t-statistic • t = (40,000 / = 11.344 • Step 3: Compare the t- statistic against a critical value • For (200-1) = 199degrees of freedom, the critical t-value is roughly 1.66for a 90% confidence level, 1.984 for a 95% confidence level, and 2.626 for a 99% confidence level • Step 4: State your conclusion • We REJECT the null hypothesis that the population mean is roughly $30,000 with 99% confidence

18. Difference in Means Testing Example: • Say we want to determine if two school with different educational models towards students with disabilities exhibit different average exam scores. We have the following information on mean scores for the state proficiency test in reading: • School with an inclusive model (35 students) • Mean: 257 • Standard Deviation: 10.262 • School with a pullout model (21 students) • Mean: 253 • Standard Deviation: 12.015 • Is there a significant difference in reading proficiency scores between the two models?

19. Difference in Means Testing Class Example: Solution • Step 1: State the null and alterative hypothesis • H0: There is no difference in average test scores between the two groups • HA: There is a difference in average test scores between the two groups • Step 2: Calculate the t-statistic • t = (257 / = 1.272 • Step 3: Compare the t- statistic against a critical value • For (35-1)+(21-1) = 54 degrees of freedom, the critical t-value is slightly less than 1.676 for a 90% confidence level • Step 4: State your conclusion • We FAIL TO REJECT the null hypothesis that there is no difference between the two groups

20. A Short Cut: Confidence Intervals • Thus far, we have only discussed t-test/Chi-squared test method of significance testing • For means and beta significance testing (only), we can use an alternative method: Confidence intervals • Represents an (estimated) range where true mean/beta should lie • Rather than follow a 4 step process, we calculate the confidence interval • If the null hypothesis value lies within the range we can NOT reject it • If it lies outside it, we can reject it

21. Confidence Intervals: The Math

22. Confidence Interval Significance Testing Example • Let’s calculate a 95% confidence interval for our previous average graduate student income one year out of school • Mean = $40,000 • Standard Deviation = $12,467 • Sample Size = 200

23. Calculate the Confidence Interval • Calculate the confidence interval • 40,000 ± ()*1.984 • 40,000 ± (881.55)*1.984 • 40,000 ± 1,748.995 • (38,251.005, 41,748.995) • Since 30,000 lies outside this confidence interval, we can reject our null that for the population the average graduate salary is $30,000

24. Significance Testing For Cross-Tabs • When creating cross-tabs/contingency tables, we summarize the frequency of data that falls under each category of our two (categorical) variables • Dependent variable: ROW variable • Independent variable: COLUMN variable • In compiling a cross-tab, we can also analyze the relationship between two categorical variables via a Pearson Chi-Squared significance test • Are paired observations for two (categorical) variables independently distributed?

25. Cross-Tabs: An Example

26. Cross-Tabs: An Example of Significance Testing • Say we are interested in analyzing the relationship between two categorical variables (fitness and residency) • A Pearson Chi-Squared test of independence/ association enables us to test whether people from different states differ in the frequency which they report they work-out • H0: There is no difference (association) between individual reporting of fitness by state • HA: There is a difference (association) between individual reporting of fitness by state

27. Significance Testing for Cross-Tabs: 4 Steps • Significance testing (via Pearson’s Chi-Squared test) for hypotheses of association/independence involves 4 steps • Establishing a null and alternative hypothesis • Calculating a Chi-squared test-statistic which we assume possesses a Chi-Squared Distribution • Compute in STATA/SPSS • Comparing the Chi-squared test-statistic against a critical value • Degrees of freedom • Corresponding p-value • Making a conclusion about the null hypothesis

28. Cross-Tab significance Testing: Steps 1 and 2

29. Cross Tab Significance Testing: Step 3 • Once you have your Chi-squared statistic, you need to compare it to a critical χ2 value • Confidence level (above 90%) • Degrees of freedom: (rows-1)(columns-1) • NOTICE: Chi-squared critical values, UNLIKE t-values, INCREASE with higher degrees of freedom • Similar to t-testing, critical values determine your “non-rejection region” and “rejection region”

30. Critical Values for Chi-Squared Distributions

31. Conclusions From the Critical Value: Step 4 • After you have determined a critical value to compared your Chi-Square statistic against, you are ready to make a conclusion • Two types of conclusions: • If χ2 stat > Critical t-value: REJECT Null Hypothesis of no association with 90/95/99% confidence • If χ2 stat < Critical t-value: FAIL TO REJECT the Null Hypothesis of no association with 90/95/99% confidence

32. Cross Tabs Significance Testing: Class Example • You have produced the following cross tab for self-reported fitness by state residency status • Conduct a four-step significance test (assume the χ2stat is equal to 6.43)

33. Cross Tabs Significance Testing Class Example: Solution • Step 1: State the null and alterative hypothesis • H0: There is no association between reported fitness level and state of residency • HA: There is an association between reported fitness level and state of residency • Step 2: Calculate the Chi-squared statistic • χ2 stat = 6.43 • Step 3: Compare the Chi-squared statistic against a critical value • For (2-1)(3-1) = 2 degrees of freedom, the critical χ2value is 5.99 for a 95% confidence level, and 9.21 for a 99% confidence level • Step 4: State your conclusion • We REJECT the null hypothesis of no association, but ONLY with 95% confidence

34. STATA Lab Exercises • Tabulating a t-test for means in STATA using a small and large dataset • Tabulating a difference in means test in STATA using a small and large dataset • Calculating 90%, 95% and 99% confidence intervals for means in STATA • Tabulating tests of association between two categorical variables (cross-tabs) in STATA