Hypothesis Testing III 2/15/12

1. Hypothesis Testing III • 2/15/12 • Statistical significance • Errors • Power • Significance and sample size • Section 4.3 • Professor Kari Lock Morgan • Duke University

2. To Do • Project 1 Proposal(due today, 5pm) • Homework 4 (due Monday) • If you turn in your HW4 by 5pm on Friday (either slide it under the door of Old Chem 216 or email it to your TA), it will be graded by class on Monday • NO LATE HOMEWORK ACCEPTED! • Study/prepare for Exam 1!

3. Exam 1 • Exam 1: • In-class portion: Wednesday, 2/22 • Lab portion: Thursday, 2/23 • In-class portion: (75%) • Open only to a calculator and one double sided page of notes prepared by you • Emphasis on conceptual understanding • Lab portion: (25%) • Open to everything except communication of any form with other humans • Emphasis on actually analyzing data

4. Practice • Last year’s in-class and lab midterms, with solutions, are available on the course website (under documents) • Full solutions to ALL the essential synthesis and review problems from Units 1 and 2 are available on the course website • Doing problems is the key to success!!!

5. Keys to In-Class Exam Success • Work lots of practice problems! • Take last year’s exams under realistic conditions (time yourself, do it all before looking at the solutions, etc.) • Prepare a good cheat sheet and use it when working problems • Read the corresponding sections in the book if there are concepts you are still confused about

6. Keys to Lab Exam Success • Primarily, make sure you know how to summarize, visualize, create an interval, and conduct a test for any one variable or relationship between two variables. • Beyond that, make sure you are comfortable with the content from the labs • Open-book does NOT mean you don’t have to study. You will not have time to look up every command you need during the exam.

7. Office Hours before Exam • You have LOTS of opportunities for help! • Today, 3 – 5pm (Prof Morgan) • Friday, 1 – 3 pm (Prof Morgan • Sunday, 5 – 7 pm (Jessica) • Sunday, 7 – 9 pm (Michael) • Monday, 3 – 4 pm (Prof Morgan) • Monday, 4 – 6 pm (Christine) • Tuesday, 3 – 6 pm (Prof Morgan) • Tuesday, 6 – 8 pm (Yue) • (My office hours next week have been moved to Monday and Tuesday to answer questions before the exam)

8. Statistical Conclusions Strength of evidence against H0: Formal decision of hypothesis test, based on  = 0.05 :

9. Red Wine and Weight Loss • Resveratrol, an ingredient in red wine and grapes, has been shown to promote weight loss in rodents, and has recently been investigated in primates (specifically, the Grey Mouse Lemur). • A sample of lemurs had various measurements taken before and after receiving resveratrol supplementation for 4 weeks BioMed Central (2010, June 22). “Lemurs lose weight with ‘life-extending’ supplement resveratrol. Science Daily.

10. Red Wine and Weight Loss In the test to see if the mean resting metabolic rate is higher after treatment, the p-value is 0.013. Using  = 0.05, is this difference statistically significant? (should we reject H0: no difference?) (a) Yes (b) No

11. Red Wine and Weight Loss In the test to see if the mean body mass is lower after treatment, the p-value is 0.007. Using  = 0.05, is this difference statistically significant? (should we reject H0: no difference?) (a) Yes (b) No

12. Red Wine and Weight Loss In the test to see if locomotor activity changes after treatment, the p-value is 0.980. Using  = 0.05, is this difference statistically significant? (should we reject H0: no difference?) (a) Yes (b) No

13. Red Wine and Weight Loss In the test to see if mean food intake changes after treatment, the p-value is 0.035. Using  = 0.05, is this difference statistically significant? (should we reject H0: no difference?) (a) Yes (b) No

14. Formal Decisions Suppose many researchers around the world are all investigating the same question of interest. If the null hypothesis is true, using  = 0.05, what proportion of hypothesis tests will incorrectly reject the null? None 95% 5% It depends

15. Errors • There are four possibilities: Decision  TYPE I ERROR Truth  TYPE II ERROR

16. Red Wine and Weight Loss • In the test to see if resveratrol is associated with food intake, the p-value is 0.035. • If resveratrolis not associated with food intake, a Type I Error would have been made • In the test to see if resveratrol is associated with locomotor activity, the p-value is 0.980. • If resveratrolis associated with locomotor activity, a Type II Errorwould have been made

17. Errors • Usually, we have no way of knowing whether an error has been made, without doing another study • Analogously, we have no way to knowing whether our confidence interval actually contains the true parameter

18. Errors If the null hypothesis is true, what is the probability of making a Type I error? 0  1 –  It depends

19. Significance Level • Why would you use a smaller significance level, like  = 0.01? • Why would you use a larger significance level, like  = 0.10?

20. Analogy to Law Ho Ha A person is innocent until proven guilty. Evidence must be beyond the shadow of a doubt.  Types of mistakes in a verdict? Convict an innocent Type I error Release a guilty Type II error

21. Errors If the alternative hypothesis is true, what is the probability of making a Type II error? 0  1 –  It depends

22. Power • The power of a hypothesis test is the probability of correctly detecting a significant effect, is there is one (correctly rejecting the null hypothesis when it is false)

23. Errors • There are four possibilities: Decision  TYPE I ERROR Truth  TYPE II ERROR

24. Power What factors influence the power of a test? Sample size True value or effect size Variability of values (standard deviation)

25. Power If you want to increase the power of your test, what can you do? Increase the sample size Make the true value farther from the null value Decrease the standard deviation of your variables Any of the above

26. Significance and Sample Size www.lock5stat.com/statkey

27. Significance and Sample Size • The smaller the sample size, the • (a) smaller • (b) larger • the chance of a Type II error (failing to reject the null hypothesis, even when it is false).

28. Significance and Sample Size • Just because you fail to get significant results, does NOT mean the null hypothesis is true • This is particularly true for small sample sizes. Unless the truth is very far from the null value, it is hard to find significant results if the sample size is small. • With small sample sizes, Type II Errors are very likely!

29. Statistical vs Practical Significance • With small sample sizes, even large differences or effects may not be significant • With large sample sizes, even a very small difference or effect can be significant • A statistically significant result is not always practically significant, especially with large sample sizes

30. Statistical vs Practical Significance • Example: Suppose a weight loss program recruits 10,000 people for a randomized experiment. • A difference in average weight loss of only 0.5 lbs could be found to be statistically significant • Suppose the experiment lasted for a year. Is a loss of ½ a pound practically significant?

31. Videogames and GPA If you get put with a roommate who brings a videogame to college, does that lower your GPA? What are the null and alternative hypotheses? • H0: pv – pnv= 0, Ha: pv – pnv< 0 • H0: µv – µnv= 0, Ha: µv – µnv< 0 • H0: pv – pnv< 0, Ha: pv – pnv= 0 • H0: µv – µnv< 0, Ha: µv – µnv= 0

32. Videogames and GPA If you get put with a roommate who brings a videogame to college, does that lower your GPA? A study at Berea college conducted this test and obtained a p-value of 0.036. What does this mean? • The probability that H0 is true is 0.036 • The probability that H0 is false is 0.036 • The probability of seeing a difference in mean GPA as extreme as that in the sample is 0.036 • If H0 is true, the probability of seeing a difference in mean GPA as extreme as that in the sample is 0.036 Source: Stinebrickner, R. and Stinebrickner, T.R. (2008). “The Causal Effect of Studying on Academic Performance,” The B.E. Journal of Economic Analysis & Policy: Vol. 8: Iss. 1 (Frontiers), Article 14.

33. Videogames and GPA • In the study about roommates bringing videogames to college and GPA, the p-value is 0.036. Using  = 0.05, what would you conclude? • People assigned roommates who bring videogames have significantly lower GPAs • People assigned roommates who bring videogames do not have lower GPAs • Nothing

34. Videogames and GPA • Based on this p-value, can you conclude that getting put with a roommate who brings a videogame to campus causes you to have a lower GPA, on average? • Yes • No

35. Significance and Causation • The p-value alone tells you whether there is a significant association between two variables, but NOT whether this is a causal association • The data collection method tells you whether causal conclusions are possible, but not whether an association is significant • If the study is a randomized experiment AND the p–value indicates statistically significant results, only then you can conclude that the explanatory variable has a causal effect on the response variable

36. Videogames and GPA • Roommates are assigned randomly at Berea college. Based on this knowledge and the p-value (0.036), can you conclude that getting put with a roommate who brings a videogame to campus causes you to have a lower GPA, on average? • Yes • No

37. Videogames and GPA • The study also tested whether students who bring a videogame to college themselves have lower GPAs, on average. The p-value of this test is 0.068. Using  = 0.05, what would you conclude? • People who bring videogames have significantly lower GPAs • People who bring videogames do not have lower GPAs • In this study, the difference in average GPA between students who bring videogames and those who don’t is not statistically significant • Nothing • Either (c) or (d)

38. Summary • In making formal decisions, reject H0 if the p-value is less than α, otherwise do not reject H0 • There are two types of errors that can be made in hypothesis testing: rejecting a true null or failing to reject a false null • The larger your sample size, the higher your chance of finding a significant result, if one exists