Significance Tests:

1. Significance Tests: THE BASICS Could it happen by chance alone?

2. Statistical Inference • Confidence Intervals—Use when you want to estimate a population parameter • Significance Tests—Use when you want to assess the evidence provided by data about some claim concerning a population • AN OUTCOME THAT WOULD RARELY HAPPEN BY CHANCE IF A CLAIM WERE TRUE IS GOOD EVIDENCE THAT THE CLAIM IS NOT TRUE

3. Overview of a Significance Test • A test of significance is intended to assess the evidence provided by data against a null hypothesis H0 in favor of an alternate hypothesis Ha. • The statement being tested in a test of significance is called the null hypothesis. Usually the null hypothesis is a statement of “no effect” or “no difference.” • A one-sided alternate hypothesis exists when we are interested only in deviations from the null hypothesis in one direction H0 : =0 Ha : >0 (or <0) • If the problem does not specify the direction of the difference, the alternate hypothesis is two-sided H0: =0 Ha: ≠0

4. HYPOTHESES • NOTE: Hypotheses ALWAYS refer to a population parameter, not a sample statistic. • The alternative hypothesis should express the hopes or suspicions we have BEFORE we see the data. Don’t “cheat” by looking at the data first.

5. CONDITIONS • These should look the same as in the last chapter (for confidence intervals) • Random • Data is from an SRS or from a randomized experiment • Normal • For means—population distribution is Normal or you have a large sample size (n≥30) to ensure a Normal sampling distribution for the sample mean • For proportions—np≥10 and n(1-p)≥10 (meaning the sample is large enough to ensure a Normal sampling distribution for • Independent • Either you are sampling with replacement or you have a population at least 10 times as big as the sample to make using the formula for st. dev. okay.

6. CAUTION • Be sure to check that the conditions for running a significance test for the population mean are satisfied before you perform any calculations.

7. Test Statistic • A test statistic comes from sample data and is used to make decisions in a significance test • Compare sample statistic to hypothesized parameter • Values far from parameter give evidence against the null hypothesis (H0) • Standardize your sample statistic to obtain your TEST STATISTIC

8. P-values & statistical significance • The probability (computed assuming H0 is true) that the test statistic would take a value as extreme or more extreme than that actually observed is called the P-value of the test. The smaller the P-value, the stronger the evidence against the null hypothesis provided by the data. • “Significant” in the statistical sense doesn’t mean “important”. It means simply “not likely to happen just by chance.” • The significance level α is the decisive value of the P-value. It makes “not likely” more exact. • If the P-value is as small or smaller than α, we say that the data is statistically significant at levelα.

9. INFERENCE TOOLBOX (p 705) DO YOU REMEMBER WHAT THE STEPS ARE??? Steps for completing a SIGNIFICANCE TEST: • 1—PARAMETER—Identify the population of interest and the parameter you want to draw a conclusion about. STATE YOUR HYPOTHESES! • 2—CONDITIONS—Choose the appropriate inference procedure. VERIFY conditions (Random, Normal, Independent) before using it. • 3—CALCULATIONS—If the conditions are met, carry out the inference procedure. • 4—INTERPRETATION—Interpret your results in the context of the problem. CONCLUSION, CONNECTION, CONTEXT(meaning that our conclusion about the parameter connects to our work in part 3 and includes appropriate context)

10. Step 1—PARAMETER • Read through the problem and determine what we hope to show through our test. • Our null hypothesis is that no change has occurred or that no difference is evident. • Our alternative hypothesis can be either one or two sided. • Be certain to use appropriate symbols and also write them out in words.

11. Step 2—CONDITIONS • Based on the given information, determine which test should be used. Name the procedure. • State the conditions. • Verify (through discussion) whether the conditions have been met. For any assumptions that seem unsafe to verify as met, explain why. • Remember, if data is given, graph it to help facilitate this discussion • For each procedure there are several things that we are assuming are true that allow these procedures to produce meaningful results.

12. Step 3—CALCULATIONS • First write out the formula for the test statistic, report its value, mark the value on the curve. • Sketch the density curve as clearly as possible out to three standard deviations on each side. • Mark the null hypothesis and sample statistic clearly on the curve. • Calculate and report the P-value • Shade the appropriate region of the curve. • Report other values of importance (standard deviation, df, critical value, etc.)

13. Step 4—INTERPRETATION • There are really two parts to this step: decision & conclusion. TWO UNIQUE SENTENCES. • Based on the P-value, make a decision. Will you reject H0 or fail to reject H0. • If there is a predetermined significance level, then make reference to this as part of your decision. If not, interpret the P-value appropriately. • Now that you have made a decision, state a conclusion IN THE CONTEXT of the problem. • This does not need to, and probably should not, have statistical terminology involved. DO NOT use the word “prove” in this statement.

14. Example 1 • Your buddy (Jake) claims to be an A student (meaning he has a 90 average). You don’t know all of his grades but based on what you have seen you think this claim is an overstatement. You took a simple random sample of his grades and recorded them. They are: 92, 87, 86, 90, 80, 91. You also know that all his grades in the class have a standard deviation of 3.5.

15. Step 1 • We want to determine whether Jake is accurate in his measure of his course grade. • Our null hypothesis is that Jake has a course average of 90. • Our alternative hypothesis is that Jake’s course average is below a 90. • H0:  = 90 • Ha:  < 90

16. Step 2 • Since we know the population standard deviation we will be performing a z-test of significance. (NOTE-in practice, we rarely know sigma) • We were told that our selection of grades was an SRS of Jake’s scores. • The box plot shows moderate left skewness. Our sample is not large so we must assume that the population of all of Jake’s grades are approximately normal in distribution in order for our sampling distribution to be approximately normal. Using the IQR(1.5) method for determining outliers we see that there are no outliers in this sample of grades. • Provided Jake has at least 60 overall grades, we are safe assuming independence and using the necessary formula for standard deviation.

17. Step 3 • A curve should be drawn, labeled, and shaded. • You can use the formula to calculate your z test statistic for this problem •  In this case z = -1.6330 • Mark this on your sketch. • Based on our calculations the P-value is 0.0512. • , σ=3.5, n=6

18. Step 4 • Since there is no predetermined level of significance if we are seeking to make a decision, this could be argued either way. If Jake were correct about being an A student, we would only get a sample of grades with an average this low in roughly 5.1% of all samples. • There is not overwhelming evidence against H0, however, this is enough to convince me that H0 can be rejected. • Our evidence may not be strong enough to convince Jake that he is wrong. However, based on this evidence, I do not believe Jake is accurate about his average being a 90. It doesn’t appear that Jake is the A student he claims to be.

19. WARNINGS • Tests of significance assess evidence against H0 • If the evidence is strong, reject H0 in favor of Ha • Failure to find evidence against H0 means only that data are consistent with H0, not that we have clear evidence that H0 is true • If you are going to make a decision based on statistical significance, then the significance level αshould be stated before the data are produced.