Understanding Statistical vs Practical Significance in Testing

1. Lesson 11 - 3 Use and Abuse of Tests

2. Knowledge Objectives • Distinguish between statistical significance and practical importance • Identify the advantages and disadvantages of using P-values rather than a fixed level of significance

3. Construction Objectives • none this lesson

4. Vocabulary • Statistically Significant – when observed results are unlikely under the assumption that the null hypothesis is true. When results are found to be statistically significant, we reject the null hypothesis • Practical Significance – refers to things that are statistically significant, but the actual difference is not large enough to cause concern or be considered important

5. Choosing a Level of Significance How small a P-value is convincing evidence against the null hypothesis? This depends on two circumstances: • How plausible is H0?-- if H0 represents something people have believed for years, then you will need strong evidence (a small P-value) • What are the consequences of rejecting H0?-- if rejecting H0 costs the company lots of money, then you will need strong evidence. There is no sharp border between “statistically significant” and “statistically insignificant”

6. Example 1 Your company has developed a new antibacterial cream. From previous research you know that with no medication, a small cut on the inner forearm will heal on average in 7.6 days with a standard deviation of 1.4 days. You want to test this new cream at the 5% significance level. We cut 25 volunteer college students and apply the cream to the wounds. The mean healing time for them was 7.1 days. We will assume that σ = 1.4 days.

7. Example 1 cont μ = 7.6 days (to heal the cut) Hypothesis: H0: Ha: Conditions: 1: 2: 3: μ< 7.6 days SRS -- assume that our volunteers represent a simple random sample (shaky for general population vs college students) Normality -- 25 is not considered a large enough number for CLT to apply. We would need to check graphs (box-plot and normality plot) to verify reasonableness of normality Independence -- easy to assume each test is independent for the others

8. Example 1 cont x – μ0 7.1 – 7.6 Z0 = ----------- = ------------------ σ/√n 1.4/√25 = -1.79 P-value = P(z < Z0) = P(z < -1.79) = 0.0367 (unusual !) Calculations: Interpretation: P-value = 0.0367  so only 4% of the time could we get a more extreme value. Since this is less than α = 0.05, we reject H0 and conclude that the cream promotes quicker healing While statistically significant, a ½ day sooner healing is not practically important.

9. Significance Statistical Significance is not the same thing as practical importance. Large sample sizes can promote statistical significance with not practical importance. Remember to graphically examine your data. Outliers can make you conclude the wrong results if you blindly follow the common significance tests. Examine confidence intervals, since they provide estimates of the effect without judging whether it is too large to occur by chance.

10. Other Warnings • Over Analysis • Remember, even with a 5% level of significance, one in 20 studies is likely to show the unusual • Remember our test’s required conditions • Tests may give odd results if conditions not met • Garbage In = Garbage Out (GIGO) • Remember the population of interest and don’t generalize beyond them

11. Significance Tests Remember our 3 conditions: • SRS – simple random samples • not all of our sampling techniques fit this • Normality • samples too small for Central Limit Theorem to apply • samples with outliers or extremely skewed distributions • Independence • hard to get large samples from small populations Hawthorne effect – knowledge of a study increases short-term productivity (watching effects the watched)

12. Summary and Homework • Summary • P-values are more informative than a simple reject or fail to reject conclusion • Very small effects can be highly significant, especially with a very large sample size • Statistical significance does not mean practical importance • Significance tests are not always valid; GIGO! • Homework • pg 722 11.44 - 48