Significance Tests

1. Significance Tests FPP 26-27

2. Significance tests • Question: • Given the collected data, is there evidence against a specified hypothesis about the corresponding parameter? • In other words, are the data consistent or not with a specified hypothesis?

3. Logic of significance tests: • Proof by contradiction: • 1. assume some hypothesis is true • 2. find a statistic (a quantity that depends on data) that takes on extreme values when assumed hypothesis is false • 3. Calculate the value of this statistic in the collected data • 4. Calculate the probability of observing a value of the statistic as or more extreme than the observed value, under the assumed hypothesis • 5. when this probability is small, one of two things happened • A. the assumed hypothesis is correct and a rare event occurred • B . the assumed hypothesis is incorrect. • since rare events are by definition rare, we interpret small probabilities as evidence that the assumed hypothesis is false. • When the probability is not small, the data provide insufficient evidence to claim that the assumed hypothesis is false.

4. Significance test for a population percentage • Civil rights and the 1960s • In the court case Swain vs. Alabama (1965), the prosecution alleged there was discrimination against black people in grand jury selection.  Census data from the time indicates that 25% of people eligible for grand jury service were black.  A random sample of 1050 people called to appear for possible jury duty contained 177 black people.   Is there evidence of discrimination? • Reference: Devore, J.  Probability and Statistics for Engineering and the Sciences.  Pacific Grove, CA: Duxbury, 2000, p. 339

5. Step 1: Formulate hypothesis • Claim: There is discrimination • The opposite of this claim is called the null hypothesis. It usually can be translated as there is nothing unusual going on. • The claim is called the alternative hypothesis. It usually can be translated as there is some unusual pattern in the data • H0: P = 0.25 vs HA: P < 0.25

6. Step2: Find a relevant statistic • Values of the sample percentage of black jurors much smaller than 0.25 suggest the null hypothesis is not true • Sample proportion = 177/1050 = 0.1689. • Is this much smaller than 0.25? • A good way to determine this is by converting the difference between 0.1689 and 0.25 to standard units

7. Step 3: Calculate z in data • We get: • The sample percentage of black jurors is six SE away from zero

8. Step 4: Calculate the p-value • When n (the sample size) is large enough, we an use a standard normal curve to calculate the probability of seeing a value of z less (i.e.as or more extreme) the observed value of -6.06 • To find the probability we need the distribution of z. Do we know it?

9. Conclusion in Swain case • Because the p-value is approximately 0, we reject the null hypothesis. It is very unlikely that we would observe a sample percentage of 16.89% or smaller if the true percentage was 0.25. The data suggest that black jurors were indeed selected less frequently than would have been expected. The data provide some evidence of discrimination.

10. Stating hypothesis • Null Hypothesis (H0) • The statement being tested in a test of significance is called the null hypothesis • Usually the null hypothesis • is a statement of “no effect” or “no difference”, • is a statement about a population, • is expressed in terms of a (some) parameter(s). • Example H0: =0

11. Stating hypothesis • Alternative Hypothesis ( Ha ) • name given to the statement we hope or suspect to be true instead of H0 • Example Ha: 0 • Hypotheses always refer to some population or model, not a particular outcome • We must decide whether the alternative hypothesis (Ha) should be one-sided or two-sided

12. Stating hypothesis • One-sided alternative hypotheses: • Example: Ha: μ< 0. Ha: μ > 0 • Two-sided alternative hypothesis: • Example: Ha: μ≠ 0

13. Stating hypothesis • Choosing one-sided or two-sided Hypothesis • The alternative hypothesis should express the hopes or suspicions we had in mind when we decided to collect the data • It is cheating to first look at the data and then frame Ha to fit what the data show • If you do not have a specific direction in advance, use a two-sided alternative

14. Stating hypothesis • Example: Your company hopes to reduce the mean time () required to process customer orders. At present, this mean is 3.8 days. You study the process and eliminate some unnecessary steps. • Q: Did you succeed in decreasing the average process time? Target: to show that the mean is now less than 3.8 days. • So alternative hypothesis is one-sided • The null hypothesis is “no change” value Ho: μ= 3.8 vs Ha: μ< 3.8

15. Stating hypothesis • The mean area of several thousand apartments in a new development is advertised to be 1250 sqft. A tenant group thinks that the apartments are smaller than advertised. They hire an engineer to measure a sample of apartments to test their suspicion. • H0:=1250 vs. Ha: <1250

16. Stating hypothesis • Experimenters on learning in animals sometimes measure how long it takes a mouse to find its way through a maze. The mean time is 18 seconds for one particular maze. A researcher thinks that a loud noise will cause the mice to complete the maze slower. She measures how long each of 10 mice takes with a noise as stimulus • H0:=18 vs. Ha: >18

17. Stating hypothesis • Last year, your company’s service technicians took an average of 2.6 hours to respond to trouble calls from business customers who purchased service contracts. Do this year’s data show a different average response time? • H0:  = 2.6 vs. Ha:   2.6

18. Test Statistic • After correctly formulating the null and alternative hypothesis we make a comparison between the hypothesized value and the data by using a test statistic. • Many test statistics can be thought of as a standardized distance between a sample estimate of a parameter and the value of the parameter specified by the null hypothesis • Most test statistics have generic form: • Test statistic for a proportion : • Test statistic for a mean:

19. P-values • A test of significance assesses the evidence against the null hypothesis and provides a numerical summary of this evidence in terms of a probability • The idea is that “surprising” outcomes are evidence against Ho • A surprising outcome is one that is far from what we would expect if Ho were true

20. P-values • A test of significance finds the probability of getting an outcome as extreme or more extreme than the actually observed outcome • The direction or directions that count as “far from what we would expect” are determined by the alternative hypothesis • Definition: The probability, assuming that H0 is true, that the test statistic would take a value as extreme or more extreme than that actually observed is called the P-value of the test • the smaller the P-value, the stronger the evidence against H0 provided by the data

21. P-values • What does “as or more extreme really mean”? • When the alternative has a > sign, “as or more extreme” means use area to the right of the test statistic in p-value calculation • When the alternative has a < sign, “as or more extreme” means use area to the left of the test statistic in p-value calculation • When the alternative uses a “as or more extreme” mean values of the test statistic far from zero in positive and negative directions. • For these type of alternative hypthoses, add areas to the left of -|test statistic| and to the right of |test statistic|

22. P-values

23. Interpretation of a p-value • Common misinterpretations of p-values • The p-value is not the probability that the null hypothesis is true. (the null is either true or not) • Also, (1-p-value) is not the probability that the alternative hypothesis is true. (the alternative is either true or not true) • Correct interpretation • The p-value is the probability of getting a value of a test statistic as or more extreme than the value of the statistic computed from the collected data, under the assumption that the null hypothesis is true

24. Enough evidence? • Below are some guidelines for judging p-values. (Don’t treat these as “golden standards”) p-value Evidence against H0 < 0.01-ish very strong > .01-ish and <.05-ish moderate > .05-ish and < .10-ish weak > .10 ish practically none

25. Etruscan example • In the eighth century B.C., the Etruscan civilization was the most advanced in all of Italy. Its art forms and political innovations were destined to leave indelible marks on the entire Western world. Originally located in the region now known as Tuscany, it spread rapidly across the Apennines and eventually overran much of Italy. But as quickly as it came, it faded. Militarily it was no match for the burgeoning Roman legions, and by the dawn of Christianity it was all but gone. • No chronicles of the Etruscan empire have ever been found, and to this day its origin remains shrouded in mystery. Were the Etruscans native Italians or were they immigrants? And if they were immigrants, where did they come from? Much of our knowledge of the Etruscans derives from archaeological investigations and anthropometric studies… (for example) body measurements to determine… origins.” (Source: Larsen and Marx, Statistics, 2001, p. 513.) • A team of archaeologists collected 84 skulls of Etruscan men and measured their head breadth (in mm). Let’s assume that these 84 men are a random sample of Etruscan men. If the Etruscan men were native, it makes sense to think that the population average head breadth of Etruscans is comparable to the head breadth of modern Italians, 132.44 mm. This assumes evolution has not shifted average head size substantially over the last 2800 years, an assumption that is reasonably close to true.

26. Exploratory data analysis for Etruscans

27. Significance test • Step1: Specify the null and alternative hypothesis • Claim: true average breadth of Etruscan heads differs from 132.44 • Ho:μ = 132.44 vs Ha: μ≠ 132.44 • Step2: compute a test statistic • The sample average is over 17 SE’s away from the hypothesized average of 132.44 • Step3: calculate the p-value • For all intents and purposes this p-value is zero why? • Step4: make a conclusion • There is enough evidence in the data to conclude that modern Italians and the Estruscans have different average head sizes.

28. A more wordy conclusion • It’s practically impossible to observe a difference of 17 SE’s by chance alone. Our initial assumption in the null hypothesis is very unlikely to be true. The data overwhelmingly suggest that modern Italians and the Etruscans have different average head sizes, indicating that Etruscans were not native to Italy. • For those interested, current theory is that Etruscans came from Asia. But, it remains a mystery how they got to Italy

29. Significance test using JMP

30. Example 1 • A sample of 40 recovery alcoholics was given the State-Trait Inventory Test. The mean score of the 40 recovery alcoholics was 38 with a sample SD of 7. A psychologist suspected that recovering alcoholics in general had a higher mean score than the norm of 35. Do the sample justify the suspicion?

31. Example 2 • There was concern among health officials in a community that an unusually large percentage of babies with abnormally low birth weight were being born. Abnormally low birth weight here is defined as less than 88 ounces. A sample of 180 births showed 14 babies with abnormally low birth weight. The proportion births that the officials expect to be abnormally low is 5%. Do the data support the health officials claims?

32. Statistical significance • To formalize testing further, some researchers advocate strict p-value cutoffs when deciding whether or not to reject null hypotheses. • Example: reject the null hypothesis when the p-value is less than 0.05. Otherwise, do not reject it.

33. Statistical significance • These cut-offs are called “significance levels”. • They are typically labeled with the Greek letter α (alpha). • Example: for a statistical significance level of 0.05, we write • α = 0.05 • When the null hypothesis is rejected, the term used to describe the outcome of the test is “statistically significant”. • Made-up example with typical language: • “We go a p-value of 0.036 and used α = 0.05. The results are statistically significant at the 0.05 level.

34. My opinion about statistical significance • DO NOT RELY BLINDLY ON A FIXED CUT-OFF • Consider two p-values: 0.050001 and 0.049999. • These two p-values provide the same amount of evidence against the null hypothesis. • But if we judge strictly by the 0.05 cut-off we don’t reject the null for 0.050001 and we do for 0.04999. • Ridiculous no? Consider p-values on their own merits

35. Type I and Type II errors • Possible errors from decision to reject or not to reject the null hypothesis • Type I error = reject when Ho is true • Type II error = fail to reject when Ha is true • Hypothesis testing is not perfect. You never know if you are making one these errors! • Important to replicate study whenever possible to reduce these errors

36. The role of sample size • The chance of a making a Type I error does not depend on sample size. (Sample sizes incorporated into test statistics). • The chance of making a Type II error decreases as sample size increases. (Be wary when using test based on small sample sizes)

37. The role of sample size • When the hypothesized value is NOT very different from the actual value of the parameter, you need a large sample size to reduce the chance of a Type II error. • In many grant proposals, you have to justify the study size by methods that attempt to minimize the chance of Type II errors. • These methods are called power analyses.

38. The role of sample size • Inferences are always improved by obtaining as much (accurate and relevant) data as possible. • With large enough sample size, you can reject any false null hypothesis • However,

39. Practical vs. statistical significance • When you get a statistically significant result, consider whether it is practically significant. • If your sample size is large enough you’ll be able to detect a difference between the hypothesised value of a parameter and its true value if Ho is wrong. • But is this difference of practical significance • Example of weight lifting study

40. Dangers of excessive fishing • With enough hypothesis tests, you’ll find something statistically significant. • Some of these statistically significant results may really be Type I errors. • Try to avoid excessive fishing for statistical significance. If you perform many tests, be sure to report how many you do. And, see if results are replicated in separate studies

41. Non-significant results • Failing to reject a null hypothesis is not a failed study • It is just as important to learn that a null hypothesis explains data well as it is to learn that it does not

42. Relationship between CI and hypothesis tests • You can use CIs like a hypothesis test • Example: Say your null hypothesis is Ho: p = 0.5. • If 95% CI does not contain null hypothesis vale, e.g. (0.64, 0.70), then the two sided test has p.value < 0.05 • If 95% CI contains the null hypothesis value, e.g. (0.47, 0.87), then the two-sided test has p-value > 0.05

43. CIs vs Hypothesis tests • Hypothesis test can identify parameter values that are inconsistent with the data. • They do not specify parameter values that plausibly could have produced the data. • Confidence intervals do this. Hence, when given a choice use CIs over hypothesis tests.

44. Important caveat • A hypothesis test will not remedy a poorly designed study • Bad data yield unreliable p-values