1 / 15

Understanding Statistical Significance and Inference

Learn about making statistical decisions, significance levels, practical significance, type I and type II errors, power of tests, and ways to increase statistical power.

ereedy
Download Presentation

Understanding Statistical Significance and Inference

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 10.3-10.4 Making Sense of Statistical Significance & Inference as Decision

  2. Choosing a Level of Significance • “Making a decision” … the choice of alpha depends on: • Plausibility of H0: • How entrenched or long-standing is the current belief. If it is strongly believed, then strong evidence (small ) will be needed. Subjectivity involved. • Consequences of rejecting H0 : • Expensive changeover as a result of rejecting H0? Subjectivity! • No sharp border – only increasingly strong evidence • P-Value of 0.049 vs. 0.051 at the ,0.05 alpha-level? No real practical difference.

  3. Statistical vs. Practical Significance • Even when we reject the Null Hypothesis – and claim – “There is an effect present” • But how big or small is the “effect”? • Is a slight improvement a “big enough deal”? • Statistical significance is not the same thing as practical significance. • Pay attention to the P-Value! • Look out for outliers • Blind application of Significance Tests is not good • A Confidence Interval can also show the size of the effect

  4. When is it not valid for all data? • Badly designed experiments and surveys often produce invalid results. • Randomization is paramount! • Is the data from a normal population distribution? Is the sample big enough to insure a normal sampling distribution? (allows you to be able to generalize/infer about the population) • Is the population greater than ten times the sample? (affects sample st. dev.) • Individuals in the sample are independent.

  5. HAWTHORNE EFFECT • Does background music cause an increase in productivity? • After discussing the study with workers - a significant increase in productivity occurred • Problems: No control … and the idea of being studied • Any change would have produced similar effects

  6. Beware the Multiple Analyses • If you test long enough … you will eventually find significance by random chance. • Do not go on a “witch-hunt” … looking for variables that already stand out … then perform the Test of Significance on that. • Exploratory searching is OK … but then design a study.

  7. ACCEPTANCE SAMPLING • A decision MUST be made at the end of an inference study: • Fail to Reject the lot (“accept?”) • Reject the lot • H0: the batch of potato chips meets standards • Ha: the potato chips do not meet standards • We hope our decision is correct, but …we could accept a bad batch, or we could reject a good one. (both are mistakes/errors)

  8. TYPE I AND TYPE II ERRORS • If we reject H0 (accept Ha) when in fact H0 is true, this is a Type I error. (α - alpha) • If we reject Ha (accept H0) when in fact Ha is true, this is a Type II error. (β - beta)

  9. EXAMPLE 10.21 ARE THE POTATO CHIPS TOO SALTY? • Mean salt content is supposed to be 2.0mg • The content varies normally with  = .1 mg • n = 50 chips are taken by inspector and tests each chip • The entire batch is rejected if the mean salt content of the 50 chips is significantly different from 2mg at the 5% level • Hypotheses? z* values? Draw a picture with acceptance and rejection regions shaded.

  10. EXAMPLE 10.21 ARE THE POTATO CHIPS TOO SALTY? • What if we actually have a batch where the true mean is μ = 2.05mg? • There is a good chance that we will reject this batch, but what if we don’t! What if we accept the H0 and fail to reject the “out of spec … bad” batch? • This would be an example of a Type II error …accepting μ = 2 when in reality μ = 2.05

  11. EXAMPLE 10.21 ARE THE POTATO CHIPS TOO SALTY? • Finding the probability of a Type II error • Step 1 … find the interval if acceptance for sample means, assuming the μ =μ0 = 2. • … (1.9723, 2.0277) • Now find the probability that this interval/region would contain a sample mean about μa = 2.05 • Standardize each endpoint of the interval relative to μa = 2.05 and find the area of the alternative distribution that overlaps the H0 distribution acceptance interval.

  12. EXAMPLE 10.21 ARE THE POTATO CHIPS TOO SALTY? • So …  = 0.0571 … a Type II Error … we are likely to (in error) accept almost 6% of batches too salty at the 2.05mg level • And …  = 0.05 … a Type I Error … we are likely to (in error) reject 5% of salty batches at the perfect 2mg level

  13. SIGNIFICANCE AND TYPE I ERROR • The significance level alpha of any fixed number is the probability of a Type I error. That is, the probability  that the test will reject H0 when H0 is nevertheless true.

  14. POWER • The probability that a fixed level  significance test will reject H0 when a particular Ha is in fact true is called the power of the test against the alternative. • The power of a test is 1 minus the Probability of a Type II error for that alternative … • Power =1 - 

  15. INCREASING POWER • Increase alpha () …  and  “work at odds” of each other • Consider an alternative (Ha) farther away • Increase sample size (n) • Decrease sigma ()

More Related