CRIM 483

1. CRIM 483 Statistical Significance

2. Hypothesis Testing • The null hypothesis assumes that a relationship occurs simply by chance • The research hypotheses represents the more extreme outcome • The significance level or probability of an outcome determines whether the null or the research hypothesis is a more attractive explanation

3. Statistical Significance • In a comparison of adolescent attitudes toward maternal employment a significant difference is found between adolescents whose mothers worked and adolescents whose mothers did not work • Significant means that any difference between the attitudes of the two groups is due to some systematic influence and not due to chance • What is the systematic influence…having a mother who worked vs. a mother who didn’t work • Assumes that you have accounted for all other variables that may affect this relationship (i.e., the relationship is non-spurious)

4. Understanding Significance • Since the world is not perfect, we cannot be 100% sure that we are not wrong when we conclude that an outcome is significant • There will ALWAYS be a certain amount of error that cannot be controlled • Therefore, researchers use significance levels as a safety net • Significance levels=the risk associated with not being 100% confident that what you observe in a study is due to what is begin tested • p < .05 means that there is a 1 chance in 20 that any differences found were not due to the hypothesized reason but due to some other, unknown reason • The probability of observing such an outcome is less than .05 (rare); therefore, there must be an influencing factor

5. Understanding Significance, Cont’d. • A significance level is the risk a research is willing to take that he/she is wrong • p < .01 = 1% risk of being wrong • p < .05 = 5% risk of being wrong—conventional standard • p < .10 = 10% risk of being wrong • Significance levels are associated with a critical value (e.g., a z-score that represents the point at which 5% of scores rest above the value) • Any value above the critical value is considered significant because any such value will be associated with a probability that is less than .05

7. Interpreting Significance • When you conclude that an outcome is significant, you are basically saying: “This could not have occurred by chance alone—something else is going on.” • The null (assumes chance) is not the most attractive explanation • The research hypothesis (assumes difference/inequality) is a more attractive explanation for the outcome

8. Example • You want to test the difference in achievement between children who went to preschool and children who didn’t. • What is the null? • What is the research hypothesis? • You must do your best to account for all relevant factors, have the best sample, utilize the best data collection methods, etc. • If you conclude that the difference in achievement is due to preschool attendance, you still have to accept a level of risk that you are wrong • If you use p<.05, then you are implying that the degree of risk that you willing to take that you will reject a null hypothesis when it is actually true is 5%. • If, in reality, there is no difference, you have made a Type I error by rejecting the null when it was actually true

10. Different Types of Errors • Type I: Reject null when it is actually true • Significance level is associated with this error • p<.05 = There is a 5% chance that you will reject the null when it is actually true, concluding there is a difference when there really is none. • Type II: Accept null when it is actually false • Also known as a false negative • More conservative approach • Important to remember: We never really know the true nature of the null hypothesis because the population is never directly tested • Impractical • Use inferential statistics to infer information about the population from a sample

11. Controlling Error • Type I Error • Use the significance level to control the likelihood that you will make this error • Type II Error • More difficult to control • Related to sample size: As the sample size increases, the likelihood of making a Type II Error decreases • Why? Because the larger the sample, the more closely it matches the characteristics of a population • The more a sample reflects the characteristics of a population, the less likely you will accept a false null

12. Significance v. Meaningfulness • Small differences can often be significant • Group 1 (Classroom Teaching) Reading Test Average Score=75.6 • Group 2 (Computer Learning) Reading Test Average Score=75.7 • Difference=.1 and sig. at .05 level • What does it mean?

13. Determining Meaningfulness • Study has a strong conceptual base and lends meaning to the significance of the outcome • Relative impact of making a change reflected in the significant difference (would an increase in .5 point warrant an overhaul of the system) • Null results are just as important as finding significant results

14. Using Inferential Statistics • Select representative sample • Collect appropriate data from or on the sample • Use appropriate statistical test to determine if difference (outcome) is significant • Draw a conclusion about the relationship and infer knowledge about the population from what is found in the sample

15. Using Significance Testing • State the null hypothesis & research hypothesis • Set the level of risk (or level of significance) • Select the appropriate test statistic • Compute the test statistic value • Determine the value needed to reject the null hypothesis using the appropriate table of critical values for the particular statistic (i.e., identify the critical value for the level of risk selected) • Compare the obtained value to the critical value • If the obtained value is > the critical value, the null hypothesis cannot be accepted • If the obtained value doe not exceed the critical value, the null hypothesis is the most attractive explanation

16. Selecting the Appropriate Test Statistic