S519: Evaluation of Information Systems

1. S519: Evaluation of Information Systems Social Statistics Inferential Statistics Chapter 8: Significantly significant

2. Last week

3. This week • What is significance and why it is important • Type I and Type II errors • How inferential statistics works • How to select the proper statistical test for your research

4. The concept of significance • Significance • Any difference between the attitudes of the two groups is due to some systematic influence and not due to chance.

5. Significance • Example • Hypothesis: There is a significant difference in attitude toward maternal employment between adolescents whose mothers work and adolescents whose mothers do not work, as measured by a test of emotional state. • There are many other reasons to affect this hypothesis, for example?

6. Significant level • Significant level is the risk associated with not being 100% confident that the null is true (there is no difference between data or variables) • If the significant finding occurred at the 0.05 level (p<0.05), this means that there is 1 chance in 20 (5%) that there is no difference in data (null is true): any differences found were not due to the hypothesized reason, but to some other unknown reasons, or by chance.

7. Significant level • p-value • The probability for the null hypothesis to be true • The probability for no difference in data or variables • p>0.05 (non significant): more than 5% chance (5% to 99%) that the null is true (no difference in data) – accept null • p<0.05 (statistically significant): less than 5% chance that the null is true (no difference in data) – reject null

8. Null and research hypothesis • Research hypothesis • There is a difference in the academic achievement of children who participated in a preschool program and children who did not participate. • Null hypothesis • The two groups are equal to each other on some measure of achievement. • As a good researcher, your job to show that any difference that exists between these two groups is due only to the effects of the preschool experience.

9. Statistical significance • A result is called statistically significant  it is unlikely to cause by chance • A statistically significant difference  there is statistical evidence that there is a difference • It does not mean that the difference is necessarily large, important, or significant in common understanding

10. Null hypothesis • Either true or false • But null cannot be tested directly (as it is applied to the population) • The researchers do not know the real true nature of the null hypothesis, and it is hard to know and to test • That is why we need inferential statistics

11. Table 8.1 (s-p179)

12. Type I error • Defines the risk that you are willing to take in any test of the null hypothesis • Conventional: 0.01 ~ 0.05 • Example: if the level of significance is 0.05  there is a 5% chance you will make the Type I error: to reject it when the null is true. • It is not proper to say “on 100 tests of the null hypothesis, I will make an error on only 5” • As it is normally associated with one test

13. Type I error • p<.05 or p<.01 (reject the null) • The risk to make the Type I error (reject the real true null) is less than 5% or 1% chance • p>.05 or p=n.s. (nonsignificant) (accept the null) • The probability to make the Type I error (reject the real true null) exceeds .05

14. Type II error • The risk of accepting the real false null hypothesis • It is sensitive to the number of subjects in a sample • The size of sample increases  Type II error decreases • If the sample is more closer to the population, the likelihood that you will accept a false null hypothesis decreases

15. Significance • Statistical significance means • Example: Group 1 with training to read using computer, Group 2 with training to read using classroom teaching • A reading test: Group 1=75.6, Group 2=75.7 • When using t test: result is statistically significant at the .01 level • How to interpret: computers do better than classroom teaching

16. Inferential statistics • Descriptive statistics: describe the characters of a sample • Inferential statistics: infer something about the population based on the sample’s character

17. How inference works • Mother-work group and mother-not-work group • Select representative samples of two groups • Conduct a test for each member in these two groups, calculate the mean scores • Select a proper statistical test • Draw a conclusion (to a population): • If statistically significant: the difference is due to moms • If not significant: the difference is not due to moms

18. How to select which test • Flow chart (s-p186) • http://rimarcik.com/en/navigator/

19. A template for significant test • 1. a statement of the null hypothesis • 2. setting a level of risk associated with the null hypothesis (level of significance or Type I error, p) • 3. select a proper statistical test (see Fig 8.1) • 4. set up the sample and experiment, and compute the test statistic value • 5. determine the value needed for rejection of the null hypothesis using proper tables – critical value (see appendix) • 6. compare the computed value and the obtained value • 7. if computed value > critical value: reject the null; if computed value < critical value: accept the null

20. Exercise • Are the following statements true, and why: • A Type I error of 0.05 means that in 5 tests out of 100 tests of the research hypothesis, I will reject a true null hypothesis • It is possible to set the Type I error to 0 • The smaller the Type I error rate, the better the results