hypothesis testing with special focus on simulation

1. hypothesis testing with special focus on simulation Hypothesis Testing for Simulation

2. Hypothesis Test answers yes/no question with some statistical certainty • H0 = default hypothesis • is a statement • Ha = alternate hypothesis • is the precise opposite Hypothesis Testing for Simulation

3. X = test statistic (RANDOM!) • sufficient (uses all avail. data) • often Z, T, N are used as notation • FX = its probability distribution • a = P[reject H0 | H0 true] Hypothesis Testing for Simulation

4. ca = critical region for a • a = P[X in ca | H0] • a is our (controllable) risk Hypothesis Testing for Simulation

5. TWISTED LOGIC • We WANT to reject H0 and conclude Ha, so... • We make a very small, so... • If we can reject, we have strong evidence that Ha is true • This construct often leads to inconclusive results • “There is no significant statistical evidence that...” Hypothesis Testing for Simulation

6. IMPORTANT • Inability to reject <> H0 true Hypothesis Testing for Simulation

7. POWER OF THE TEST • b = P[X not in ca | Ha] • 1-b = P[correctly rejecting] Hypothesis Testing for Simulation

8. VENACULAR • a is type I error • Probability of incorrectly rejecting • b is type II error • Probability of incorrectly missing the opportunity to reject Hypothesis Testing for Simulation

9. UNOFFICIAL VENACULAR • type III error – answered the wrong question • type IV error – perfect answer delivered too late Hypothesis Testing for Simulation

10. EXAMPLE! • Dial-up ISP has long experience & knows... Hypothesis Testing for Simulation

11. We get DSL, observe 12 samples Hypothesis Testing for Simulation

12. IS DSL FASTER? • H0: mDSL = 50 • Ha: mDSL < 50 • test with P[type I] = 0.01 Hypothesis Testing for Simulation

13. PROBABILITY THEORY • Z ~ tn-1 • Must know the probability distribution of the test statistic IOT construct critical region Hypothesis Testing for Simulation

14. for n = 12, a = 0.01, ca = -2.718 99% of the probability above -2.718 Hypothesis Testing for Simulation

15. our test statistic -2.33 Hypothesis Testing for Simulation

16. -1.796 (0.05) -2.33 (0.021) -2.718 (0.01) • 0.021 called the p-value • Given H0, we expect to see a test statistic as extreme as Z roughly 2% of the time. Hypothesis Testing for Simulation

17. CONFIDENCE INTERVALS m Based on the sample So they are RANDOM! la ua • For a given a • P[la <= m <= ua] = 1-a Hypothesis Testing for Simulation

18. GOODNESS-OF-FIT TEST • Discrete, categorized data • Rolls of dice • Miss distances in 5-ft. increments • H0 assumes a fully-specified probability model • Ha: the glove does not fit! Hypothesis Testing for Simulation

19. TEST STATISTIC “chi-squared distribution with gnu degrees of freedom” Hypothesis Testing for Simulation

20. n = observations - estimated param • Did you know... if Zi~N(0, 1), then Z12+ Z22+...+ Zn2 ~ cn2 Hypothesis Testing for Simulation

21. CELLS • H0 always results in a set of category cells with expected frequencies • EXAMPLE • Coin is tossed 100 times • H0: Coin Fair Hypothesis Testing for Simulation

22. CELLS AND EXPECTED FREQUENCIES Hypothesis Testing for Simulation

23. EXAMPLE • Cannon places rounds around a target • H0: miss distance ~ expon(0.1m) • Record data in 5m intervals • (0-5), (5-10), ...(25+) Hypothesis Testing for Simulation

24. EXPONENTIALS E(X)=1/l Hypothesis Testing for Simulation

25. RESULTS Hypothesis Testing for Simulation

26. Hypothesis Testing for Simulation

27. TEST RESULTS • Degrees of Freedom • 6 cells • 0 parameters estimated • n = 6 • For the c62 distribution, the p-value for 14.14 is about p=0.025 • REJECT at any a > 0.025 Hypothesis Testing for Simulation

28. DIFFERENT H0 • H0: the miss distances are exponentially distributed • Ha: the exponential shape is incorrect • We estimate the parameter, we lose one degree of freedom Hypothesis Testing for Simulation

29. RESULTS 2 Hypothesis Testing for Simulation

30. Hypothesis Testing for Simulation

31. n = 5 • p-value for 7.83 is larger than 0.05 • CANNOT REJECT • CONCLUSION? Hypothesis Testing for Simulation

32. SIMULATION vs. STATISTICS • Statistics • Sample is fixed and given • Conclusion is unknown • Significance is powerful • Simulation • Sample is arbitrarily large • Conclusion is known • We need another thought about what is meaningful Hypothesis Testing for Simulation

33. SAMPLE SIZE EFFECT m = 100 s = 10 Hypothesis Testing for Simulation

34. HOW LARGE IS A DIFFERENCE BEFORE IT IS MEANINGFUL? Hypothesis Testing for Simulation

35. SUMMARY • You probably knew the mechanics of HT • You might have a new perspective Hypothesis Testing for Simulation