Statistical Decision Making

1. Statistical Decision Making • Analysts must often make decisions about some condition in the real world. • Assume that you have finished your BA in Political Science or MA in policy studies and have been hired as the environmental affairs officer for the city of Morgantown. • You must make the following assessment: • Does the water supply in Morgantown comply with safe drinking water standards? • Let’s see!

2. Hypothesis Testing We can decide the facility is In Compliance. or We can decide the facility is in Violation. (not in compliance)

3. If we decide: The facility is in compliance – and it is. or We decide the facility is in violation – and it is. Then we are fine. We have made a correct decision But sometimes… No, make that often… we’re not correct. We make mistakes! Statistics gives us some rules to reduce the likelihood of making these mistakes.

4. Example of a statistical test • In order to test the facilities water we will collect several bottles of water at different times. • We take this sample… • (Do not be confused by the chemistry. Each bottle of water is a sample to the chemist, while all of the separate bottles of water are the sample to the statistician.) • And we calculate the average amount of the pollutant – e.g. trihalomethanes.

5. The average of the samples is thus Our hypothesis is thus ppb (the standard for trihalomethanes) or 80 micrograms per Liter (15 drops in your average swimming pool) Because we will say that the standard actually refers to a population distribution with a mean equal to the standard, we can restate this is conventional statistical terms For info on EPA standards

6. An aside on Standards Drinking water standard are set by EPA to establish the Maximum Legal Contaminant Level (MCL) The level is set to the point where the concentration is expected to produce an acceptable level of morbidity & mortality “Acceptable” is a social construct, not an absolute fact or a hard fact. It a belief based on research and societal norms. Acceptable dose is inferred from LC50 What is the value of a human life What is it worth to you?

7. Hypotheses • The hypothesis is: • The facility is in violation. • Sometimes referred to as “The alternate hypothesis.” • The null hypothesis is: • The facility is in compliance. • In this instance, you want to be able to reject the alternate hypothesis, or more properly, fail to reject the null hypothesis.

8. Decision

9. Decision In Compliance

10. Decision In Compliance In Violation

11. Decision In Compliance In Violation Real World

12. Decision In Compliance In Violation In Compliance Real World

13. Decision In Compliance In Violation In Compliance Real World In Violation

14. Decision In Compliance In Violation In Compliance Real World In Violation

15. Decision In Compliance Do not reject H0 In Violation Reject H0 In Compliance Real World In Violation

16. Decision In Compliance Do not reject H0 In Violation Reject H0 In Compliance Correct Real World In Violation

17. Decision In Compliance Do not reject H0 In Violation Reject H0 In Compliance Correct Real World Correct In Violation

18. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect In Compliance Correct Real World Correct In Violation

19. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect In Compliance Correct Real World Incorrect Correct In Violation

20. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect Prob. =  In Compliance Correct Real World Incorrect Correct In Violation

21. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect Prob. =  In Compliance Correct Prob. = 1- Real World Incorrect Correct In Violation

22. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect Prob. =  In Compliance Correct Prob. = 1- Real World Incorrect Prob. =  Correct In Violation

23. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect Prob. =  (Type I error) In Compliance Correct Prob. = 1- Real World Incorrect Prob. =  Correct In Violation

24. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect Prob. =  (Type I error) In Compliance Correct Prob. = 1- Real World Incorrect Prob. =  (Type II error) Correct In Violation

25. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect Prob. =  (Type I error) In Compliance Correct Prob. = 1- Real World Incorrect Prob. =  (Type II error) Correct Prob. = 1- In Violation

26. Decision In Compliance Do not reject H0 In Violation Reject H0 Incorrect Prob. =  (Type I error) In Compliance Correct Prob. = 1- Real World Incorrect Prob. =  (Type II error) Correct Prob. = 1- (Power of the test) In Violation

27. Power of the Test (1- ) • We wish to use statistical tests that are Powerful, since they will be more likely to detect statistically significant differences. • We can improve the power of our tests by • Increasing the sample size • Choosing a statistical tests with greater power than another appropriate test • Testing in instances where the relationship, or difference, is stronger – which often means waiting until the effect has grown enough • The concept of effect size is an important related topic.

28. Effect Size • Statistical tests with large data sets will often produce results that are statistically significant. • There are tools for informing the researcher how meaningful the “difference” that is found is • Most noticeably, effect size: • Calculated several ways (Cohen’s D, Pearson’s R), effect size is ‘standardized’ with the following rule of thumb: • < 0.1 = trivial effect • 0.1 - 0.3 = small effect • 0.3 - 0.5 = moderate effect • > 0.5 = large difference effect • See Zint & Montgomery “MEERA: Power Analysis, Statistical Significance, & Effect Size”

29. Margin of error for Sampling From Wikipedia: “Margin of Error”

30. Alpha (α) • Note that alpha (α) is: • The probability of rejecting the null hypothesis when the null is in fact true. • It is the probability of making a Type I error • By convention, we usually set α=.05 (1 time out of 20 by chance alone) • A good working rule is to always use α=.05 until you know when not to….

31. Beta () • Beta () is: • The probability of failing to reject the null hypothesis when the alternate is in fact true. • It is thus the probability of making a Type II error. • We can never really know , because we never know the “true” situation. • α and  are inversely related.

32. Balancing α and  • If we have complete information then we would wish to balance the costs of potential errors with their likelihood of occurring. • The expected result of an event is the sum of the costs and benefits of that outcome times the probability of each outcome. • To balance testing. (e.g. given only 2 possible outcomes) Costs of Type I error * α = Costs of Type II error *  • In reality, we never have enough information to properly assess this.

33. Environmental vs. Industrial Protection •  provides us a measure of environmental protection. • αprovides us a measure of industrial protection. • If you increase the possibility of making one type of error, you will decrease the likelihood of the other type of error.