Chapter Nine: Evaluating Results from Samples

1. Chapter Nine: Evaluating Results from Samples • Review of concepts of testing a null hypothesis. • Test statistic and its null distribution • Type I and Type II errors • Level of significance, probability of a Type II error, and power.

2. Objective is to develop concepts that are useful for planning studies. • Choose test statistic (usually this is a sample mean) • Balance two types of errors by choice of α (probability of a Type I error) and β (probability of a Type II error) • Determine sample size so that α and β specifications are met. • Even arbitrary rules have defined α and β.

3. Statistical Sampling of Populations • ASSUME there is a defined population and that we want to estimate such parameters as the mean of the population. • Observe a small number of randomly selected members from the larger population. • Apply appropriate statistics to the sample to generate correct conclusions.

4. Why random samples? • Other sampling procedures have failed consistently. • 1932 Literary Digest Poll. • 1948 Opinion Polls of Outcome of Presidential race. • Random samples, correctly drawn, have known properties. • Value of theorems and mathematical argument.

5. Procedure of Examining a Test of a Hypothesis • Start with question. • Is a student a random guesser or knowledgeable about statistics? • Specify a null hypothesis • H0: Student is a random guesser. • Null hypothesis should be plausible. • Probability measures generated by null hypothesis should be easy to compute with.

6. Procedure of Examining a Test of a Hypothesis • Specify a statistic, a sample size, and a rule. • Statistic=S4, number of correct answers in a four question true-false test, with questions of equal difficulty in random order. • Sample size is the length of the test, 4 questions (an unreasonably short examination). • Rule is to reject H0 when S4≥4. • The value 4 is the “critical value”. • This is an example of a “binomial test.”

7. Procedure of Examining a Test of a Hypothesis • Determine the properties of the procedure. • Specify the null distribution of your test statistic. • Specify the alternative that you want to be able to detect with low error rate. • Student who has an 80 percent chance of correctly answering each question.

8. Type I error • Definition: to reject the null hypothesis when it is true. • This example: Call a random guesser a knowledgeable student. • Can a Type I error happen? • Yes; anytime a random guesser gets 4 or more correct answers (that is, exactly 4 correct). • Hopefully, the probability of a Type I error is small.

9. Level of Significance • Definition: The level of significance is the probability of a Type I error (reject the null hypothesis when it is true) and is usually denoted by α. • That is, α=Pr0{Reject H0} • Example problem: α=Pr0{S4≥4} • Notice that α is a right sided probability. • Problem is to find α.

10. Cumulative Distribution Function • Use the cumulative distribution function (cdf) to get the answer. • Definition: The cdf of the random variable X at the argument x (FX(x)) is the probability that the random variable X≤x; that is, FX(x)=Pr{X≤x}. • Use table look-up on reported cdf to get answer.

11. CDF of Test Statistic under Null and Alternative Distributions

12. Finding Level of Significance α • α=Pr0{Reject H0} • In this problem, α=Pr0{S4≥4} • This is a right-sided probability. • Since cdf’s are left-sided probabilities, use the complement principle. • Pr0{S4≥4}=1-Pr0{S4≤3}=1-F0(3)=1-0.9375 • The answer is 0.0625.

13. Type II error • Definition: to accept the null hypothesis when it is false. • This example: Call a knowledgeable student a random guesser. • Can a Type II error happen? • Yes; anytime a knowledgeable student makes a mistake. • Hopefully, the probability of a Type II error is small.

14. Probability of a Type II error β • Definition: The probability of a Type II error (accept the null hypothesis when it is false) is usually denoted by β. • That is, β=Pr1{Accept H0}. • Example problem: β=Pr1{S4≤3}=F1(3)=0.5904.

15. Summary of Findings • The probability of a Type I error α=0.0625 • The probability of a Type II error β=0.5904 for a student who is able to answer 80 percent of the true-false questions correctly. • The value of α is within reasonable norms. • The value of β is unreasonably large. • The test procedure is not satisfactory.

16. Further Interpretation of error rates • The Type I error rate α should be small and roughly equal to β. • Minimax argument assuming cost of a Type I error is same as cost of a Type II error. • Increasing the sample size while holding α constant will reduce β. • There is a tradeoff of α and β. • Changing the design of the measuring process may lead to reduced error rates.

17. Power • Definition: The power of a procedure is 1-β. • Error rates should be small • Power should be large.

18. Additional Definitions • Statistic: a random variable whose value will be completely specified by the observation of an experimental process. • Parameter: a property of the population being studied, such as its mean or variance. • Standard error: standard deviation of a statistic.

19. Observed significance level. • Definition of observed significance level: a statistic equal to the probability of observing a result as extreme or more extreme from the null hypothesis as observed in the given data. • Three types of observed significance level: right, left, and two-sided.

20. Finding the observed significance level in this example. • A student takes the four question true-false test and has two answers correct. What is the observed significance level? • What side? Here, right side. • Apply definition of observed significance level: osl=Pr0{S4≥2}.

21. Finding the observed significance level in this example • Use complement rule to get to a left sided probability. • Pr0{S4≥2}=1- Pr0{S4≤1}=1-F1(1). • The answer is 1-0.3125=0.6875. • This is a statistic whose value is the probability that a random guesser does as well or better than the student who took the test.

22. Summary of Lecture • We have reviewed the basic concepts of tests of hypotheses. • We have shown how to find the error rates and observed significance level in a binomial test from a tabulation of the cdfs. • Interpretation point is that error rates should be small. Increase the sample size, change the design, or change the specifications if error rates are too large.