Tests of Significance and Measures of Association

1. Tests of Significance and Measures of Association

2. Definitons • Test of Significance – Given a random sample drawn from a population, a test of significance is a formal test evaluating the probability that an event or statistical result based on the sample could have happened by random chance. • With most tests of significance we look for low p-values. The lower the p-value, the lower the probability that the event or result could have happened by random chance. What a p-value tells us is that “1-the p-value” percent of the possible samples we could have drawn from the population would contain our estimate.

3. Null and Alternative Hypotheses • Null hypothesis – Often labeled H0. One perspective: an estimated result is due to random sampling error. A statement that the estimated parameter or parameter difference is random in the population. • Null hypothesis – Another perspective is that the estimated result is consistent with some externally imposed value, say due to theory or prior research. • Alternative hypothesis – Often labeled Ha. Your estimated result. If we are to have confidence in an estimated result, then we need evidence that the result could not have resulted from random chance.

4. In inferring from a sample to a population, we can be wrong in either of two ways. • False rejection error. Falsely rejecting a true null hypothesis. This type of error is usually called Type I Error. Here the researcher concludes there is a relationship in the population, when there is actually none. • False acceptance error. Falsely accepting a false null hypothesis. This type of error is usually called Type II Error. Here the researcher concludes there is no relationship in the population, when there actually is one. • Which type of error is worse (if you don’t want to mislead science)? Type I error is worse. Why? • Consider the following graph as a way of organizing your thoughts on Type I and II error.

6. Comparing the Means of Two Groups • Consider the variables from the National Election Studies which give respondents’ feeling thermometer about the Democratic party. • Men and women differ in their feelings about the Democratic party. However, in order to test the significance of these differences we need the standard error of the mean differences. If the variances of the two samples are the same, then we can simply calculate these from the standard errors for men and women. • Square the two estimates of the standard error for the two samples for men and women. Sum the squared standard errors. Take the square root of the sum of the squared standard errors. • That is,

8. Two Approaches to Hypothesis Testing • Confidence Interval Approach- Here we use the standard error to determine the smallest plausible mean difference in the population. Or • P-value approach- Here we determine the exact probability of obtaining the observed sample difference, given that the null hypothesis is true.

9. Look up the appropriate p-value in the distribution. The t-distribution depends on the degrees of freedom. Last week we estimated one parameter, so the degrees of freedom was N-1. Here we are estimating two parameters so the degrees of freedom is N-2.Alternatively, let STATA compute the p-value. It returns a p-value of 0.0005. In other words, given the sample drawn, there is only a 0.0005 chance that the true mean difference in the population is zero. Said differently, only 5 in 10,000 of the samples drawn from the population would report a mean of zero.

10. Comparing Sample Proportions • The standard error of a difference in proportions is easy to compute.Let p1 and p2 be the proportions for two groups. Then, let q1=(1-p1) and q2=(1-p2) be the complements of these two proportions. Then, the standard error of the difference in proportions is given

11. Example

12. Is this a statistically significant difference? Using the rule of thumb that if the estimated coefficient is roughly twice the standard error, the difference is statistically significant. • We observe an estimated coefficient (0.093) about 3 times as large as it’s standard error (0.031).

13. Chi-square Test of Significance • A Chi-square test evaluates whether the observed dispersion of cases deviates from what one would expect if the null hypothesis were correct. • The Chi-square test is commonly used when one is conducting a cross-tabular analysis. • The test is based on the difference between the observed frequencies in a table versus the expected frequency if there is no relationship.

14. Example:

16. Summing the numbers in bold the table, we have a Chi-squared statistic of 15.2. We compare this statistic to a table of Chi-squared statistics with degrees of freedom equal to (r-1)(c-1)

18. Measures of Association • Statisticians have developed a number of measures of association for evaluating the strength of relationships. • PRE- Proportional Reduction in Error. PRE measures give a number that ranges from 0 to 1. 0 represents no association and 1 represents a perfect association. • Lambda and Somers d are two different measures of association, commonly used in evaluating relations in a cross-tabular analysis.

19. Here there are 226+358=584 errors in prediction when hypothesizing that gender makes a difference. Calculate

20. Somers dyx- An alternative measure of association which counts concordant and discordant pairs.

21. Clowfrequent=7*5+7*6=77; Clowoccasional=5*6=30 Thus, there are 77+30=107 pairs that are concordant with there being a positive relationship. Work diagonally and down. Dhighfrequent=3*5+3*4=27; Dhighoccasional=5*4=20 Thus, there are 47 pairs that are discordant with there being a positive relationship. Work diagonally and up. There are 60 more pairs that fit a positive than negative relationship. However, we need the number of ties to convert this number to a measure of association from 0 to 1. Tlow=7*5+7*4+5*4=83; Thigh=3*5+3*6+5*6=63; T=146 Somers dyx=(107-47)/(107+47+146)=0.20