Download
1 / 46
460 likes | 555 Views
Psych 5500/6500. Assumptions Underlying the t Test for Independent Means. Fall, 2008. Assumptions. We will look at four assumptions, the first three are mathematical in nature: Independence of scores. Both populations are normally distributed. The two populations have the same variance.
E N D
Psych 5500/6500 Assumptions Underlying the t Test for Independent Means Fall, 2008
Assumptions We will look at four assumptions, the first three are mathematical in nature: • Independence of scores. • Both populations are normally distributed. • The two populations have the same variance. And we will add one more: • The mean is meaningful.
Assumption of the Independence of Scores This is a critical assumption in that violating it has severe consequences on the validity of the analysis.
Independence (cont.) To discuss it, let’s set up a simple experiment. Group 1Group 2 Fred Sally Ethel George The assumption of independence for this t test is that Fred’s score is independent of Ethel’s (within-group independence) and is also independent of Sally’s and George’s (between-group independence).
Independence (cont.) Group 1Group 2 Fred Sally Ethel George Technically, ‘independence’ means that how far above or below Fred’s score is from the mean of his group, cannot be predicted by knowing the same about Ethel’s or Sally’s or George’s scores.
Assumption of Normality This t test is based upon the assumption that both populations are normally distributed. The effects of failure to meet this assumption are somewhat different here than they were for the t test of a single group mean, as the sampling distribution of interest now, the one whose shape will be influenced by whether the assumptions are true or not, is that of t test for a single group t test for two independent groups
Determining Normality In a previous lecture we examined ways of determining whether or not a population is normally distributed based upon a sample. These techniques apply here. For each group we consider whether or not the population from which it was sampled is normally distributed.
Effects of Non-normality Boneau, C. A. (1960). The effects of violations of assumptions underlying the t test. Psychological Bulletin, 37, 49-64. Deviations from normality in the populations are not serious if the two populations are of the same shape, or nearly so. There seems to be little problem if both populations are symmetrical. If they are skewed then there is a problem if they are of different skewness or have different variances, as this will cause the sampling distribution to be skewed. This problem lessens under all but the most extreme cases as the size of the groups approaches 30.
There are limitations to Boneau’s approach. He only examined two-tailed tests, and as we have seen in a previous lecture the deviations of alpha from .05 due to skewness are much more severe with one-tailed tests. Also, Boneau didn’t examine the effect of violations of assumptions on the power of the experiments. In looking over the recent literature it appears that most authors advocate that when N is relatively small and the data are clearly non-normal in a way that involves skewness, that you consider either 1) normalizing the data through transformations (covered soon) or 2) using a test less affected by violations of normality (e.g. a non-parametric test, also covered soon).
Assumption of Equal Variances The t test for independent groups assumes that both populations have the same variance σ²1= σ²2 (homogeneity of variance) rather than different variances σ²1 σ²2 (heterogeneity of variance).
Detecting Heterogeneity The easiest way to detect heterogeneity is to simply compare the variances (or standard deviations) of the two groups. Of course, any difference in the variances could just be due to chance (i.e. the population variances are the same but the sample variances are not) There are statistical tests that can be used to determine from the data in the two groups whether or not the population variances differ. H0: σ1²=σ2² Ha: σ1²σ2²
Tests for Heterogeneity Two of the popular tests are the F ratio and Levene’s F. We will be covering these tests later in the semester when we have enough information to understand how they work (SPSS automatically provides Levene’s when you do a t test). The tests have the familiar problem that we can ‘reject H0’ (prove the variances differ) but we can’t ‘accept H0’ (prove the variances are the same). There is also the related issue of low power when the N’s are small and hypersensitivity to differences in variance when the N’s are very large.
Graphing Options There are at least a couple of good graphing options to see how much the variances of the two groups differ: • Scatter plots • Box plots We covered both of these in the lecture on normality, we will turn to them again but this time we will have them show both groups on the same graph, which makes it easier to see if the two groups differ in terms of variability of scores.
Graphing Options The next slide presents the data we will be graphing. To understand these graphic techniques we will first look at the control group by itself, then look at how both groups can be displayed on the same graph.
Scatter Plot First we will look at a scatter plot of the data in the control group. I particularly like this type of graph when the N of the samples is rather small. The disadvantage of the way SPSS does this type of graph is that when a score occurs more than once, the circles representing that score overlap, and you can’t tell from the graph how many times the score occurred. For example, in the following graph it is easy to see the spread of scores, but the fact that a score of ‘5’ occurred twice is concealed.
We can see in the scatter plot on the previous slide that the scores were more or less grouped around the center value, with one rather low score and one rather high score. Next we will have SPSS plot both groups on the scatter plot (next slide). Now we can see the spread of both groups, notice that the mean of the treatment group looks to be higher than the mean of the control group, and that the treatment group might have less variance than the control group.
Box Plot Now we will look at box plots. Again we will start by looking at just the control group and then look at displaying both groups on the same graph. I have decided to repeat a couple of slides from when we covered box plots before to remind you how they work.
Elements of the Box Plot The next slide shows the elements of a box plot. The plot divides the scores into quartiles (i.e. the lowest 25% of the scores, the second 25%, the third 25% and the highest 25%). Usually the lower bound marks the lowest score and the upper bound marks the highest score, but SPSS does something slightly different. The height of the box (the distance from the 25th percentile to the 75th percentile) is called the 'interquartile range', to simplify the following description we will call this one 'step'. SPSS draws the upper boundary at the highest score that is not more than 1.5 steps above the box. Any point above that is marked as either an 'outlier' (if it is between 1.5 and 3 'steps' above the box) or as an 'extreme score' (if it is more than 3 'steps' above the box). The same thing is done for the lower boundary.
The next slide shows the box plot of the control group in our example. There are no outliers or extreme scores, so the upper and lower boundaries represent the highest and lowest score. Note that we can see that the sample is slightly positively skewed (the spread of scores above the median is greater than the spread of scores below the median).
Two Groups We can ask SPSS to display the box plots for both groups on the same graph (see the next slide). We can see that the medians of the two groups are quite different (and the means probable will be too), that the groups differ in how spread out their scores are, and we can see to what degree each group is skewed. Note the bottom whisker is missing from the treatment group, this is because the lowest score is also the 25th percentile (look back at the data, there are not many scores and the lowest three scores are all 6’s, making ‘6’ both the lowest score as well as the 25th percentile).
What to do About Heterogeneity Fortunately, Monte Carlo studies have shown that the assumption of homogeneity can be violated with little effect on the validity of the t test so long as the two groups have the same or very similar sizes (i.e. when N1 N2). If the N’s differ greatly, however, then heterogeneity can seriously affect the validity of the t test. We will take a look at what to do in that case.
Student’s t test & Welch’s t test The standard t test, the one we have already covered, is sometimes known as Student’s t test. It is based upon the assumption of homogeneity of variances. A t test that does not depend upon that assumption is Welch’s t test, sometimes known as the t test for unequal variances.
Standard Error: Student’s t A difference in the two t tests is evident in their respective formulas for computing the standard error. We will begin by repeating the formula from Student’s t: then...
Standard Error: Welch’s t Welch’s t doesn’t assume that the two estimates are of the same variance, and thus doesn’t pool them: Standard error Welch’s t: Standard error Student’s t:
Welch’s t formula (t’) When N1=N2 the standard error comes out to be the same in Student’s t and in Welch’s t’, and thus the tobt is the same as well.
Welch’s t Degrees of Freedom The new standard error, however, leads to a different degrees of freedom formula. When the N’s are equal and the est.σ²’s are equal then this formula gives the same df’s as Student’s t, but when the N’s differ or the est.σ²’s differ then this leads to a smaller df than in Student’s t (and the Welch df is often not a whole number).
Type 1 Error Rate (Monte Carlo) From Coombs et al. (1996) as adapted and cited by Ruxton (2006)
Which to Use? • If σ²1=σ²2 then you may use either Student’s or Welch’s t test (but they may differ in power) • If σ²1σ²2 and N1 N2 then then you may use either Student’s or Welch’s t test (but they may differ in power). • If σ²1σ²2 and N1 N2 (differ more than a little)then you have to use Welch’s. The challenge is that we don’t know the values of σ²1 andσ²2, we only have the population variance estimates from the samples (groups) which will almost always differ at least a little.
Strategies In my review of the literature I have run across two suggested strategies: • Use the more familiar Student’s t test unless there is sufficient evidence that the two population variances differ, in which case use Welch’s t’. • Always use Welch’s t’, then you don’t have to worry about violating the assumption of homogeneity of variance.
Strategy 1 With this strategy you usually first do a Levene’s test to see if their is sufficient evidence that the population means are not equal. If Levene’s test is statistically significant (i.e. it’s p.05) then you have proof the variances are different and so you use Welch’s t’, otherwise use Student’s t. The logical problems associated with this (possible lack of power or hypersensitivity, and the inability to prove H0 is true) were discussed earlier.
Strategy 1 (cont.) There is also a problem associated with using one statistical test (e.g. Levene’s) to determine which other statistical test is appropriate to use (e.g. Student’s vs. Welch’s). When tests are linked in that way the issue of the overall probability of making an error in the analysis becomes complex. See Ruxton (2006) for references if you would like to know more.
Strategy 2 The second strategy is to simply always use Welch’s t’, as it doesn’t depend upon the assumption that the variances are equal.
Power Considerations Playing around with SPSS this is what I found (keeping the difference between the two group means the same in all cases): • When N1= N2 and est.σ²1= est.σ²2 then both t tests had the same p value. • When N1= N2 and est.σ²1 est.σ²2 then Student’s t had the lower p value. • When N1 N2 and est.σ²1= est.σ²2 then Student’s t had the lower p value. • When N1 N2 and est.σ²1 est.σ²2 then Welch’s t’ had the lower p value.
Selecting a Strategy It is a complicated choice, the advantage of Strategy 2 is that you never have to worry about whether the variances are the same or not. But, for example, if you have greatly different N’s, and the population variances are indeed the same, then Student’s t would be appropriate and it would have more power. Whichever strategy you choose the choice must be a priori, you can’t look at the analysis and then select the approach that had the lower p value.
Final Assumption: The Mean is Meaningful Quote from one of my mentors (a behaviorist involved in single-subject research): “I study the behavior of individuals, you study the behavior of means.” I’d like to end on a bit of a philosophical note. There is some reason to question whether or not the study of means is meaningful. There are decades’ (a century’s?) worth of precedence in saying that the mean is a meaningful thing to study, so I’ll take the time only to present the case against it.
Santa Claus Point one: the mean is exactly as real as Santa Claus, they are both important cultural ideas that influence our behavior but neither really exist. There are no means out there in the territory, they exist only in our models. Counterpoint: if we take anthropologist Gregory Bateson’s definition of ideas, then we would say that ideas are far more important in models of nature than are such ‘real’ things as physical forces and objects.
Individual Differences Point two: the focus on means takes us away from something much more important, individual differences. To explain this point I will pick on studies that exam gender differences. We could look at, for example, a study that shows that males have better spatial abilities than females. The point here is that the statement I just made is fundamentally inaccurate in a very important way, I should have said that the mean score of males is higher than the mean score of females. We often forget to include that information in our statements but its presence is crucial.
The t test for independent means may lead you to say that the mean of the females is lower than the mean of the males, but look at the hypothetical graph, many many females have spatial ability scores that are above the mean score of the males and many many males have scores that are below the mean of the females. The mean is a huge generalization, and like all generalizations is useful in that it makes the world much simpler but at the expense of hiding all the specific cases in which it does not apply.
Counterpoint: analyses that focus on differences between means might limit us to just understanding the behavior of means. Multiple regression (a topic we cover in the section on the Model Comparison Approach), however, provides an approach for focusing more on individual differences.
Recommended Reading Boneau, C. A. (1960). The effects of violations of assumptions underlying the t test. Psychological Bulletin, 37, 49-64. Legendre, P. & Borcard, D. Appendix: t-test with Welch correction, in Statistical comparison of univariate tests of homogeneity of variances. http://biol10.biol.umontreal.ca/BIO2041e/Correction_Welch.pdf Ruxton, G. D. (2006). The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test. Behavioral Ecology, 17, 688 - 690. Welch BL. 1938. The significance of the difference between two means when the population variances are unequal. Biometrika, 29, 350–62.
