Some Alternative Approaches

1. Some Alternative Approaches Two Samples

2. Outline • Scales of measurement may narrow down our options, but the choice of final analysis is up to the researcher • The following are examples of how one might approach a simple situation involving two samples of data • They are not necessarily in lieu of the traditional approach or NHST, but are considered better in that framework (e.g. due to focus on effect size), but some actually do take a different approach • Some may be appropriate to both independent samples and paired samples

3. Raw Scale • Sometimes the difference in means is large enough and the scale well known enough that it would be an obviously meaningful difference to anyone with knowledge of the research situation • Example: random sample of high school and middle school students and height

4. Interval for the Mean difference • This is actually nothing different than the standard t-test and gives you the exact same information if using the same alpha • Does the interval for the difference between means contain the null hypothesized value? • If not reject • This should be part of your default reporting anyway.

5. Equivalence Testing • Recall that non-rejection of a nil null does not mean we can accept it • Absence of Evidence ≠ Evidence of Absence • We’ll talk more in detail regarding equivalence testing as it shifts focus to effect size and multiple outcomes, but the gist is that instead of establishing a difference between groups, the goal is to establish similarity • The methodological approach is that we are trying to show that the similarity seen is not due chance variation, just as before we tried to show that the difference is not due to chance variation

6. ‘Inferential’ Confidence Intervals • Can be used as part of the equivalence testing approach • Specifically they allow for tabled and graphical display that easily communicates statistical difference • Regular group mean CIs i.e. CIs for their own mean, in plots are misleading, they can overlap but the test still be statistically significant • With the ICI approach, the two CIs for the groups’ respective means are adjusted1 so that if the CIs for the mean of each group do not overlap, one knows they are statistically different at e.g. the .05 level

7. Robust t-test • A simple robust t-test would involve trimmed means and a pooled winsorized variance • As such it can easily be converted to a robust measure of Cohen’s d type effect size • Using Wilcox’s libraries (see class webpage) • yuenbt(group1, group2,tr=.2, alpha=.05,plotit=T)

8. Non-Parametric approaches • Bootstrap t-test would allow us better performance in instances of non-normality • Old methods rely on ranking the data, and as such end up in a different metric that may not be as easily interpretable • Bootstrapping can also be applied to the other approaches discussed here, e.g. a bootstrapped CI for a robust effect size.

9. Bayesian Approach • See additional specific slides (coming soon) • Gist: multiple hypotheses, prior information, posterior probabilities of the hypotheses, ‘updating’

10. Effect size • Effect size and an interval estimate of it provides an NHST approach but with entire focus shifted to effect size • Example vs. the nil null: does the interval contain zero? • If not reject • Also measures of overlap, to be talked about in more detail elsewhere, frame the effect size in terms of group overlap

11. Graphical Approach • Often group comparisons are made with a bar graph and error bar, which is not only very dull, but contains no information about the distribution of scores, and is often using the ‘wrong’ CIs to convey group differences • Better approaches require a program that’s capable of decent graphics

12. Regression • Using the group membership as the predictor • As we have noted, proper coding insures the exact same information will result as the t-test itself

13. Shifting Prediction to Group Membership • Instead of evaluating mean differences, predict group membership (typically with several predictors) • Candidate analyses • Logistic Regression • Discriminant Function Analysis

14. Vector Difference Norm • Paired or equal N independent samples • Example data: Total taxation from labor and consumption as %GDP • Basic Question: How much have taxation rates changed over this time period? • (1981 = 38, 1995 = 40) • Mean difference = 2.0 • t(17) = -2.55, p = .021 • Problem: Some go up, don’t change, or even go down. And the average doesn’t give us a sense of ‘net change’ • One goes up 2 another down 2, and so you’re going to say the average difference is zero? • How might we get such an estimate? • Employ the difference norm1

15. Vector Difference Norm • Multiplying a vector requires that we multiply the column by a row of the same number of values1 • The “’” indicates the vector has been transposes (i.e. flopped onto its side) • The first value of 260 in and of itself isn’t interpretable • However, if we took the 1981 values, raised those by 10% and performed the same operation we did using the 1995 values, we’d come to nearly the same result (265.8) • Interpretation: The difference from 1981 to 1995 is equivalent to a 10% increase to all their values

16. Gist • Theory, the goals of research and the communication of ideas drive the analysis more than data type • As has been mentioned before, learning statistics is like learning a new language, and you’ll want to increase your vocabulary • There are always several options within any research setting for viable and appropriate analysis • Using multiple, unique, and/or uncommon methods may be necessary to communicate the specific understanding you have of the information you’ve collected.