BMS 617

1. BMS 617 Lecture 14: Non-parametric tests Marshall University Genomics Core Facility

2. Parametric Tests • The following tests all assume that the data are sampled from populations in which the values are normally distributed: • Unpaired t-test • Paired t-test • Assumes that the differences within pairs are samples of a normally distributed population • ANOVA • Data which is normally distributed can be completely summarized by the mean and standard deviation • These two values completely determine the distribution • These are the parameters for the distribution • These tests work by estimating one or both of these parameters • Known as parametric tests Marshall University School of Medicine

3. Non-parametric tests Tests which make no assumptions about the distribution are known as non-parametric tests Most commonly-used forms are based on ranking (ordering) the values in the data set and analyzing only the ranks This form of test is extremely robust to outliers The following are non-parametric versions of parametric tests They are used in similar situations to their parametric versions, but make no assumptions about normality. Marshall University School of Medicine

4. Non-parametric analogs of parametric tests Marshall University School of Medicine

5. The Mann-Whitney Test The Mann-Whitney test is the non-parametric equivalent of the unpaired T-test Use when you want to compare a variable between two groups, but you have reason to believe the data is not sampled from a normally-distributed population Marshall University School of Medicine

6. How the Mann-Whitney Test works • The Mann-Whitney test works as follows: • Compute the rank for all values, regardless of which group they come from • Smallest value has a rank of 1, next smallest has a rank of 2, etc. • Choose one group: for each data point in that group, count the number of data points in the other group which are smaller • Sum these values, and call the sum U1 • Similarly compute U2, or use the fact that U1+U2=n1n2Let U=min(U1,U2) • The distribution of U under the null hypothesis is known, so software can compute a p-value Marshall University School of Medicine

7. The Wilcoxon matched-pairs signed-rank test • The Wilcoxon matched-pairs signed-rank test is used for paired data • Before and after treatment, etc. • Unlike the paired t-test, it does not assume the differences are samples from a normally- distributed population • Basic procedure: • Compute all signed differences • Rank the differences by their absolute value • Sum the ranks for the positive differences, and sum the ranks for the negative differences. • Compute the difference between these two sums of ranks • The distribution of this value under the null hypothesis is known Marshall University School of Medicine

8. Spearman’s Rank Correlation • Spearman's Rank Correlation is used to test for dependence in the ordering of two variables • The variables only need be ordinal • Do not need to be interval variables (no scale) • Works by computing the ranks of each variable • Then just compute the Pearson correlation coefficient of the ranks • Does not assume normally distributed populations • Does not test for a linear relationship • Just a monotonic (increasing/decreasing) one Marshall University School of Medicine

9. Pros and cons of non-parametric tests • Pros of non-parametric tests: • Since non-parametric tests do not rely on the assumption of normally-distributed populations, they can be used when that assumption fails, or cannot be verified • Cons of non-parametric tests: • If the data really do come from normally-distributed populations, the non-parametric tests are less powerful than their parametric counterparts • i.e. they will give higher p-values • For small sample sizes, they are much less powerful: • Mann-Whitney p-values are always greater than 0.05 if the sample size is 7 or fewer • Nonparametric Tests typically do not compute confidence intervals • Can sometimes be computed, but often require additional assumptions • Non-parametric tests are not related to regression models • Cannot be extended to account for confounding variables using multiple regression techniques Marshall University School of Medicine

10. Choosing between parametric and non-parametric tests • The choice between parametric and non-parametric tests is not straightforward • A common, but invalid, approach is to use normality tests to automate the choice • The choice is most important for small data sets, for which normality tests are of limited use • Using the data set to determine the statistical analysis will underestimate p-values • If data fail normality tests, a transformation may be appropriate • The most "honest" approach is to perform in independent experiment with a large sample to test for normality, and then design the experiment in hand based on the results of this • This is almost always impractical • For well-used experimental designs, an almost-equivalent approach is to follow customary procedure • Essentially assuming this has been carried out in some way already Marshall University School of Medicine

11. How much difference does it make? • The central limit theorem ensures that parametric tests work well with non-normal distributions if the sample is large enough • How large is large enough? • Depends on the distribution! • For most distributions, sample sizes in the range of dozens will remove any issues with normality • You will still increase your statistical power by using a transformation if appropriate • Conversely, if the data really come from a normally-distributed population and you choose a non- parametric test, you will lose statistical power • For large samples, however, the difference is minimal • Small samples present problems: • Non-parametric tests have very little power for small samples • Parametric tests can give misleading results for small samples if the population data are non- normal • Tests for normality are not helpful for small samples Marshall University School of Medicine

12. Conclusions • The bottom-line conclusion is that large samples are better than small samples • In general, the larger the better • Of course, it can be prohibitively time consuming and/or expensive to analyze large samples • If your experimental design is going to use a small sample, you need to be able to justify the data come from a normally distributed population • If this is a common experimental design that is conventionally analyzed this way, that may be good enough • For a new methodology, you should really perform an independent experiment with a large sample to test for normality first • Use the results of this to guide the data analysis for future experiments Marshall University School of Medicine

13. Computationally-intensive non-parametric methods • The non-parametric methods we examined worked by analyzing the ranks of the data • Another class of non-parametric tests is the class of computationally-intensive methods • There are two subclasses: • Permutation or randomization tests: • Simulate the null distribution by repeatedly randomly reassigning group labels • Compare the "real" data to the generated null distribution • Bootstrapping techniques: • Effectively generate many samples from the population by resampling from the original sampleLook at the distribution of summary data from the generated samples • These techniques still require a reasonable sample size to begin with • Big enough to generate enough distinct permutations or bootstraps Marshall University School of Medicine

14. Summary • Rank-based non-parametric tests are available as replacements for parametric tests • Less powerful than parametric counterparts but work when the data are not sampled from a normal distribution • Choice of test should not be automated • Should be part of experimental design and not depend on the data • Choice is less important for large data sets • But lose most power for small data sets • Permutation and Bootstrapping techniques also provide alternatives to parametric tests Marshall University School of Medicine