BMS 617

1. BMS 617 Lecture 7 – T-tests Marshall University Genomics Core Facility

2. T-tests • T-tests refer to a family of statistical tests, in which a mean, or difference in two means, is assumed to be sampled from a T-distribution • As a statistical test, T-tests compute p-values • The probability of seeing a mean, or difference of means, this large, assuming the null hypothesis is true • Interpreting a t-test involves knowing the null hypothesis Marshall University School of Medicine

3. Types of T-test • One-class T-test: • Null hypothesis is that the mean of a set of values is equal to some fixed value • Two-class T-tests: • Two groups of values • Unpaired T-test • Most common T-test • Null hypothesis is that the values in each group are sampled from distributions with equal means • Paired T-test • Each sample in the first group is paired with a sample in the second group • Null hypothesis is that the mean of the difference between each pair is zero Marshall University School of Medicine

4. Example: One-class T-test • Recall our body-temperature data from earlier • n=130 samples of body temperature • Mean m=36.82C, SD s=0.41C • We wanted to use this to test the hypothesis that mean body temperature was μ=37C • Under the assumptions of the one-class t-test, the value t=(μ-m)/(s/√n) follows a t-distribution with n-1 degrees of freedom • For this example, t=5.006 Marshall University School of Medicine

5. Computing a p-value for a given t • To get a p-value for t=5.006, we either use software or a table • Need to know the degrees of freedom • In this case, d.f.=130-1=129 • Tables either give the probability value for a given t and df, or critical t-values for given probabilities and df • From tables or software, the probability that t<=5.006 is p=0.9999991 • We want to know the probability of seeing a result this extreme, assuming the mean is 37C, i.e. assuming t follows a t-distribution with 129 d.f. • This is the probability either that t>5.006 or that t<-5.006 • A two-tailed, one-class t-test • P(t>5.006)=1-0.9999991=0.0000009 • So P(t<-5.006)=0.0000009 • and p=2 x 0.0000009 = 0.0000018 Marshall University School of Medicine

6. Assumptions for a one-class t-test • The one-class t-test is accurate under the following assumptions: • The samples are random (or representative) • The observations are independent • The data are accurate • The data are sampled from a population that is normally distributed Marshall University School of Medicine

7. One-class T-test and the Confidence Interval • We saw earlier how to compute a confidence interval for the mean of this data set • Calculate w=t*s/√n • The confidence interval is from m-w to m+w • t* is the value from the t-distribution for which P(t>t* or t<-t*)=1-confidence • e.g. for a 95% confidence interval we want P(t>t* or t<-t*) =0.05 • So P(t>t*)=0.025. From tables or software, t*=1.979. • This gives a 95% confidence interval of [36.75, 36.89] • Knowing the 95% confidence interval does not contain the null hypothesis value of 37 is equivalent to knowing the p-value is less than 0.05 Marshall University School of Medicine

8. Two-class unpaired T-test: Example • For an example of a two-class, unpaired T-test, consider the GRHL2 expression data we saw earlier from Cieply et al., Cancer Research 2012. • Compared expression of GRHL2 in different breast cancer cell lines, classified as Basal A, Basal B, or Luminal. • Compare the Basal A expression to the Basal B expression Marshall University School of Medicine

9. GRHL2 Expression Data Marshall University School of Medicine

10. GRHL2 Expression Data Marshall University School of Medicine

11. How an unpaired t-test works An unpaired t-test works by computing the difference of the means of the two samples Assuming the null hypothesis – that the difference of the two means is zero – the difference of the sample means, divided by a pooled standard error of the mean, will be distributed with a t-distribution Marshall University School of Medicine

12. Unpaired t-test for the GRHL2 data Marshall University School of Medicine

13. Assumptions for the unpaired T-test • The unpaired T-test works along essentially the same assumptions as the one-class T-test: • The samples are random or representative • The observations are independent • The data are accurate • The values in the populations are at least approximately normally distributed • Additionally, the t-test we used here assumes: • The the populations have the same standard deviation Marshall University School of Medicine

14. Assumption of equal variances • In the Basal A vs Basal B GRHL2 comparison, the Basal B samples have higher SD (0.7859) than the Basal A samples (0.4463) • The t-test we ran assumed the samples came from populations with equal variances (i.e. equals standard deviations) • A test can be run to see if the data are consistent with the assumption of equal variances • The distribution of the square of the ratio of the standard deviations is known under the assumption that the population variances are equal Marshall University School of Medicine

15. If the assumption of equal variances is violated • A modified t-test can be used, which doesn’t make the assumption of equal variances • Called the “Welch T-test” • Has less power than the standard unpaired t-test • As usual, testing your data set in order to decide which test to use can give misleading results • Typically will give over-optimistic p-values • In the ideal world, we would run an experiment specifically to determine if the assumption of equal variances holds, then use that to determine how to analyze our real experiment Marshall University School of Medicine

16. Rules of thumb for the assumption of equal variances • Unequal variances will only badly affect the t-test if the number of samples in each group is small and unequal • In other cases the t-test is very robust to violations of this assumption • In practice, I do the following: • If the number of samples is equal, I use the regular t-test • If the number of samples in both groups is at least 5, no matter if they are equal, I use the regular t-test • If there is reason to believe the variances should be equal (e.g. if all the variance comes from technical replicates), I use the standard t-test • Otherwise, I use the Welch T-test Marshall University School of Medicine

17. 95% Confidence Intervals and Unpaired t-tests Marshall University School of Medicine

18. 95% Confidence Intervals and unpaired T-tests • The unpaired t-test results computed the difference between the means and the 95% confidence interval for that difference • For this example the 95% confidence interval of the difference of the means was [-2.373, -1.348] • If this confidence interval doesn’t contain zero, this is equivalent to p<0.05 • We can also compute the confidence interval for each mean independently • If these confidence intervals do not overlap, then the p value is definitely less than 0.05 • In fact, it must be a lot less… • If the confidence intervals do overlap, then p may or may not be less than 0.05 • Cannot deduce anything from the error bars in this case Marshall University School of Medicine

19. Other error bars and statistical significance • What if the bar chart uses SD or SEM for the error bars • SD tells us about the amount of scatter in the data • Nothing about the precision with which the mean is measured • Overlapping, or non-overlapping, SD error bars have nothing to do with statistical significance • SEM measures the precision with which we approximate the mean • But interpretation depends on knowing the sample size • We can deduce the following: • If SEM error bars overlap, then the difference is definitely not statistically significant at p=0.05 (in fact, p is much bigger…) • If SEM error bars do not overlap, the p may or may not be less than 0.05 Marshall University School of Medicine

20. Error bars and statistical significance summary Marshall University School of Medicine

21. Paired t-tests • Paired t-tests are used when the comparison is between samples which are paired between the groups • Before and after treatments on a set of patients • Pair the “before value” on patient A with the “after value” on patient A • The “before value” on patient B with the “after value” on patient B, etc • Studies in which subjects are recruited to two groups in a matched fashion • Match a control patient with a treatment patient based on age, sex, weight, height… • Difficult type of study to perform • Twin or sibling studies • Lab experiments in which treated and control samples are handled in parallel • Plate cells, divide into two, treat one half and use the other as control • Repeat the next day with another plate, etc Marshall University School of Medicine

22. Example • In a recent experiment, we performed expression profiling on a set of eight mice • Should not use t-test here without correcting for multiple hypotheses, but this is a good example for demonstration • Four litters of mice were bred, and two male mice selected (at random if necessary) from each litter • One mouse from each pair was treated and one was used as a control • Analyzing these data with a paired test has the potential to eliminate any litter-litter variation Marshall University School of Medicine

23. Example data • Actual data are read counts for the gene of interest • After sequencing, align all reads to the genome • Count the number of reads that align to each gene for each sample Marshall University School of Medicine

24. Paired t-test • There are two distinct null hypotheses we can make about paired data • Both say “the data is no different between the two groups” • One is that the difference between the values in the groups is zero (“paired t-test”) • The other is that the ratio of the values in the groups is 1 (“ratio paired t-test”) Marshall University School of Medicine

25. Paired T-test results Marshall University School of Medicine

26. How a paired t-test works • A paired t-test is really just a one-class t-test! • Computes the differences for each pair • And then tests the null hypothesis that those differences are samples from a normal distribution with mean zero • The confidence interval is just the confidence interval of the mean differences Marshall University School of Medicine

27. Ratio paired t-tests • The ratio paired t-test tests the null hypothesis that the ratio of the paired values is 1 • This is done simply by a mathematical trick: take the log of all the ratios • Then perform a regular paired t-test with the log ratios instead of the differences • Log(1)=0 • Software performing a ratio paired t-test takes care of computing the logs, performing the test, and then transforming the mean difference of log ratios, confidence interval of that mean, etc, back to ratio values Marshall University School of Medicine

28. Graphing paired data • Plotting bar charts, or even column-scatter plots, of paired data does not show the pairing between the data values • A better presentation is a connected column scatter plot • Column scatter plot with lines connecting the paired data points Marshall University School of Medicine

29. Connected column scatter plot Marshall University School of Medicine

30. t-test summary • t-tests are a family of statistical hypothesis tests • Generate a p-value • Remember how to interpret! • Null hypotheses: • For a one class t-test, the null hypothesis is that the samples are drawn from a normally-distributed population with a specified mean • For an unpaired t-test, the null hypothesis is that the samples are drawn from two normally-distributed populations with equal means • For a paired t-test, the null hypothesis is that the differences between matched values are samples of a normally-distributed population with mean zero • Equivalent to a one class t-test on the differences between matched values Marshall University School of Medicine