Multiple Testing Procedures

1. Multiple Testing Procedures Examples and Software Implementation

4. Multiple Testing in Action Examples From New Book Multiple Testing Procedures with Applications to Genomics (2007). S. Dudoit and M. J. van der Laan.

5. Multiple Testing Software R package multtest()

7. Main functions: mt.rawp2adjp() • Adjusted p-values are computed for simple (Marginal) FWER and FDR controlling procedures based on a vector of raw (unadjusted) p-values. • Possible methods • Bonferroni single-step adjusted p-values for strong control of the FWER. • Holm (1979) step-down adjusted p-values for strong control of the FWER. • Hochberg (1988) step-up adjusted p-values for strong control of the FWER (for raw (unadjusted) p-values satisfying the Simes inequality). • Sidak single-step adjusted p-values for strong control of the FWER (for positive orthant dependent test statistics). • Sidak step-down adjusted p-values for strong control of the FWER (for positive orthant dependent test statistics). • BH adjusted p-values for the Benjamini & Hochberg (1995) step-up FDR controlling procedure (independent and positive regression dependent test statistics). • BY adjusted p-values for the Benjamini & Yekutieli (2001) step-up FDR controlling procedure (general dependency structures). • Returns adjusted p-values and rank index

8. Main functions: MTP() • A user-level function to perform multiple testing procedures (MTP). • Available Tests (robust versions available for t-tests and f-tests) • One-sample t-test • Two-sample t-test (equal unequal variances, and paired) • F-test (block design as well) • lm.XvsZ : t-stat for coefficients of Xj ~Z, for each gene (Xj ) in matrix • lm.YvsXZ : t-stat for coefficients of Y~Xj + Z, where Z are additional covariates • coxph.YvsXZ: same as lm.YvsXZ but for cox proportional hazards survival models • Controls Error Rates • Fwer • gFwer • FDR • TPPFP • Multiple Testing Methods • single-step maxT • single-step minP • step-down maxT • step-down minP • Bootstrap and permutation null distributions are available. • Returns estimates, statistics, raw and adjusted p-values, etc.

9. Software Example • Objective: Identify differentially expressed genes between B-cell acute lymphoblastic leukemia (ALL) patients with BCR/ABL fusion and cytogenetically normal B-cell ALL patients • BCR/ABL is one of the most frequent cytogenetic abnormalities in human leukemia • Known to be highly expressed in chronic myeloid leukemia (CML) and acute myeloid leukemia (AML), studies are investigating its prognostic relevance in B-cell ALL patients • Identify differentially expressed genes which distinguish BCR/ABL ALL patients from normal ALL patients. • Data available online in Bioconductor experimental data package ALL • Data is reduced to only B-cell ALL samples of BCR/ABL or NEG (normal) molecular types • 79 patients total: 37 BCR/ABL and 42 NEG • Probe set (12,625) is filtered according to von Heydebreck et al. (2004), and mapped into genes  2073 genes remaining

10. Single-step maxT procedure using MTP() • Based on 2-sample Welch t-statistics and non-parametric estimation of null distribution using bootstrap sample of B=5,000 • X=gene set, Y=BCR/ABL classification, seed=999 • SSmaxT is class MTP with attributes • Summary, print, and plot methods are available

11. maxT Results • summary(SSmaxT) • print(SSmaxT)

12. plot(SSmaxT)

13. Single-step minP procedure using MTP() • If keep.nulldist=TRUE in original MTP call, to apply alternative multiple testing procedure, MTP() object can be updated • summary(SSminP)

14. minP Results • print(SSmaxT)

15. plot(SSminP)

16. Comparing Single-step minP and maxT Results • At FWER level a=0.05 • maxT identifies 13 genes • minP identifies 25 genes • 12 genes are identified by both methods

17. FWER controlling Marginal Mutiple testing using mt.rawp2adjp() • Bootstrap unadjusted p-values are provided by MTP() call (SSmaxT) • Apply Marginal FWER controlling procedures (Bonferroni, Holm, and Hochberg) using mt.rawp2adjp()

18. FWER controlling Marginal Mutiple testing using mt.rawp2adjp() • Compare the number of rejected null hypotheses and their ranks at various a cut-offs

19. Comparison Plots

21. Summary

22. MTPs

24. Acknowledgments • Sandrine Dudoit who provided the slides and examples for this presentation • Mark van der Laan

25. References for Section 3

26. References for Section 3 (con’t)

27. References for Section 3 (con’t)

28. References for Section 3 (con’t)

29. References for Section 3 (con’t)

30. References for Section 3 (con’t)

31. References for Section 3 (con’t)