Accelerating Sparse Canonical Correlation Analysis for Large Brain Imaging Genetics Data

1. Accelerating Sparse Canonical Correlation Analysis for Large Brain Imaging Genetics Data Jingwen Yan, Hui Zhang, Lei Du, Eric Wernert, Andew J. Saykin, Li Shen

2. Outline • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results

3. Imaging Genetics Behavior: Disorders, Complex interactions, phenomena, diseases. UCI, S. Potkin et al. Systems Genes Cells

4. Imaging Genetics Underlying Biological Pathway and Mechanism

5. Imaging Genetics Biological Pathway Candidate Gene/SNP Genome-wide SingleROI Circuit Whole Brain

6. Outline • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results

7. SCCA X1 Y1 X1 Y1 X1 Y1 X2 Y2 X2 Y2 X2 Y2 W’X Xu Yv X3 Y3 X3 Y3 X3 Y3 Xn Yn Xn Yn Xn Yn Massive UnivariateAnalysis Multivariate Multiple Regression Canonical Correlation Analysis

8. SCCA • Sparse canonical correlation analysis (SCCA) • R package: Penalized Multivariate Analysis (PMA) (Witten, et al, 2009) • X, Y : imaging and genetics data respectively • : sparse penalties, mostly norm • For simplicity, assuming and • Bi-convex and non differentiable problem • Iterative solution = 1, = 1

9. SCCA • Sparse canonical correlation analysis (SCCA) • Problem • Iterative solution • , ) is the soft thresholding operator and is chosen so that = 1, = 1, 1. = 1, 2. = 1,

10. Outline • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results

11. Computational Challenges • Example SCCA run at a small scale • Participants: 1000 • Genotype: 3,200 SNPs • Phenotype: 10,000 voxels • Permutation: 10,000 permutation tests • Running time: more than 12,000 hours • Scale up • Genotype (array): 6M SNPs • Genotype (NGS): 40M variants • Phenotype: 200K voxels, imaging, cognitive and biomarker • Permutation: 10M permutation to reach p=10-7 • Parameter tuning via cross-validation • 10-fold cross-validation coupled with an 11-by-11 grid search • SCCA runs: 10×11×11 = 1,210

12. Acceleration with MKL • Intel Math Kernel Library (MKL) • accelerate application performance and reduce development time • highly vectorized and threaded linear algebra, fast fourier transforms (FFT), vector math and statistics functions • MKL has been optimized to utilize • multiple processing cores • wider vector units • more varied architectures available in a high end system • MKL can provide parallelism transparently and speed up programs with supported math routines without changing code. • Compiling R with MKL

13. Acceleration with Offload Model • Xeon Phi SE10P Coprocessor • 60 cores with 8GB GDDR5 • Intel x86 instruction set • Usage of familiar programming models, software, and tools • Pros • The host system can offload computing workload partially to the Xeon Phi • Independently run a compatible program

14. Computational Platform • Texas Advanced Computing Center Stampede cluster • MKL + offload • Each computing node • Two Intel Xeon E5-2680 processors each with eight cores @2.7GHz. • 32GB DDR3 memory • The Xeon Phi SE10P Coprocessor has 61 cores with 8GB GDDR5 • The NVIDIA K20 GPUs on each node have 5GB of on-board GDDR5 • Software • CentOS6.3. • Stock R 3.01 package compiled with the Intel compilers (v.13) and built with MKL v.11.

15. Outline • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results

16. Synthetic data (Genetics) • FREGENE genome simulator • Simulate sequence-like data over large genomic regions in large diploid populations • Simulated data • N=1,000 diploid individuals over 20,000 generations • 10 Mb genome with the average mutation rate as 2.5e-8 /site/generation • 3,274 SNPs with minor allele frequency (MAF) greater than 0.05 included • Four SNP data sets (i.e., g500, g1000, g2000, and g3274) by taking the first 500, 1,000, 2,000, and 3,274 SNPs from the entire data, respectively.

17. Synthetic data (Genetics)

18. Synthetic data (Imaging) • Assumption • Each image with multiple regions of interest (ROIs) • Voxel within each ROI highly correlated • Simulation • Random positive definite non-overlapping group structured covariance matrix • Apply Cholesky decomposition to obtain the background imaging data • Individual: N=1000, Size: 100x100 • We created three sets of phenotypic imaging data (i.e., p1000, p5000, and p10000), consisting of 1,000, 5,000 and 10,000 voxels respectively

19. Synthetic data (Imaging)

20. Outline • Imaging Genetics • Sparse Canonical Correlation Analysis (SCCA) • Computational Challenges and Methods • Data Simulation • Experimental Results

21. Results • R snowfall package (sfLapply) with MKL and offload model Baseline Parallel (MKL+ offload)

22. Results Correlation coefficient between the first pair of canonical components • Accelerated SCCA implementations yielded the same results • These correlation coefficients are close to the ground truth value of 1

23. Results

24. Conclusion • Initial steps to accelerate the SCCA implementation for brain imaging genetics applications. • Parallelism achieved in system implementation level to accelerate linear algebra computation using math kernel library (MKL) and partial offloading computing workload. • The 2-fold speedup, although encouraging, is still insufficient to handle extremely large-scale neuroimaging genetics data • millions of image voxels and millions of SNPs. • Future work • Big data analytic strategies at the parallel computing model level • Parallelization of multiplicative algorithms using MapReduce and CUDA. • Application to accelerate enhanced SCCA models as well as other bi-multivariate statistical models for analyzing brain imaging genetics data.

25. Acknowledgement • This research was supported by • NIH R01 LM011360 • NIH U01 AG024904 • NIH RC2 AG036535 • NIH R01 AG19771 • NIH P30 AG10133 • NSF IIS-1117335

26. Thank you