Storage Solutions for Bioinformatics

1. Storage Solutions for Bioinformatics Li Yan Director of FlexLab, Bioinformatics core technology laboratory liyan3@genomics.cn http://www.genomics.cn/FlexLab/index.html Science and Technology Division, BGI-Shenzhen

2. OUTLINE • Background • Hardware Infrastructure of Data Storage • Data Management • Data Storage Architecture In BGI • Distributed Computing on Storage Server

3. Background: Fast Growing Big Data

4. Sequencing, sequencing and sequencing

5. Background

6. Fast growing big data • From small genomes to large complex genomes • E. coli Genome: 4.9M • Caenorhaditiselegans Genome: 100M • Human Genome: 3G • Wheat Genome: 16G • Salamander: 45G • From one sample to populations • Human Genome: 3 billion DNA subunits (A,T,C,G) • 80~100X Sequencing: 600GB Raw data for individual study • 1000 Genome Project: 600TB Raw data for population study • From the first generation sequencing to the second generation sequencing

7. Long-Term Data Storage Needs • Properly secure the data • Plan for data redundancy, which generally means we mirror data with two or more copies • Available(24x7x365) for all kinds of uses • Readily accessible and in the right format • Fast Data Transfer for collaborations • Fast Network server(Aspera) instead of mailing a hard drive • Scalable, easy to scale up • Choosing reliable file systems

8. Hardware infrastructure of data storage

9. Type of Storageinfrastructure • Disk library • A high-capacity storage system that holds a quantity of CD-ROM, DVD or magneto-optic (MO) disks in a storage rack and feeds them to one or more drives for reading and writing. • Magnetic tape • A high-capacity data storage system for storing, retrieving, reading and writing multiple magnetic tape cartridges. • Redundant array of independent disks (RAID) • RAID is a storage technology that combines multiple disk drive components into a logical unit • Direct-attached storage (DAS) • a digital storage system directly attached to a server or workstation, without a storage network in between • Network-attached storage (NAS) • Network-attached storage (NAS) is file-level computer data storage connected to a computer network providing data access to heterogeneous clients. • Storage area network (SAN) • A storage area network (SAN) is a dedicated network that provides access to consolidated, block level data storage.

12. Software Level of Data storage

13. Data flow of NGS Alignment Assembly Raw Data Sequencer Association Complex workflow • Annotation of features • Variations/Mutations • Protein Structural • Gene Expressions • Function Networks Meaningful Biology Data Data Store

14. Data Management • Classify the data into different levels • First Level of Storage: Dynamic, fast, Temporary • Secondary Level of storage: Slower than first level, but enduring and safety • Third Level of storage: High capacity medium for backups and archives • Choosing file systems • Current popular distributed file systems include: Lustre, HDFS, MogileFS, FreeNAS, FastDFS, OpenAFS, MooseFS, pNFS, and GoogleFS.

15. Classify the data into different levels • First Level of Storage: Dynamic, fast, Temporary • intermediate results of data analysis • Reference data • … • Secondary Level of storage: Slower than first level, but enduring and safety • Sequencing raw data • Meaningful data • Third Level of storage: High capacity medium for backups and archives • Backups and archives of raw data and meaningful data

16. Distributed File systems • Lustre • lustreis a large, safe and reliable, highly available cluster file system, which is developed and maintained by the SUN. Lustre can support more than 10,000 nodes, the number to the number of PB storage system. • Hadoop(HDFS) • Hadoop and not just a hadoop distributed file system for storage, but designed for general-purpose computing device in the form of large-scale distributed applications running on the cluster framework. • OneFS • OneFS enables to scale data access capacity to more than 1.6 petabytes and up to 10 Gb/sec of throughput for a single cluster capacity of up to 10 GBS (Gigabytes per second) of throughput. Distributed file systems Storage Server

17. Distributed File systems • MogileFS(www.danga.com) • FreeNAS( www.openqrm.org ) • FastDFS(code.google.com / p / fastdfs) • OpenAFS( www.openafs.org ) • MooseFS(derf.homelinux.org) • pNFS( www.pnfs.com ) • GoogleFS

18. Data compression&& Data security • Data compression • Common used: • Lemple-Ziv, BWT • Exclusive used for DNA sequences: • Biocompress, GeneCompress, CTW-LZ, GeNML, fqzcomp, sam_comp • Data security • Raid system failure/ Redundancy • File system • Network

19. Data Storage Architecture In BGI

20. Data Storage Architecture In BGI Two Copies Archiving Write Write Tape Library Write Read Sequencers Compute Nodes Read Write

21. Data Storage Architecture In BGI Two Copies Archiving Write Write Tape Library Write Read Sequencers Compute Nodes Read First Level Storage Write

22. Data Storage Architecture In BGI Two Copies Archiving Write Write Second Level Storage Tape Library Write Read Sequencers Compute Nodes Read Write

23. Data Storage Architecture In BGI Two Copies Archiving Write Write Tape Library Write Read Third Level Storage Sequencers Compute Nodes Read Write

24. Data Storage Architecture In BGI Two Copies Archiving Write Write Tape Library Write Read Sequencers Compute Nodes Read Write

25. Distributed Computing on Storage Server

26. Traditional Genome Assembly Costly, Unscalable NGS read file Sequence Assembly Large memory server >500GB Users Storage

27. Distributed Genome Assembly Several storage server(IBM3630*16 for human genome) …… Assembly Cost effectively, Scalable

28. Hecate Constructing de bruijn Graph Solving Tiny Repeats Merging Bubbles Merging Contigs Scaffolding

29. Reads Gaea 2.1 Reference genome Distributed Indexing for load balancing Preprocessing Flexible splitting tolerates more mistmatches Locating Aligning Dynamic Programming for robust gap alignment SNP calling Standard mapping quality for SNP calling

30. Q&A