1 / 49

David A. Bader

BioPerf : An Open Benchmark Suite for Evaluating Computer Architecture on Bioinformatics and Life Science Applications. David A. Bader. Collaborators. Vipin Sachdeva (U New Mexico, Georgia Tech, IBM Austin) Tao Li (U Florida) Yue Li (U Florida) Virat Agrawal (IIT Delhi)

karah
Download Presentation

David A. Bader

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BioPerf: An Open Benchmark Suite for Evaluating Computer Architecture on Bioinformatics and Life Science Applications David A. Bader

  2. Collaborators • Vipin Sachdeva (U New Mexico, Georgia Tech, IBM Austin) • Tao Li (U Florida) • Yue Li (U Florida) • Virat Agrawal (IIT Delhi) • Gaurav Goel (IIT Delhi) • Abhishek Narain Singh (IIT Delhi) • Ram Rajamony (IBM Austin) BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  3. Acknowledgment of Support • National Science Foundation • CAREER: High-Performance Algorithms for Scientific Applications (06-11589; 00-93039) • ITR: Building the Tree of Life -- A National Resource for Phyloinformatics and Computational Phylogenetics (EF/BIO 03-31654) • DEB: Ecosystem Studies: Self-Organization of Semi-Arid Landscapes: Test of Optimality Principles (99-10123) • ITR/AP: Reconstructing Complex Evolutionary Histories (01-21377) • DEB Comparative Chloroplast Genomics: Integrating Computational Methods, Molecular Evolution, and Phylogeny (01-20709) • ITR/AP(DEB): Computing Optimal Phylogenetic Trees under Genome Rearrangement Metrics (01-13095) • DBI: Acquisition of a High Performance Shared-Memory Computer for Computational Science and Engineering (04-20513). • IBM PERCS / DARPA High Productivity Computing Systems (HPCS) • DARPA Contract NBCH30390004 BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  4. Contributions of this Work • An open source, freely-available, freely-redistributable suite of applications and inputs, BioPerf, which spans a wide variety of bioinformatics application • www.bioperf.org • Performance study on PowerPC G5, IBM Mambo simulator, and Alpha BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  5. Outline • Motivation • Bioinformatics Workload • BioPerf Suite • Performance Analysis on PowerPC G5 and Mambo • Conclusions and Future Work BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  6. Motivation • Improve performance on a wide range of bioinformatics applications • Heterogeneous in problems, algorithms, applications • BioPerf workload assembled as a representative set of bioinformatics applications important now and expected to increase in usage over the next 5—10 years • Decide if this is YAW “yet another workload” or rather unique in its characteristics BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  7. Related Work • General benchmark suites: SPEC • Domain-specific benchmarks • TPC, EEMBC, SPLASH, SPLASH-2 • Few special benchmark for bioinformatics • Previous attempts have been incomplete: Analysis on old architectures (BioBench) [Albayraktaroglu et al., ISPASS 2005] • Included proprietary codes in benchmark suite (BioInfoMark) [Li et al., MASCOTS 2005] • Previous suites not available for download • Included several non-redistributable packages • Inputs not articulated and not included with benchmark suite for similar comparisons BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  8. Guiding Principles for BioPerf • Coverage: The packages must span the heterogeneity of algorithms and biological and life science problems important today as well as (in our view) increasing in importance over the next 5-10 years. • Popularity: Codes with larger numbers of users are preferred because these packages represent a greater percentage of the aggregate workloads used in this domain. • Open Source: Open source code allows the scientific study of the applicatio performance, the ability to place hooks into the code, and eases porting to new architectures. • Licensing: Only packages for which their licensing allows free redistribution as open source are included. This requirement eliminated several popular packages, but was kept as a strict requirement to encourage the broadest use of this suite. • Portability: Preference was given to packages that used standard programming languages and could easily be ported to new systems (both in sequential and parallel languages). • Performance: We gave slight preference to packages whose performance is well-characterized in other studies. In addition, we strived for computationally-demanding packages and included parallel versions where available. BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  9. BioPerf Suite • Pre-compiled binaries (PowerPC, x86, Alpha) • Scalable Input datasets with each code for fair comparisons • Scripts for installation, running and collecting outputs • Documentation for compiling and using the suite • Parallel codes where available • Available for download from www.bioperf.org BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  10. BioPerf workload Area Package Executables Sequence homology Word-based BLAST blastp, blastn Profile-based HMMER hmmpfam, hmmsearch Sequence Alignment Pairwise FASTA ssearch, fasta MultipleCLUSTALW clustalw, clustalw_smp Multiple TCOFFEE tcoffee Phylogeny Parsimony/Likelihood PHYLIPdnapenny, promlk Gene Rearrangement GRAPPA grappa Protein Structure Prediction PREDATOR predator Gene Finding GLIMMER glimmer,glimmer-package Molecular Dynamics CE ce BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  11. ALIGNMENT S1 : ACGCTGATATTA ACGCTGATAT---TA S2 : AGTGTTATCCCTA AG--TGTTATCCCTA S1 : ACGCTGATATTA ACGCTGATAT---TA S2 : AGTGTTATCCCTA AG--TGTTATCCCTA MATCH Sequence Alignment • Sequence Alignment one of the most useful techniques in computational biology • Sequence Alignment : Stacking the sequences against each other, with gaps if necessary, to expose similarity. BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  12. ALIGNMENT S1 : ACGCTGATATTA ACGCTGATAT---TA S2 : AGTGTTATCCCTA AG--TGTTATCCCTA S1 : ACGCTGATATTA ACGCTGATAT---TA S2 : AGTGTTATCCCTA AG--TGTTATCCCTA MISMATCH Sequence Alignment • Sequence Alignment one of the most useful techniques in computational biology • Sequence Alignment : Stacking the sequences against each other, with gaps if necessary, to expose similarity. BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  13. ALIGNMENT S1 : ACGCTGATATTA ACGCTGATAT---TA S2 : AGTGTTATCCCTA AG--TGTTATCCCTA S1 : ACGCTGATATTA ACGCTGATAT---TA S2 : AGTGTTATCCCTA AG--TGTTATCCCTA “GAPS” Sequence Alignment • Sequence Alignment one of the most useful techniques in computational biology • Sequence Alignment : Stacking the sequences against each other, with gaps if necessary, to expose similarity. BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  14. V S N — S — S N A — S A — — — A S A N S V S N S Multiple Sequence Alignment • Bring the greatest number of similar characters into same column. • Provides much more information than pairwise alignment Run-time of dynamic programming solution = O(2k nk) 6 sequences of length 100  6.4X1013 calculations Hence heuristics employed BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  15. Sequence Homology • Find similar sequences (DNA/protein) to an unknown sequence (DNA/protein). • Computationally expensive • Size of data is huge and grows exponentially every year • Public databases available: Genbank, SwissProt, PDB NCBI Genbank DNA sequences 5 million sequences Swissprot Protein Sequences 160,000 sequences PDB Protein Structure 32,000 structures Problems with computational approach • Exact alignment is O(l2) dynamic programming solution • Quicker but less accurate heuristics employed BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  16. Blast • Basic Local Alignment Search Tool • Developed by NCBI • The most important bioinformatics application for its popularity BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  17. FASTA • Also performs pairwise sequence alignment BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  18. ClustalW • Multiple sequence alignment (MSA) program BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  19. T-Coffee • A sequential MSA similar to ClustalW with higher accuracy and complexity BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  20. Hmmer • Align multiple sequences by using hidden Markov models BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  21. Phylogenetic Reconstruction • Study the evolution of all sequences and all species The Tree of Life (10-100M organisms) • Find the best among all possible trees. • Given n taxa, number of possible trees (2n-3)!! • 10 taxa  2 million trees • Approaches like maximum parsimony, maximum likelihood, among others BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  22. Phylogeny Reconstruction: Phylip • Collection of programs for inferring phylogenies • Methods include • Maximum parsimony • Maximum likelihood • Distance based methods. • Input: Aligned dataset of 92 cyclophilins proteins of eukaryotes each of length 220 BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  23. A C A D Tobacco X E Y B E Z W C F B D F Phylogeny Reconstruction: GRAPPA • Genome Rearrangements Analysis under Parsimony and other Phylogenetic Algorithm • Freely-available, open-source, GNU GPL • already used by other computational phylogeny groups, Caprara, Pevzner, LANL, FBI, Smithsonian Institute, Aventis, GlaxoSmithKline, PharmCos. • Gene-order Phylogeny Reconstruction • Breakpoint Median • Inversion Median • over one-billion fold speedup from previous codes • Parallelism scales linearly with the number of processors • [Bader, Moret, Warnow] • Campanulaceae • Bob Jansen, UT-Austin; • Linda Raubeson, Central Washington U Gene-order based phylogeny Input: 12 bluebell flower species of 105 genes BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  24. Protein Structure Prediction • Find the sequences, three dimensional structures and functions of all proteins and vice-versa • Why computationally? • Experimental Techniques slow and expensive • Problems with computational approach • Little understanding of how structure develops • Does function really follow structure ? BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  25. Protein Structure : Predator • Tool for finding protein structures. • Relies on local alignments from BLAST, FASTA • Input: 20 sequences from Swissprot each of length about 7000 residues. BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  26. CE (Combinatorial Extension) • Find structural similarities between the primary structures of pairs of proteins. BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  27. Gene-Finding: Glimmer • Gene-Finding: Find regions of genome which code for proteins. • Widely used gene finding tool for microbial DNA. • Input: Bacteria genome consisting of 9.2 million base pairs BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  28. Pre-compiled binaries • PowerPC • x86 • Alpha BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  29. BioPerf Performance Studies • Analysis at the instruction and memory level on PowerPC • Livegraph data helps to visualize performance as it varies during phases of a run • Identify bottlenecks of current processors and make inputs for better performance on future processors • Ongoing work using Mambo simulator (IBM PERCS) • Pre-compiled Alpha binaries for the majority of benchmarks for simulation • In order to reduce the simulation time, we collect the simulation points for those benchmarks by using SimPoint BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  30. Conclusions • Bioinformatics is a rapidly evolving field of increasing importance to computing • BioPerf is a first step to characterize bioinformatics workload: infrastructure to evaluate performance • Performance data collected so far provides insight into the limitations of current architectures BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  31. Related Publications • D.A. Bader, V. Sachdeva, A. Trehan, V. Agarwal, G. Gupta, and A.N. Singh, “BioSPLASH: A sample workload from bioinformatics and computational biology for optimizing next-generation high-performance computer systems,” (Poster Session), 13th Annual International Conference on Intelligent Systems for Molecular Biology (ISMB 2005), Detroit, MI, June 25-29, 2005. • D.A. Bader, V. Sachdeva, “BioSPLASH: Incorporating life sciences applications in the architectural optimizations of next-generation petaflop-system,”(Poster Session), The 4th IEEE Computational Systems Bioinformatics Conference (CSB 2005), Stanford University, CA, August 8-11, 2005 • D.A. Bader, Y. Li, T. Li, V. Sachdeva, “BioPerf: A Benchmark Suite to Evaluate High-Performance Computer Architecture on Bioinformatics Applications,” The IEEE International Symposium on Workload Characterization (IISWC 2005), Austin, TX, October 6-8, 2005 BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  32. Backup Slides BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  33. BioPerf on PowerPC • PowerPC G5 dual-processor machine • Uniprocessor performance ( nvram boot-args=1 ) • CPU frequency of 1.8 Ghz • 1 GB of physical memory available. • Codes compiled using gcc-3.3 with no additional optimizations. • MOnster tool of C.H.U.D package used for collecting hardware performance counters • Instruction and Memory level analysis BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  34. Clustalw Algorithm Summary • Pairwise alignment of all sequences against one another. • dynamic programming step • Generate guide tree for aligning sequences • Sequences with highest similarity get aligned first • Sequence-group and group-group alignments (progressive) • All possible pairwise alignments between sequence and group are tried. Highest scoring pair is how it gets aligned to the group. • All possible pairwise alignments of sequences between groups are tried; highest scoring pair is how groups get aligned • Clustalw uses calculations from step 1 for this step BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  35. Progressive alignment step (29.8%) Almost all instructions are ppc Clustalw Livegraphs • Input: 318 sequences each of length almost 1050 Pairwise alignment step (70.1%) ppc instructions lag the total instructions Guide tree formation (<0.1%) of total time BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  36. Clustalw Livegraphs Instructions executed increase remarkably L1D hit rate almost 100% L1D Hit Rate falls down Instructios executed low BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  37. Instruction count increases in progressive alignment Clustalw Livegraphs Is performance directly related to branch mispredicts ? Branch mispredicts is high in dynamic programming Branch mispredicts falls in progressive alignment Instruction count is low BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  38. Clustalw livegraphs Almost all branch mispredicts caused due to condition register mispredict BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  39. But what about loads per instruction ? Instruction count is low Instruction count increases in progressive alignment Loads per instruction falls in progressive alignment Loads per instruction is high in dynamic programming BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  40. Clustalw livegraphs - smaller inputs Same performance characteristics but with longer progressive alignment step • Smaller input - 44 sequences of length 583 BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  41. Clustalw livegraphs – smaller inputs Same performance characteristics but with longer progressive alignment step BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  42. Clustalw livegraphs – smaller inputs Almost all branch mispredicts caused due to condition register mispredict BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  43. Clustalw livegraphs – smaller input Can we use Mambo with smaller input sizes for more performance analysis ? Same performance characteristics but with longer progressive alignment step BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  44. Using Mambo with Clustalw and other applications • Collect separate outputs for each phase of the run • Inserted “callthru exit” into the source code separating each part • Dump the system statistics at the end of each phase • mysim stats dump • mysim caches stats dump • MamboClearSystemStats (clean the previous statistics) • Multiple “mysim go” in the .tcl file. BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  45. Progressive alignment uses results from first step – high branch and loads Clustalw on Mambo Mambo offers far more detailed instruction profiling than G5 ? Pairwise alignment – high loads and arithmetic instructions BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  46. Comparing large datasets with small datasets Branch mispredicts lesser due to smaller dynamic programming step Is it feasible to use smaller input datasets for accurate simulation results ? Branch mispredicts much higher High increase in L1d hit rate BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  47. Summary of BioPerf performance Highest branch mispredicts and TLB misses High % of ld/st/io instruction Highest instructions executed per cycle Low TLB misses High loads per instruction Very low % of ld/st/io High L1d Hit rate Low branch mispredicts BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  48. Summary of BioPerf performance Low TLB misses High branch mispredicts Mid-range instructionsper cycle High loads per instruction Low % of ld/st/io instructions BioPerf: an open bioinformatics and life sciences workload, David A. Bader

  49. Summary of BioPerf Performance Lowest instruction rate Lowest loads per instruction Low % of ld/st/io instructions Lowest L1D and L2D hit rate Low branch mispredicts and TLB misses BioPerf: an open bioinformatics and life sciences workload, David A. Bader

More Related