1 / 6

PETASCALE DATA STORAGE INSTITUTE

5,000. 500. 50. 5. 5. 10. 3. 10. 2. 10. PETASCALE DATA STORAGE INSTITUTE. The Drive to Petascale Computing Faster computers need more data, faster. Checkpoint at Terabytes/sec Petabyte files Billions of files Revisit programming for Input/Output

coyne
Download Presentation

PETASCALE DATA STORAGE INSTITUTE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 5,000 500 50 5 5 10 3 10 2 10 PETASCALE DATA STORAGE INSTITUTE • The Drive to Petascale Computing • Faster computers need more data, faster. • Checkpoint at Terabytes/sec • Petabyte files • Billions of files • Revisit programming for Input/Output • Data center automation • Acceleration for search Everything Must Scale with Compute 2015: 100 PF -- Computing Speed TFLOP/s Memory 2,500 TeraBytes Year 2012 250 ‘08 25 ‘04 Disk 50 -- 2011: 10 PF 2.5 5 PetaBytes .5 ‘00 .05 1 5 Application Performance 50 200 Parallel I/O -- 2008: 1 PF 500 .5 .5 5,000 200 GigaBytes/sec 5 2,000 5 • PDSI Thrusts: • Data Capture • Education & Dissemination • Innovation 20,000 50 50 -- 2005: 100 TF Metadata 500 Inserts/sec Network Speed 500 Gigabits/sec ArchivalStorage GigaBytes/sec -- 2001: 10 TF http://www.pdsi-scidac.org/

  2. PETASCALE DATA STORAGE INSTITUTE Roadrunner & PanFS MPP2 & Lustre Jaguar & Lustre Pheonix& XFS Lightning & PanFS Q & PFS Red Storm & Lustre Blue Mountain & XFS Steeped in Terascale Experience Seaborg & GPFS http://www.pdsi-scidac.org/

  3. Goal specifications & complaints Managerial tier Automation Agents Statistics & predictions ••• Administrator supervisor Monitoring info Configuration settings Storage bricks I/O replies & requests pNFS ••• MDS Mechanical tier HPC Apps pNFS Driver 1. SBC (blocks)2. OSD (objects)3. NFS (files) pNFS server Layout grant &revoke Storage Manager PETASCALE DATA STORAGE INSTITUTE Strategic Plan • IT Automation • Instrumentation • Visualization • Machine Learning • Diagnosis • Adaptation • Scaling Further • Global/WAN access • Federated security • Metadata at scale • Para-virtualization Peta-Bytes Tera-B/sec Giga-files Mega-CPUs • Outreach • Storage-research-list • Collaboration w/ other Scidacs Innovation • Education • Workshops • Tutorials • Course materials Education & Dissemination • HPC NFS • Parallel NFS • Secure NFS • IETF Standard • App Workloads • INCITE resources • Trace & replay tools (e.g. BLAST, CCSM, Calore, EVH1, MCNP, GYRO, Sierra, QCDand other Scidacs) Data Capture • API Standards • POSIX API • Rich metadata • Compute-in-disk • Archive API • Quality of Storage • Failure Data • Capture & publish • Computer Failure Data Repository • (e.g. LANL’s outages by root cause) NFSv4extendedw/ layouts Tera-Bytes Giga-B/sec Mega-files Kilo-CPUs http://www.pdsi-scidac.org/

  4. PETASCALE DATA STORAGE INSTITUTE Participating Organizations Carnegie Mellon UniversityGarth Gibson (PI) University of California, Santa Cruz Darrell Long (co-PI) University of Michigan, Ann ArborPeter Honeyman (co-PI) Los Alamos National Laboratory Gary Grider (co-PI) Lawrence Berkeley National LaboratoryBill Kramer (co-PI) Oak Ridge National LaboratoryPhilip Roth (co-PI) Pacific Northwest National LaboratoryEvan Felix (co-PI) Sandia National LaboratoryLee Ward (co-PI) http://www.pdsi-scidac.org/

  5. PETASCALE DATA STORAGE INSTITUTE Programming for Storage • The Need for Training Programmers for Storage • HPC IT managers work for users who program apps • Often performance of apps/workflows dependent on storage • Many times best solutions would be to change the program • Reality is app specialists intolerant of requests to reprogram for better storage performance • That is, reprogramming for storage performance often doesn’t get done • Approach: Create tools, training to help a priori • Give programmers libraries, performance debugging tools that avoid or detect poor storage patterns • Give tutorials, case studies, help pages showing weak programming approaches and how to improve them http://www.pdsi-scidac.org/

  6. PETASCALE DATA STORAGE INSTITUTE Example from BioInformatics • Pseudo code example from IT manager -- single thread for( I=0, I<1000, I++){ for( J=0, J<1000, J++){ buf = compute (I,J); f = open( “file_foo”); lseek(f, offset(I,J)); write(f, buf, lengthof(buff)) close(f); }} • Buf turns out to be small, unaligned, fixed length • Obvious fixes: • Open/close outside both loops • Malloc sizeof 1000000*lengthof(buff), copy into it in memory, one write at end http://www.pdsi-scidac.org/

More Related