1 / 10

HDF5/SRB Integration July 10, 2006

HDF5/SRB Integration July 10, 2006. Peter Cao xcao@ncsa.edu HDF, NCSA. Mike Wan mwan@sdsc.edu SRB, SDSC. Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration. Current Status. Present the work at the TeraGrid '06 Publish HDF5 AIP documents

usoa
Download Presentation

HDF5/SRB Integration July 10, 2006

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5/SRB IntegrationJuly 10, 2006 Peter Caoxcao@ncsa.edu HDF, NCSA Mike Wanmwan@sdsc.edu SRB, SDSC Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

  2. Current Status • Present the work at the TeraGrid '06 • Publish HDF5 AIP documents • White paper: http://hdf.ncsa.uiuc.edu/hdf-aip-html/ • HDF5 METS template: http://hdf.ncsa.uiuc.edu/hdf-aip-html/hdf5_mets_template.xml • Finish h5ingest command line tool • Create HDF5 METS template file • Validate HDF5 METS document • Setup a demo server to support SCEC files

  3. Current Status • Work on test suite and bug fix • Add code to separate HDF5 I/O time and SRB time • Test large files and dataset (>2GB) • Fix bug at srb client handler • Work on performance improvement • Implement a fairly large set of changes for the performance improvement by transfer raw data by byte-stream • Need to test on large files

  4. Next Month • More tests on performance for transferring raw data • Add more features to HDFView for SRB support • Integrate the software into the SRB configuration and distribution

  5. Potential SAC Projects • SDSC ENZO project • Enzo, 3D cosmological hydrodynamics code, simulating the process of massive star formation and destruction • HDF5 is used as file format and parallel file I/O access • FLASH Program • The UC/DOE collaboration on creating three-dimensional, virtual reality projections of the cosmic explosions • HDF5 is used for storing the data and high I/O access • SCEC Terascale Earthquake Simulations • Over 100 TB data/year • Collections at SRB – 2.6 million files, 114 Terabytes

  6. TeraShake Surface Seismograms • 4D Array (1.2 TB) • Time (22,728) • Horizontal (3,000) • Vertical (1,500) • Vector Component (3) • Each file: • 22,728 x 3,000 x 5 x 1 • 1,363,680,000 Bytes • TeraShake scenario • 900 files

  7. Example HDF5 File HDF5 File 32-bit float 22,728 3,000 25

  8. File on SRB server

  9. Select a Subset

  10. HDFView

More Related