1 / 10

HDF5/SRB Integration August 28, 2006

HDF5/SRB Integration August 28, 2006. Peter Cao xcao@ncsa.edu HDF, NCSA. Mike Wan mwan@sdsc.edu SRB, SDSC. Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration. Current Status. Improved transferring raw data (SDSC) Tested performance

upton
Download Presentation

HDF5/SRB Integration August 28, 2006

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5/SRB IntegrationAugust 28, 2006 Peter Caoxcao@ncsa.edu HDF, NCSA Mike Wanmwan@sdsc.edu SRB, SDSC Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

  2. Current Status • Improved transferring raw data (SDSC) • Tested performance test #1 (270MB) from 18 to 15 (seconds) test #2 (550MB) from 231 to 35 (seconds) test #3 (820MB) from 734 to 113 (seconds) • Project plan for SRB-FLASH

  3. Things to move forward • Improve GUI component in HDFView to open remote files (NCSA) • Support primitive writing capabilities for HDF5 files in SRB (NCSA and SDSC) • Write documentation and extensive testing (NCSA and SDSC) • Release the production system and publish documentation (NCSA and SDSC)

  4. SAP/SAC program: FLASH • FLASH is a adaptive-mesh simulation code for astrophysical hydrodynamics • FLASH slice extractor retrieves a specific 2D slice from a 3D AMR (adaptive mesh refinement) Slice (gas temperature) and isosurface (gas density) in particle/mesh simulation

  5. SRB-FLASH: architecture

  6. SRB-FLASH: benefits • No need for large storage on local machine. Terabytes of data files sit only on the sever machine • Instant access to the data and metadata. Since only the selected data/metadata are transferred to the client instead of the whole multi-terabyte file, time on the network is tremendously reduced • Data sharing among scientists. Since data sits on the server machine, it can be shared with scientists without multiple local copies of the same file. Also, everyone will have access to the updated data after a new simulation run

  7. SRB-FLASH: project plan • This project will be divided into three phases: phase I, phase II and phase III • Phase I and part of phase II are required • Phase III and part of phase II are optional, depending on the duration and amount of the funding • The time estimates are based on an half (50%) FTE

  8. SRB-FLASH: milestone Phase I: a command line client utility and server function to support selection of FLASH AMR mesh data.

  9. SRB-FLASH: milestone Phase II: enhanced utilities querying FLASH specific metadata and statistics, and performing data interpolation and simple mathematic manipulation.

  10. SRB-FLASH: milestone Phase III: development of general framework that can be adapted to other applications. We need to scope out the milestone for this phase based on the experience of phase I and II.

More Related