1 / 19

HDF5-iRODS

HDF5-iRODS. Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS Workshop XII October 16, 2008. Imagine. 100 Frames x 1 GB = 100 GB. 1 GB. HPSS. DB. HPC. Outline. HDF5-iRODS module Applications Demo (if time permits). What is iRODS?.

kendall-tyson
Download Presentation

HDF5-iRODS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5-iRODS Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS Workshop XII October 16, 2008 HDF and HDF-EOS Workshop XII, Denver, CO

  2. Imagine 100 Frames x 1 GB = 100 GB 1 GB HPSS DB HPC HDF and HDF-EOS Workshop XII, Denver, CO

  3. Outline • HDF5-iRODS module • Applications • Demo (if time permits) HDF and HDF-EOS Workshop XII, Denver, CO

  4. What is iRODS? • Stands for iRule Oriented Data Systems. • Developed by the Storage Resource Broker (SRB) team at the San Diego Supercomputer Center (SDSC). • A data grid software system that enables a customizable architecture for sharing data distributed across heterogeneous resources. HDF and HDF-EOS Workshop XII, Denver, CO

  5. What is iRODS? Distributed Storage Database System Rule System For more information and download, visit www.irods.org HDF and HDF-EOS Workshop XII, Denver, CO

  6. Motivation High-performance distributed data system HDF and HDF-EOS Workshop XII, Denver, CO

  7. Get the file HDF5 HDF5 Whole File Access I need to see the eye of Hurricane Bob! client server Transfer large file – slow! HDF and HDF-EOS Workshop XII, Denver, CO

  8. Get me the eye of hurricane Bob HDF5 HDF5 Object or Subset Level Access I need to see the eye of Hurricane Bob! client server Small transfer – fast! HDF and HDF-EOS Workshop XII, Denver, CO

  9. HDF5-iRODS Module Distributed Storage HDF5 iRODS Module Database System Rule System Micro-services HDF and HDF-EOS Workshop XII, Denver, CO

  10. HDF5 HDF5-iRODS Data Flow client server HDF5 Library HDF5 Object or Subset (File, Group, Dataset, Subset of Dataset, Attribute) HDF5 Object or Subset (File, Group, Dataset, Subset of Dataset, Attribute) iRODS message (pack/unpack) iRODS message (pack/unpack) HDF and HDF-EOS Workshop XII, Denver, CO

  11. New iRODS Micro-services • Five iRODS micro-services • msiH5File_open • msiH5File_close • msiH5Dataset_read • reads entire dataset or subset of dataset • msiH5Dataset_read_attribute • msiH5Group_read_attribute Rule Engine msiH5Dataset_read H5Dataset.read() File HDF and HDF-EOS Workshop XII, Denver, CO

  12. HDF5-Enabled iRODS Server • HDF5 library • Other external libraries (SZIP, ZLIB) • iRODS version 1.1 or later from https://www.irods.org/index.php/Downloads/ Follow the README instruction at module/hdf5 HDF and HDF-EOS Workshop XII, Denver, CO

  13. Client Application Requirements • HDF5 object header files and client handlers • iRODS client library and header files • HDF5-iRODS JNI for Java applications only • $HOME/.irods/.irodsEnv irodsHost 'kagiso.hdfgroup.uiuc.edu' irodsPort 1247 irodsUserName 'rods‘ … For more information and download, visit http://www.hdfgroup.org/projects/irods HDF and HDF-EOS Workshop XII, Denver, CO

  14. Example: HDFView Client Application HDF5-Enabled iRODS Server HDF and HDF-EOS Workshop XII, Denver, CO

  15. Example: HDFView HDF and HDF-EOS Workshop XII, Denver, CO

  16. Example: islice FLASH is an adaptive-mesh simulation code for astrophysical hydrodynamics problems • Command-line tool to visualize data produced by FLASH simulation runs • Data is huge (~ 100 GB) • Interesting part is small adaptive mesh 16*16*16*47531 For more information, visit flash.uchicago.edu HDF and HDF-EOS Workshop XII, Denver, CO

  17. Star Breakout point Ash Flow Example: islice “./islice -t flash.pal -m rpv1 -p 2 rundir_055_8km_hdf5_plt_cnt_0424” 2048*2048*8 (32MB) A slice from a 3D simulation of The Detonation of a White Dwarf Star HDF and HDF-EOS Workshop XII, Denver, CO

  18. Thank You! This project is sponsored by CIP/NLADR, NSF PACI Project in Support of the Collaboration between the National Center for Supercomputing Applications (NCSA) and the San Diego Supercomputer Center (SDSC). The project is managed under the CyberInfrastructure Partnership (CIP), a joint effort led by NCSA and SDSC to help scientists and engineers take full advantage of the high-end CyberInfrastructure resources funded by the National Science Foundation (NSF). HDF and HDF-EOS Workshop XII, Denver, CO

  19. Questions/comments? HDF and HDF-EOS Workshop XII, Denver, CO

More Related