1 / 10

MetaData (Management) and MPI-IO

MetaData (Management) and MPI-IO. PIs: Alok Choudhary, Wei-Keng Liao Department of ECE, Northwestern University With Bill Gropp and Rob Ross, ANL. SDM kickoff meeting July 10-11, 2001. Assess Hint – MPI-IO Info.

honey
Download Presentation

MetaData (Management) and MPI-IO

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MetaData (Management)andMPI-IO PIs: Alok Choudhary, Wei-Keng Liao Department of ECE, Northwestern University With Bill Gropp and Rob Ross, ANL SDM kickoff meeting July 10-11, 2001 choudhar@ece.nwu.edu 1

  2. Assess Hint – MPI-IO Info • MPI Info object provides MPI-IO implementations file access hints for improving I/O performance and/or for minimizing use of system resources • File info is specified on a per file basis • We can classify MPI-IO file info as follows: • Access pattern : access frequency, sub-array access, sub-array size, the number of accessing processors • Caching : turn on/off server buffering, buffering for data sieving (collective buffering), buffer size, block size • File storage : File name and permission when creating, I/O node list, striping factor, striping size, I/O device number from which to start striping choudhar@ece.nwu.edu 2

  3. MetaData Used in MPI-IO • High Level (Application Oriented) • Parallel partition patterns: data sieving • Access frequency: once, multiple times • I/O modes: read only, overwrite, create • Request chunk size: small, medium, large • Request sequence: random, strided, sequential • Low Level (Storage System Oriented) • File striping: striping factor, striping size • File caching, buffer size • Storage pattern, storage device, I/O nodes • Migrate, purge within hierarchical storage system choudhar@ece.nwu.edu 3

  4. Who Accesses and How? choudhar@ece.nwu.edu 4

  5. Where is the Metadata? • Meta data provided by users or database • File info used internally to optimize I/O • Users specify manually • Users provide both high and low level meta data • Manually choose collective / non-collective MPI I/O calls • Provided by database • Apply I/O optimization rules to determine proper MPI-file info • Use of collective / non-collective calls is determined automatically choudhar@ece.nwu.edu 5

  6. Rules to Determine I/O Strategies choudhar@ece.nwu.edu 7

  7. Application Programming Interface • Initialization: register application, record arguments for each run • Data association: build relationship between multiple datasets • Load: find data location in the storage system from previous runs, determine best I/O calls by comparing access and storage pattern • Save: choose file names, set file views, and provide hints of optimal I/O calls • Finalization: close files and connection to the database choudhar@ece.nwu.edu 8

  8. Implementation of the MDMS API choudhar@ece.nwu.edu 9

  9. Meta Data Organized in Relation Database choudhar@ece.nwu.edu 10

  10. Meta Data Management Challenge • What meta data to be collected? • For reference only or performance improvement • How to classify meta data into levels? • Levels of programming, performance, file storage, etc. • How to organize and manage meta data? • Relation tables in databases or XML files in XML database • Where to store the meta data for different levels? • Database, files, or file systems choudhar@ece.nwu.edu 11

More Related