1 / 16

Storage Resource Management (SRM) For Grid Applications A SciDAC supported middleware component

This resource management system addresses challenges like unreliable storage, data-intensive transactions, and optimization in the grid architecture. It integrates Hierarchical, Disk, and Tape Resource Managers. Developed by Arie Shoshani at Lawrence Berkeley National Laboratory. Visit http://sdm.lbl.gov/srm for more details.

hugoe
Download Presentation

Storage Resource Management (SRM) For Grid Applications A SciDAC supported middleware component

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Storage Resource Management (SRM) For Grid Applications A SciDAC supported middleware component Arie Shoshani Computing Sciences Directorate Lawrence Berkeley National Laboratory http://sdm.lbl.gov/srm

  2. Participants PI: Arie Shoshani LBNL – 2 FTEs: Arie Shoshani, PI Alex Sim, co-PI Junmin Gu Andreas Mueller Fermilab – ½ FTE: Don Petravick, Co-PI Rich Wellner

  3. Motivation • Grid architecture emphasized in the past • Security • Compute resource coordination & scheduling • Network resource coordination & scheduling (QOS) • SRMs role in the data grid architecture • Storage resource coordination & scheduling • Types of storage resource managers • Disk Resource Manager (DRM) • Tape Resource Manager (TRM) • Hierarchical Resource Manager (TRM + DRM)

  4. client client Replica catalog Request Interpreter Request Executer request planning Network Weather Service HRM DRM DRM tape system Disk Cache Disk Cache Disk Cache Where Do SRMs Fit in Grid Architecture? ... Client’s site logical query property-file index logical files site-specific files site-specific files requests pinning & file transfer requests network ...

  5. Challenges (1) • Managing storage resources in an unreliable distributed large heterogeneous system • Long lasting data intensive transactions • Can’t afford to restart jobs • Can’t afford to loose data, especially from experiments • Type of failures • Storage system failures • Mass Storage System (MSS) • Disk system • Server failures • Network failures

  6. Challenges (2) • Heterogeneity • Operating systems (well understood) • MSS - HPSS, Castor, Enstore, … • Disk systems – system attached, network attached, parallel • Optimization issues • avoid extra file transfers - What to keep in each disk caches over time • How to maximize sharing for multiple users • Global optimization • Multi-Tier storage system optimization

  7. Specific Problems • Managing resource space allocation • What if there is no space? • Managing pinning of files • What if files can be removed in the middle of a transfer • Space reservations • What if multiple files are needed concurrently • File streaming • For processing a large set of files • Pin-lock • What if you pinned files, and system deadlocks • User priorities • Access control – who can read/write a file

  8. tape system tape system Disk Cache Disk Cache HRMs in PPDG(high level view) • Monitors files written into BNL’s HPSS • Selects files to replicate • Issues request_to_put for file (or many files) Replica Coordinator HRM-COPY HRM-GET HRM (performs writes) HRM (performs reads) GridFTP GET (pull mode) LBNL BNL

  9. Details of Interactions

  10. Measurements FILE_REQUEST_FAILED Notified_Client Migration_Finished Migration_Requested Transfered_to_PDSF_from_BNL Staging_finished_at_BNL Staging_started_at BNL Staging_requested_at_BNL File replication request start

  11. Disk Cache Disk Cache Disk Cache Disk Cache Disk Cache SC 2001 Demo Setup Denver client Logical Request BIT-MAP Index Request Manager File Transfer Monitoring Legend: GridFTP DRM Control path Data Path Chicago Berkeley Livermore Berkeley server server server server GridFTP DRM FTP GridFTP HRM GridFTP

  12. Monitoring File Transfer

  13. Accomplishment • Developed HRMs and DRMs using the same uniform protocols • Deployed in PPDG • Developed Command Line interface to HRM • Wrote a joint design specification in cordination with EDG, Jlab, and Fermi (to be presented at GGF) • Wrote a paper for MSS conference • Future: develop a standard protocol • Future: deploy HRM in ORNL & NERSC for ESG II project

More Related