1 / 12

Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory

Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory. Earth System Grid. Main ESG portal 148.53 TB of data at four locations (NCAR, LBNL, ORNL, LANL) 965,551 files Includes the past 7 years of joint DOE/NSF climate modeling experiments

monte
Download Presentation

Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory

  2. Earth System Grid • Main ESG portal • 148.53 TB of data at four locations (NCAR, LBNL, ORNL, LANL) • 965,551 files • Includes the past 7 years of joint DOE/NSF climate modeling experiments • 4713 registered users from 28 countries • Downloads to date: 31TB/99,938 files • IPCC AR4 ESG portal • 28 TB of data at one location • 68,400 files • Model data from 11 countries • Generated by a modeling campaign coordinated by the Intergovernmental Panel on Climate Change (IPCC) • 818 registered analysis projects from 58 countries • Downloads to date: 123TB/543,500 files, 300 GB/day on average • Courtesy: http://www.earthsystemgrid.org

  3. The Role SRMs in ESG • Data production • Run simulations • Generate data at compute sites -> move to archives • Need robust bulk data movement – use SRMs • Data analysis • Replicate part of data to ESG portal sites • Get subsets of data to users/clients • Use SRMs to move data from any archive to portal site • Serve multiple files to users using an SRM client

  4. SRMs in ESG Disk Disk Cache Cache Client HRM @ LBNL Files Selection And Request download Disk Cache HRM @ NCAR Portal DRM @ LANL Disk Disk Cache Cache DRM @ LLNL Disk Disk Disk Cache Cache Cache NCAR MSS DISK CACHE HRM @ ORNL Disk Cache DRM – Disk Storage Manager HRM – Hierarchical Storage Manager

  5. SRM works in concert with other Grid components in ESG LBNL HPSS DISK ANL GridFTP service RLS Globus Security infrastructure HRM Storage Resource Management GridFTP server NCAR ORNL ESG Portal RLS LLNL HRM Storage Resource Management User DB ESG CA XML data catalogs GridFTP server DISK IPCC Portal XML data catalogs ESG Metadata DB MyProxy RLS DISK HPSS LAHFS DRM Storage Resource Management OPeNDAP-g RLS GridFTP server FTP server HRM Storage Resource Management GridFTP server ISI LANL DISK MCS Metadata Cataloguing Services RLS MSS Mass Torage System RLS Replica Location Services DRM Storage Resource Management DISK GridFTP server Monitoring Discovery ervices

  6. DataMover: Robust Multi-File replication • Multi-File Replication – why is it a problem? • Tedious task – many files, repetitious • Lengthy task – long time, can take hours, even days • Error prone – need to monitor transfers • Error recovery – need to restart file transfers • Stage and archive from MSS – limited concurrency, down time, transient failures • Use of FTP – large windows, concurrent transfer • Security – both for local MSS and the network • Firewalls – transfer from/to MSS must be internal to the site • Specialized MSS – HPSS at NERSC, ORNL, …, MSS at NCAR

  7. Main Idea • Take advantage of Storage Resource Managers • What do you get? • SRMs queue multi-file requests • SRMs allocate space and release space automatically • SRMs request files from remote SRMs • Recover from network failures • SRMs invoke GridFTP – use large windows & parallel streams • Special SRM in front of HPSS was developed by the SRM middleware project at LBNL and applied to PPDG • Called “Hierarchical Storage Manager” (HRM) • Queues multi-file requests to HPSS • Performs both staging and archiving • Recovers from failures during staging and archiving • For MSS at NCAR • Replace module that communicates with HPSS to communicate with NCAR-MSS

  8. Anywhere HRM-Client Command-line Interface Recovers from file transfer failures Recovers from staging failures Get list of files From directory Recovers from archiving failures HRM-COPY (thousands of files) NCAR SRM-GET (one file at a time) SRM (performs writes) SRM (performs reads) LBNL/ ORNL GridFTP GET (pull mode) NCAR-MSS Network transfer Web-based File Monitoring Tool archive files stage files Disk Cache Disk Cache DataMover: SRMs use in ESG for Robust Muti-file replication

  9. Web-Based File Monitoring Tool • Shows: • Files already transferred- Files during transfer • Files to be transferred • Also shows for • each file: • Source URL • Target URL • Transfer rate

  10. File tracking helps to identify bottlenecks Shows that archiving is the bottleneck

  11. File tracking shows recovery from transient failures Total: 45 GBs

More Related