1 / 16

SRM-Lite: overcoming the firewall barrier for large scale file replication

SRM-Lite: overcoming the firewall barrier for large scale file replication. Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory. April, 2007. First, some background: What are SRMs. SRM (DPM). SRM/ dCache. SRM/ CASTOR. SRM (StoRM). dCache. CASTOR.

gin
Download Presentation

SRM-Lite: overcoming the firewall barrier for large scale file replication

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SRM-Lite:overcoming the firewall barrierfor large scale file replication Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory April, 2007

  2. First, some background:What are SRMs SRM (DPM) SRM/ dCache SRM/ CASTOR SRM (StoRM) dCache CASTOR • Storage Resource Managers (SRMs) are middleware components whose function is to provide: • dynamic space allocation AND file management in spaces • for storage components on the local or wide-area network • Based on a common standard client/user applications SRM (BeStMan) GPFS Unix-based Disk Pools Unix-based Disk Pools CCLRC RAL Examples of some storage systems currently supported by SRMs

  3. SRM Functional Concepts • Manage Spaces dynamically • Reservation, lifetime • Negotiation • Manage files in spaces • Request to put files in spaces • Request to get files from spaces • Lifetime, pining of files, release of files • No logical name space management (done by replica location services) • Access remote sites for files • Bring files from other sites and SRMs as requested • Use existing transport services (GridFTP, https, …) • Transfer protocol negotiation • Manage multi-file requests • Manage request queues • Manage caches • Manage garbage collection • Directory Management • Uxix semantics: srmLs, srmMkdir, srmMv, srmRm, srmRmdir

  4. Example Use of SRMs in Earth Science Grid (in production for 3 years) 3100 users 120 TBs managed LBNL HPSS High Performance Storage System disk ANL CAS Community Authorization Services NCAR HRM Storage Resource Management gridFTP Striped server gridFTP server openDAPg server Tomcat servlet engine MyProxy server LLNL disk MCS client MyProxy client CAS client DRM Storage Resource Management RLS client DRM Storage Resource Management gridFTP server GRAM gatekeeper ORNL gridFTP server gridFTP HRM Storage Resource Management ISI gridFTP gridFTP server HRM Storage Resource Management MCS Metadata Cataloguing Services SOAP HPSS High Performance Storage System RLS Replica Location Services RMI MSS Mass Storage System disk disk LBNL’s SRMs inter-communicate between several sites and main portal site at NCAR to provide storage management and multi-file movement

  5. Disk Cache Disk Cache DataMover: SRMs used to provide large scale robust data streaming between sites • Problem: move thousands of files robustly • Takes many hours • Need error recovery • Mass storage systems failures • Network failures • Solution: Use Storage Resource Managers (SRMs) • File streaming paradigm • By reserving and releasing storage space automatically • Problem: too slow • Solution: • in GridFTP • Use parallel streams • Use large FTP windows • Pre-stage files from MSS • Use concurrent transfers Anywhere DataMover Get list of files SRM-COPY (thousands of files) NERSC BNL SRM-GET (one file at a time) SRM (performs writes) SRM (performs reads) GridFTP GET (pull mode) Network transfer archive files stage files Example setup for STAR high-energy-physics experiment

  6. SRM-Lite • Goal 1: automate file movement behind a firewall • a client program • to automate movement of multiple files • to/from client’s directory to a remote site • given a OTP firewall at one site • Support entire directory transfers • Recover from mid-transfer interruption and machine failure • Goal 2: pull files into user’s workstation • Use SRM-Lite by users to download files into their workstations • Using various transfer protocols (GridFTP, bbcp, https, …) • Have a GUI that shows transfer progress • Or have a command line • Support entire directory transfers • Support suspend/resume operations (e.g. on laptops)

  7. Process Steps Login to ORNL using OTP At ORNL invoke SRM-Lite User composes XML input file, srmlite.xml for selectedfiles/directories to copy from/to another site Or, user gives command lineoption for a selected file/directory SRM-Lite uses srmlite.xml orcommand line inputto automatically Push/Pull files to/from NERSC Use multiple threads for concurrent transfers SRM-Lite: a client program to automate movement of multiple files to/from client’s directory to a remote SSH servergiven a OTP firewall at one site Use Case A: OTP firewall at local site (ORNL), SSH server at remote site (NERSC) OTP Login ORNL NERSC SRM- Lite SSH Request SSH Server Local Commands srmlite.xml GridFTP/FTP/ SCP transfers Disk Cache Disk Cache HPSS

  8. Process Steps Login to ORNL using OTP At ORNL invoke SRM-Lite User composes XML input file, srmlite.txt for selectedfiles/directories to copy over to/from another SRM controlled storage system Or, user gives command lineoption for a selected file/directory SRM-Lite uses srmlite.xml orcommand line inputto automatically Push/Pull files to/from SRM at NERSC Use multiple threads for concurrent transfers Scenario: one end has SRM, The other end has a firewall, use SRM Use Case B: OTP firewall at local site (ORNL), SRM server at remote site (NERSC) OTP Login ORNL NERSC SRM- Lite SRM Request SRM srmlite.xml GridFTP/FTP/ SCP transfers Disk Cache Disk Cache HPSS

  9. Process Steps Login to ORNL using OTP At ORNL invoke SRM-Lite User composes XML input file, srmlite.xml for selectedfiles/directories to copy over to another site Or, user gives command lineoption for a selected file/directory SRM-Lite uses srmlite.xml orcommand line inputto automatically Push/Pull files to/from NERSC using either SSH or SRM Use multiple threads for concurrent transfers Scenario: one end has SRM, The other end has a firewall, use either SSH/SRM, Use Case C: OTP firewall at local site (ORNL), SRM/SSH server at remote site (NERSC) OTP Login ORNL NERSC SRM- Lite SSH Request SSH Server SRM Request SRM srmlite.txt GridFTP/FTP/ SCP transfers Disk Cache Disk Cache HPSS

  10. Process Steps Login to ORNL using OTP Create a OTP SSH tunnel to NERSC User composes XML input file, srmlite.txt for selectedfiles/directories to copy over to another site Or, user gives command lineoption for a selected file/directory SRM-Lite uses srmlite.txt orcommand line input, throughSSH tunnel to automatically Communicate with SRM-lite at other end Push/Pull files to/from NERSC using SRM Use multiple threads for concurrent transfers Can use SCP only Scenario: both ends have SRMs, both ends have a firewall, use SRM-lite on both ends, Use SSH to invoke SRM-Lite at other end SRM- Lite Use Case D: OTP firewall at both local site (ORNL), and remote site(NERSC), Use SRM-lite at both ends OTP Login ORNL NERSC OTP SSH Tunnel SSH-Tunneling SRM- Lite SRM Request srmlite.txt SRM SRM SCP transfers Disk Cache Disk Cache HPSS HPSS

  11. SRM-Lite: Status • SRM-Lite is developed • Available from: http://datagrid.lbl.gov/srmlite/ • Tested with GridFTP, SCP, HTTPS, HTTP • Tested with large number of files • Tested behind a firewall • Access from local SRMs that acess HPSS – not tested yet • Access between two filewalled system – not developed yet

  12. SRM-Lite: GUI • GUI was developed when used at user’s site(Linux, PC, MAC), called DataMover-lite (DML) • Available from: http://datagrid.lbl.gov/dml/ • Example GUIscreen • Shows info on:completed, active,and pendingtransfers • Also, file sizes,transfer times,transfer speed

  13. Extra Slides

  14. Storage Resource Managers SRMs are middleware components whose function is to provide: dynamic space allocation AND file management in spaces for storage components on the local or wide-area network Based on a common standard SRM/ L-Store SRM (StoRM) SRM/ CASTOR SRM/ CASTOR SRM/ dCache SRM (StoRM) SRM (Jlab-SRM) SRM (DPM) dCache CASTOR CASTOR client/user applications SRM (BeStMan) SRM (BeStMan) SRM (BeStMan) GPFS JASMine Unix-based disks Unix-based disks MSS CCLRC RAL Examples of storage systems currently supported by SRMs

  15. Process Steps User downloads DataMoverLite User goes to portal, select files Portal gets ALL files into SRM disk Portal generates XML input file,datamover.txt, for userselected files DML uses datamover.txt to automatically get files, and release files after move completes successfully DataMover-Lite use in ESG: a client program usedto automate movement of multiple files to client’s directory datamover.txt DataPortal NCAR User’s machine request ESG Portal User’s browser release DataMover Lite SRM GridFTP/FTP/ HTTP/HTTPS transfers Disk Cache Disk Cache MSS

  16. Another example of DML GUI

More Related