1 / 8

BeStMan and Xrootd Alex Sim Scientific Data Management Research Group

BeStMan and Xrootd are powerful tools for managing scientific data. BeStMan is an SRM implementation that provides dynamic space allocation and file management, while Xrootd offers high-performance file-based access. Together, they enable efficient storage and retrieval of large datasets in research environments.

lestersturm
Download Presentation

BeStMan and Xrootd Alex Sim Scientific Data Management Research Group

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BeStMan and Xrootd Alex Sim Scientific Data Management Research Group Computational Research Division Lawrence Berkeley National Laboratory

  2. What is SRM? • SRM : Storage Resource Manager • Well-defined storage management interface specification based on standard • Different implementations for underlying storage systems are based on the same SRM specification • Provides dynamic space allocation and file management on shared storage components on the Grid • Over 300 deployments of different SRMs in the world • Managing more than 10 PB

  3. BeStMan • Light-weight implementation of SRM v2.2 • Works on existing disk storages with posix-like file systems • E.g. NFS, GPFS, GFS, NGFS, PNFS, HFS+, PVFS, Lustre, Xrootd, Hadoop, Ibrix • Supports multiple partitions • Adaptable to other file systems and storages • Supports customized plug-in for MSS to stage/archive such as HPSS • Easy adaptability and integration to special project environments • Supports two mode • Full mode : management on request queues and spaces • Gateway mode : faster performance without queue/space management • Supports multiple transfer protocols • Supports load balancing for multiple transfer servers • Scales well with some file systems and storages • Xrootd, Hadoop, GPFS, PVFS, etc • Works with grid-mapfile or GUMS server • Simple installation and easy maintenance • Packaged in VDT using Pacman • Who would benefit from BeStMan? • Sites with limited resources and/or limited admin effort

  4. Xrootd and XrootdFS • Developed at SLAC and INFN Padova • Suite of tools for serving data • server daemons, clients (ROOT TXNetFile, POSIX), use xrootd protocol • Provides high performance file-based access • Scales on server and client side • Separate data flow and control flow • Fault tolerant • Secure • rootd compatible • Xrootd provides a Posix I/O interface • in addition to the native XTNetFile class • XrootdFS provides a Posix File System layer • Mounts the Xrootd storage cluster to a desktop machine • Provides a complete view of storage tree • Base on FUSE, implemented using Xrootd Posix I/O library

  5. How it works all together in PUT/GET in Ls/Rm/Mkdir/Rmdir Client Client XrootdFS XrootdFS Xrootd Xrootd GridFTP file transfers srmPrepareToGet/Put TURL srmReleaseFiles/srmPutDone srmLs/srmRm/srmMkdir/srmRmdir BeStMan Gateway BeStMan Gateway Gridftp server Gridftp server Gridftp server Gridftp server Gridftp server Gridftp server File system File System . . . . . .

  6. Some Use Cases • CMS • BeStMan Gateway as an SRM frontend for Hadoop at UNL • Passed all the automated CMS tests through EGEE SAM at UNL • ATLAS • BeStMan on NFS • BeStMan Gateway on Xrootd/FS, GPFS, Ibrix • STAR • Data replication between BNL and LBNL • HPSS access at BNL and NERSC • SRMs in production for over 4 years • Part of analysis scenario to move job-generated data files from PDSF/NERSC to remote BNL storage • Earth System Grid • Serving about 6000 users • Over a million files and 170TB of climate data • from 5 storage sites (LANL, LLNL, NCAR, NERSC, ORNL) • Uses an adapted BeStMan for NCAR’s own MSS

  7. Summary • BeStMan is an implementation of SRM v2.2. • Great for smaller disk-based storage and file systems • BeStMan Gateway mode on some file systems and storage gives scalable performance • Install/maintain through VDT • Works with other SRM v2.2 implementations • Servers: CASTOR, dCache, DPM, StoRM, SRM/SRB, … • Clients: FTS, PhEDEx, glite-url-copy, lcg-cp, srm-copy, srmcp, … • In OSG, WLCG/EGEE, ESG, …

  8. Documents and Support • BeStMan • http://datagrid.lbl.gov/bestman • http://hep-t3.physics.umd.edu/HowToForAdmins.html#osgBestman • http://wt2.slac.stanford.edu/xrootdfs/bestman-gateway.html • https://www.usatlas.bnl.gov/twiki/bin/view/Admins/BestMan • https://twiki.grid.iu.edu/bin/view/Documentation/BestmanGateway • https://twiki.grid.iu.edu/bin/view/Documentation/BestmanGateway-Xrootd • Xrootd • http://xrootd.slac.stanford.edu • http://wt2.slac.stanford.edu/xrootdfs/xrootdfs.html • SRM Collaboration and SRM Specifications • http://sdm.lbl.gov/srm-wg • Contact and support : srm@lbl.gov

More Related