1 / 17

Light weight Disk Pool Manager status and plans

Learn about the features and deployment of CERN's Lightweight Disk Pool Manager, a solution focused on manageability and support for Tier-2s in LCG-2 with easy configuration, maintenance, and scalability.

philipe
Download Presentation

Light weight Disk Pool Manager status and plans

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Light weight Disk Pool Manager status and plans Jean-Philippe Baud, IT-GD, CERN May 2005

  2. Disk Pool Manager aims • Provide a solution for Tier-2s in LCG-2 • This implies a few tens of Terabytes in 2005 • Focus on manageability • Easy to install • Easy to configure • Low effort for ongoing maintenance • Easy to add/remove resources • Support for multiple physical partitions • On one or more disk server nodes • Support for different space types – volatile and permanent • Support for multiple replicas of hot files within the disk pools CERN IT-GD

  3. Manageability • Few daemons to install • No central configuration files • Disk nodes request to add themselves to the DPM • All states are kept in a DB (easy to restore after a crash) • Easy to remove disks and partitions • Allows simple reconfiguration of the Disk Pools • Administrator can temporarily remove file systems from the DPM if a disk has crashed and is being repaired • DPM automatically configures a file system as “unavailable” when it is not contactable CERN IT-GD

  4. Features • DPM access via different interfaces • Direct Socket interface • SRM v1 • SRM v2 Basic • Also offer a large part of SRM v2 Advanced • Global Space Reservation (next version) • Namespace operations • Permissions • Copy and Remote Get/Put (next version) • Data Access • Gridftp, rfio (ROOTD, XROOTD could be easily added) CERN IT-GD

  5. Security • GSI Authentication and Authorization • Mapping done from Client DN to uid/gid pair • Authorization done in terms of uid/gid • Ownership of files is stored in DPM catalog, while the physical files on disk are owned by the DPM • Permissions implemented on files and directories • Unix (user, group, other) permissions • POSIX ACLs (group and users) • Propose to use SRM as interface to set the permissions in the Storage Elements (require v2.1 minimum with Directory and Permission methods) • VOMS will be integrated • VOMS roles appear as a list of gids CERN IT-GD

  6. Architecture • The Light Weight Disk Pool Manager consists of • The Disk Pool Manager with its configuration and request DB • The Disk Pool Name Server • The SRM servers • The RFIOD and DPM-aware GsiFTP servers • How many machines ? • DPM, DPNS, and SRM can be installed on the same one • RFIOD : on each disk server managed by the DPM • GsiFTP : on each disk server managed by the DPM CERN IT-GD

  7. Example: SRM put processing (1) CERN IT-GD

  8. Example: SRM put processing (2) CERN IT-GD

  9. Example: SRM put processing (3) CERN IT-GD

  10. Example: SRM put processing (4) CERN IT-GD

  11. Example: SRM put processing (5) CERN IT-GD

  12. Example: SRM put processing (6) CERN IT-GD

  13. Example: Pool definition • dpm-addpool --pool Volatile --def_filesize 200M –def_pintime 600 --s_type V • dpm-addfs --pool Volatile --server se001 --fs /data01 • dpm-qryconf • POOL Volatile DEFSIZE 200.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEFPINTIME 600 PUT_RETENP 86400 FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GID 0 S_TYPE V • CAPACITY 17.73G FREE 17.73G (100%) • se001 /data01 CAPACITY 17.73G FREE 17.73G (100.0%) CERN IT-GD

  14. Testing • Test suites written by Gilbert Grosdidier (Orsay) and Jiri Kosina (Prague) • Functionality tests (120 sub-tests) • Stress tests (40 suites running in parallel) • The tests were run from very early in the development cycle and were improved as soon as new functionality was available in the servers • The test results were immediately taken into account for the code development CERN IT-GD

  15. Status • DPM will be part of LCG 2.5.0 release but is available from now on for testing CERN IT-GD

  16. Documentation/Packaging/Installation • Reference man pages provided for each method and each server • Installation guide being written • The DPM is available as a tarball or as a set of RPMs • DPM-server-mysql • DPM-name-server-mysql • DPM-srm-server-mysql • DPM-rfio-server • DPM-client • DPM-gridftp-server • Installation scripts (Yaim) are also available CERN IT-GD

  17. Deployment • Replacement of ‘Classic SE’ • Only metadata operations needed (the data does not need to be copied) • Satisfies gLite requirement for SRM interface at Tier-2s CERN IT-GD

More Related