150 likes | 280 Views
DPM Basics and its status and plans. Wahid Bhimji University of Edinburgh GridPP Storage Workshop – Apr 2010. This talk: . Not covering all the basics of DPM many already know those. S ee gridpp wiki: http://www.gridpp.ac.uk/wiki/Disk_Pool_Manager or Sam’s talks in July:
E N D
DPMBasics and its status and plans Wahid Bhimji University of Edinburgh GridPP Storage Workshop – Apr 2010 Wahid Bhimji – DPM
This talk: • Not covering all the basics of DPM • many already know those. See gridpp wiki: http://www.gridpp.ac.uk/wiki/Disk_Pool_Manager or Sam’s talks in July: http://hepwww.rl.ac.uk/sysman/June2009/agenda.html • Will cover: • An introduction of terms. • Listsomecommonly used tools for reference • Details of recent developments and plans for discussion Wahid Bhimji – DPM
A “standard” DPM setup Pool Node Worker Node DISK rfio Pool Node DISK gridftp DPM Head Node Pool Node DISK … User … • “Pool” node (dpm-disk) • Directly mounted disk • RFIO– local transfers • GridFTP– WAN transfers • Head Node (dpm-mysql) • SRMV2 Interface • DPM server: Request tracking (MySQL) • DPM Nameserver (DPNS) (MySQL) Wahid Bhimji – DPM
Available tools • Some built-in and some in the gridpp toolkit: http://www.gridpp.ac.uk/wiki/DPM-admin-tools Latest version 2.6.3 • Let Sam or I know if you need another tool • For help ask the GRIDPP-STORAGE jiscmail list • or dpm-users-forum@cern for generic dpm questions Wahid Bhimji – DPM
Common activities • Install / upgrade: See GridPP Wiki • Look at space usage, list files • Move data around • Allocate or modify space: • “pools” given to VOs (or collection of VOs) • “Spacetokens” to groups within VOs Wahid Bhimji – DPM
Finding out whats on your storage • dpns-ls: List files in namespace • dpm-disk-to-dpns– translate filepath to dpns • dpm-list-disk - list all files on a filesystem Overall picture • dpm-listspace: Includes usage in spacetokens • dpm-qryconf: Includes usage in filesystem • dpns-du: For space usage within a namespace path. Wahid Bhimji – DPM
Moving files • dpm-delreplica or rfrm : remove from disk and namespace (but not LFC / remote catalogues) • dpm-replicate or rfcp : copy single file • dpm-drain: move a whole filesystem -Need to wrap in an “until” loop to persist if it crashes. • dpm-sql-spacetoken-replicate-hotfiles– (new(ish) in toolkit) – replicate all files within a spacetoken Wahid Bhimji – DPM
Allocating Space ALL ATLAS VO space should be in SPACETOKENS Create: dpm-reservespace Update dpm-updatespace- do not use dpm-reserve again Delete: dpm-releasespace Moving a filesystem between pools: dpm-rmfs --server pool2 --fs /gridstorage018 dpm-addfs --poolnamecms --server pool2--fs /gridstorage018 Wahid Bhimji – DPM
DPM Status Wahid Bhimji – DPM
Support • Last year central DPM support from CERN was at the level of 0.5 FTE. • Now improved to ~2 FTE: • Jean-Philippe Baud has returned • Additional developer : Ricardo Rocha Wahid Bhimji – DPM
Versions • Latest version – still 1.7.2(-5) • 1.7.3 (in certification since Oct.) Provides: • Checksum treatment • Fix for unstable SRM server (currently patched binary) • Future Program of work https://twiki.cern.ch/twiki/bin/view/LCG/DMProgramOfWork Wahid Bhimji – DPM
Future Plans - Highlights Wahid Bhimji – DPM
Long(er) term plans - Confirmed • Quotas. • Accounting: Space uses, files:read, written, replicated - bytes • Replace HTTPG by HTTPS for SRM servers • NFS 4.1 interface • VO admin role • Glue2 support “Not Scheduled” • Add GUIDs to file entries to be able to use DPM as local LFC • Automatic replication of hot files. attributes, not I/O. • Integration with Argus (banning of users and groups). • Better file system selection algorithm.. • Support for distributed file systems like Lustre/Hadoop Wahid Bhimji – DPM
Unconfirmed • Support for striping • Support for BitTorrent • Multi-vo support for xrootd • WEB admin interface • Gateway hosts to allow sites having very few IP addresses visible externally to use DPM • Provision of native python interfaces • Support for python 2.6 Wahid Bhimji – DPM
Conclusions / Discussion • DPM has many tools – use not always obvious • Significant support in UK – use us • Now good support at CERN too • And a plan of action • We can influence this • We can also help develop features of interest • E.g asked to help with Nagiousplugins • What are the priorities? Wahid Bhimji – DPM