1 / 18

Future Proof S torage with DPM

Future Proof S torage with DPM. Oliver Keeble (on behalf of the CERN IT-GT-DMS section). DPM today. Disk storage for the grid 36PB 10 sites with > 1PB Over 200 sites in 50 regions Over 300 VOs have access to DPMs. DPM today. Current production version is 1.8.3

dorak
Download Presentation

Future Proof S torage with DPM

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Future Proof Storage with DPM Oliver Keeble (on behalf of the CERN IT-GT-DMS section)

  2. DPM today • Disk storage for the grid • 36PB • 10 sites with > 1PB • Over 200 sites in 50 regions • Over 300 VOs have access to DPMs Future Proof Storage with DPM, EGI TF Prague

  3. DPM today • Current production version is 1.8.3 • https://svnweb.cern.ch/trac/lcgdm/blog/official-release-lcgdm-183 • In EPEL, EMI1 and EMI2 • No gLite – see tutorial on Wed on how to upgrade • What’s new • EPEL compliance • HTTP/DAV frontend (old dpmhttpd is gone) • NFSv4.1 frontend (read only) • Thread safe clients • Synchronous get requests • … Future Proof Storage with DPM, EGI TF Prague

  4. 1.8.4 • Next core release: 1.8.4 • First DMLITE release (see later…) • DAV frontend will start using the new libraries • Other components not using dmlite for now • Improved replication mechanism • 32bit client support • Lots of other small fixes • Feature complete… finishing certification • Change in the release process • Independent component releases • Faster, lighter releases Future Proof Storage with DPM, EGI TF Prague

  5. Refactoring & DMLITE • DMLITE is the result of a significant refactoring effort to make DPM modular • Better separation between frontends and backend • Cleaner, more open, much improved performance • Improved integration with standard building blocks • Hadoop, Memcache, S3, Lustre, … Future Proof Storage with DPM, EGI TF Prague

  6. Improved Frontends • Standard protocols, standard clients • HTTP/DAV • + extras, WAN transfers, 3rd party copy… • NFS 4.1/pNFS (r/o) • Xrootd (rewritten) • Forthcoming • GridFTP • Legacy interfaces remain untouched • No large scale revalidation required Future Proof Storage with DPM, EGI TF Prague

  7. Improved Frontends • Standard protocols, standard clients • HTTP/DAV • + extras, WAN transfers, 3rd party copy… • NFS 4.1/pNFS (r/o) • Xrootd (rewritten) • Forthcoming • GridFTP • Ubiquitous access to grid storage • Legacy interfaces remain untouched • No large scale revalidation required Future Proof Storage with DPM, EGI TF Prague

  8. Improved Backends • This is where DMLITE shines • Plugin based, open for constant evolution • Improved nameserver performance • Connection pooling, improved SQL, memcache layer, … • Support for multiple pool types • Legacy DPM, Hadoop/HDFS, S3, … • Sharing a single namespace if desired • Possibility for opportunistic pools • Federation • See the following presentation • And this is the beginning, much more coming • Python bindings, Lustre, VFS, … Future Proof Storage with DPM, EGI TF Prague

  9. Improved Backends • This is where DMLITE shines • Plugin based, open for constant evolution • Improved nameserver performance • Connection pooling, improved SQL, memcache layer, … • Support for multiple pool types • Legacy DPM, Hadoop/HDFS, S3, … • Sharing a single namespace if desired • Possibility for opportunistic pools • Federation • See the following presentation • And this is the beginning, much more coming • Python bindings, Lustre, VFS, … Future Proof Storage with DPM, EGI TF Prague

  10. I/O performance PRELIMINARY PRELIMINARY PRELIMINARY Future Proof Storage with DPM, EGI TF Prague

  11. Performance, performance, performance https://cdsweb.cern.ch/record/1458022?ln=en Future Proof Storage with DPM, EGI TF Prague

  12. Easy administration • Puppet for configuration • Popular among large data centers • Lots of modules for popular tools (which we now rely on – apache, memcache, nagios, …) • Nagios for monitoring • We reuse as much as possible • And there’s a lot already available • Added specific plugins for detailed status and performance monitoring Future Proof Storage with DPM, EGI TF Prague

  13. Easy administration • Puppet for configuration • Popular among large data centers • Lots of modules for popular tools (which we now rely on – apache, memcache, nagios, …) • Nagios for monitoring • We reuse as much as possible • And there’s a lot already available • Added specific plugins for detailed status and performance monitoring Future Proof Storage with DPM, EGI TF Prague

  14. Why “future proof”? • Standards • Leverage existing components & clients • Mature ecosystem • Less maintenance work • Community • Guidance by the stakeholders • Independence from funding cycles • Now in talks for the creation of a “DPM Collaboration” to drive the project post-EMI Future Proof Storage with DPM, EGI TF Prague

  15. The DPM Collaboration • DPM is the most numerous SE on the WLCG infrastructure • The project has never been in better shape • We are soliciting statements of support for a collaboration • In discussions with • GridPP (UK) • WLCG France • Taipei (WLCG Tier 1 using DPM) Future Proof Storage with DPM, EGI TF Prague

  16. DPM around the world Future Proof Storage with DPM, EGI TF Prague

  17. Summary • DPM has received a lot of investment thanks to EMI • These developments are now being released • This investment has gone into • Making it modular • Supporting standards (HTTP, NFSv4.1,…) • Profiting from existing technology • Performance • Manageability Future Proof Storage with DPM, EGI TF Prague

  18. Recent Issues • Issue with LFC API / Py26 (#84716) • Tracked down to issue with EPEL5 Swig version • Currently considering the best solution • Cleaning up invalid LFC replicas (#83335) • Solved… bulk requests using API/CLI tools • DAV EMI2 install failure (#85141) • Related to dependencies on gridsite, solved with the EPEL update • EMI2 LFC dies regularly (#85161) • Happens to any of lfc/dpm/dpns daemons • Documented as a known issue with 1.8.3 • https://svnweb.cern.ch/trac/lcgdm/blog/official-release-lcgdm-183 • 32 bit support (#81508) • Available in EMI2 and EPEL repositories Future Proof Storage with DPM, EGI TF Prague

More Related