1 / 10

Rucio & objectstores

This article discusses the implementation of Rucio's support for objectstores like Amazon S3, OpenStack Swift, and Google Cloud Platform. It covers the limitations of existing support, the addition of signed URLs for uploads and downloads, code clean-up, and future plans.

terrygarcia
Download Presentation

Rucio & objectstores

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Rucio & objectstores James Perry

  2. Rationale • Cloud-style objectstores are becoming increasingly popular for data storage • Amazon S3 • OpenStack Swift • Google Cloud Platform • Existing Rucio support for objectstores had limitations: • AWS S3 support required clients to store objectstore credentials • OpenStack Swift not properly supported at all • Google Cloud Platform included URL signing support in Rucio core • But only used for downloads • Code was untidy with hacks, pseudo-protocols, special casing, etc.

  3. S3 download in Rucio (old method) Client Objectstore Credentials Data file Credentials

  4. Signed URLs • Objectstores support signed URLs (TempUrls in Swift) • URL, operation (GET, PUT, DELETE) and expiry time are cryptographically signed with a secret key • Server checks the URL signature and enforces operation and time restrictions • Can be generated offline (at least in Swift and in newest version of S3) • Instead of giving objectstore credentials to clients, keep them on the Rucio server and have the server generate signed URLs for the clients

  5. Upload and download: completed • Added URL signing for S3 and Swift to Rucio core • Alongside existing GCP implementation • Downloads worked immediately • Rucio already signs the URLs returned by list_replicaswhen required • Added support for uploading to signed URLs • Required new Rucio API endpoint to get signed URL for file that does not yet exist • More complicated than downloads because (e.g.) checksums and existence checks cannot be performed on signed PUT URL

  6. S3 download in Rucio (new method) Rucio Server Credentials Request Signed URL Credentials Client Objectstore Signed URL Data file

  7. Deletion: completed • URL signing code already added to core supported DELETE operation • Added a call to sign URL when required to Rucio’s reaper daemon • A bug in underlying GFAL2 library prevented this from working • It performed a stat on the URL, not allowed on URL signed for DELETE operations • Fixed in GFAL 2.16.2

  8. 3rd party transfers (delegated to FTS) • Rucio uses FTS3 for all 3rd party (RSE-to-RSE) transfers • Signed URLS are not used for this • Technically possible but undesirable as it could result in leakage of signed URLs • Instead, FTS holds the objectstore credentials and performs authentication itself

  9. Code clean up (in progress) • Legacy objectstore code is being removed from Rucio • Fits well with core Rucio team’s current focus

  10. Future Plans • Event Service • Check that URL signing performs well enough for heavy usage by the Event Service • Auditing • Check that the existing Rucio auditing code works with objectstores and make any changes necessary • Dynafed • Can be used to make objectstores appear more like traditional grid storage systems • Check that it works properly with Rucio • Already in progress at CERN • Take advantage of objectstores fully, don’t just use as file systems • Store DUNE events as objects • Could implement a QoS prototype using them

More Related