1 / 21

Storage Update Entirely About Storage & SC4

Storage Update Entirely About Storage & SC4. GridPP 15 j.jensen@rl.ac.uk. Deployment. Preparing for SC4 All sites must have SRM No classics Aiming for SRM 2.1 Deployed in parallel with 1.1 DPM already provides a 2.1 interface. Status. (Nearly) all sites have SRM 1.1

minty
Download Presentation

Storage Update Entirely About Storage & SC4

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Storage UpdateEntirely About Storage & SC4 GridPP 15 j.jensen@rl.ac.uk Storage Middleware

  2. Deployment • Preparing for SC4 • All sites must have SRM • No classics • Aiming for SRM 2.1 • Deployed in parallel with 1.1 • DPM already provides a 2.1 interface Storage Middleware

  3. Status • (Nearly) all sites have SRM 1.1 • Either DPM or dCache (or both) • Nearly? • Occasional staff or hardware problems • Those who don’t have one in prod’n have started installations and testing • All classics accounted for • Decommissioning or migrating (DPM) Storage Middleware

  4. Status • Mailing list + community works well • ~45 members • Focus: SEs, and FTS/GridFTP/etc only when SE related • Lots of good stuff in the Wiki • Helped NGS deploy SE (DPM) • For gLite evaluation Storage Middleware

  5. SRM 2.1 • Required for SC4 • Developed extensive test clients • Testing DPM’s 2.1 interface • Depend on FNAL delivering 2.1 for dCache • Says Timur: “mid Jan” Storage Middleware

  6. dCache status • Upgrade • Most dCache Tier 2s have 1.5.{2,3} • Tier 1 had 1.6.5, now 1.6.6.1 • YAIM support • Wholly managed by GridPP • Good relations to DESY • 1.6.6.2 to go into LCG “2.7.0” • Upgrade Tier 2s to 1.6.6 • 1.6.6 has much improved logging Storage Middleware

  7. DPM status • Good collaborations between GridPP and LCG • GIP plugin in LCG “2.7.0” • We’re testing the 2.1 interfaces • Upgrade Classic SEs to DPM experiences Storage Middleware

  8. interoblems • SRM 2.1 is really SRM 2.1.1  • Two versions: 2.1.1 and 2.1.1-modified  • 2.1.1 has a more soapy array handling  • Most are now implementing 2.1.1-modified • DPM has switched, as of 1.4.1 • Publishing? • Both 1.1 and 2.1 are called “SRM” ? ? Storage Middleware

  9. interoblems • Much of the API is optional • Some things are genuinely optional • Clients don’t use full API so why implement all • “Extra” features • Access control (StoRM (INFN)) • Interop is more than using same WSDL • API is quite complex (much more than 1.1) • Semantics not spec’d in WSDL Storage Middleware

  10. interoblems SRM2 SRM2 put cycle srmCopy The Chocolate and Vanilla and Strawberry problem (cf. current dCache to DPM problems) Client srmCopy SRM2 SRM2 Storage Middleware

  11. interoblems • Not all sites will upgrade at the same time • Some sites will continue to publish 1.1 • Even DPM has two different daemons • One for 1.1 and another one for 2.1 • So they compete for 8443  • Must be able to upload via one interface and download via the other  Storage Middleware

  12. Short Term Workaround Colourful solution Each flavour can talk to any other flavour Storage Middleware

  13. Short Term Workaround Common client can talk to all flavours Storage Middleware

  14. Use get/put for file transfer for now… GridFTP 3rd party xfer SRM2 SRM2 srmGet srmPut Client Storage Middleware

  15. Remember Don't Panic Storage Middleware

  16. Patience • SRMs will interoperate • SRMs are converging • Needs testing though • They will use 2.1.1-modified • They will be able to use srmCopy between SRMs  Pessimist Optimist Storage Middleware

  17. SC4 risks • Deployment/functionality/interoperability of SRM 2.1 delayed • Keep 1.1s around • Use FTS – supposed to be able to speak to all • dCache  DPM transfer problem • Very serious – high impact • Problem not fully understood – developers informed • Problem with both lcg-* and FTS • Mitigation? Storage Middleware

  18. SC4 risks • SRM2 for CASTOR2 (dev’d at RAL) • Will affect the Tier 1 – eventually (Julyish) • Critically important for CERN • CASTOR2/SRM2 + robot upgrade • Will affect Tier 1 – eventually • FTS [talking to SRM 2.1] delayed • Keep using SRM 1.1 • Migration from YAIM to gLite XML tool • Stick with YAIM for now… Storage Middleware

  19. Metrics/Monitoring • Basic metrics: • # sites with SRMs deployed • Amount of storage published (c’ly 160 TB in T2) • Monitoring • Need for more extensive SE tests than SFT • Deploying SRM monitoring infrastructure • Exercising 1.1 up/download & 3rd party & del • Flexible – can eventually handle 1.1s and 2.1s Storage Middleware

  20. Other Implementations • Not supported for Tier 2 deployment • CASTOR2 SRM • Developed at RAL • Deployed at CERN (other CASTOR sites later) • StoRM • Developed at INFN • Not tested in UK yet • Thin layer, depends on underlying filesystem Storage Middleware

  21. Summary • All hands on SC4 preparations • Pretty well on track • Thanks to community working together • Some risks • Particularly external dependencies • Mitigating whenever possible • Taking pessimised view on interoblems • And don’t panic ! Storage Middleware

More Related