1 / 8

Xrootd Demonstrator description, schedule and results

Xrootd Demonstrator description, schedule and results. Doug Benjamin Duke University. Description. XRootd makes an excellent data discovery and data transfer system when used with a global name space It is used to augment and not replace the existing DDM system

sworley
Download Presentation

Xrootd Demonstrator description, schedule and results

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Xrootd Demonstrator description, schedule and results Doug Benjamin Duke University

  2. Description • XRootd makes an excellent data discovery and data transfer system when used with a global name space • It is used to augment and not replace the existing DDM system • Tier 1 and Tier 2 sites would act as seeds for the data sources (Read only) • See Charles’ talk from Monday and next on the technology • Tier 3 both fetch the data and serve it (sink and source – reducing load between T1/T2 and T3)

  3. Advantages • Tier 1 and Tier 2 sites can limit the bandwidth using an existing add-on to Xrootd (overloading the system not an issue) • Tier 3 sites would participate in delivering data to other Tier 3 sites • Popular data sets would be replicated as required based on usage (similar to PD2P) • Tier 3 sites co-located with Tier 2 (or Tier 1) sites could use XRootd data servers attached T2 storage for read only access • CERN based physicists can easily fetch their own ROOT files from their home institute clusters to their local desktop

  4. Stand alone T3 - Caching files with frm #2 Who has /atlas/foo? #9 Who has /atlas/foo? xrootd xrootd xrootd xrootd xrootd xrootd frm xrootd Xfr agent cmsd cmsd cmsd cmsd cmsd cmsd cmsd Xrdcp -x root://LR//atlas/foo /tmp Data Files Data Files Data Files Data Files Local Redirector Data server B #3 No local data servers have data it. B said it could get data initially Client App #1 open(“/atlas/foo”); /atlas/foo Remote Data server D Remote Data server C Remote Redirector Data server A Global Redirector #4 Try B #7 open (“/atlas/foo”) skip my redirector #6 get /atlas/foo #14 got /atlas/foo # 12 open /atlas/foo Same node # 13 send /atlas/foo # 11 Try server C #5 open /atlas/foo /atlas/foo # 10 I do! # 8 Who has /atlas/foo ? The xrootd system does all of these steps automatically without application (user) intervention! 4

  5. XRootD transfer service pluggin/Status • Transfer process calls simple shell script to fetch the data • Currently testing with XRootd copy command xrdcp in simple bash script Could use another transfer command if needed • Proof of principle testing complete • Able to copy files from Tier 2 sites to Tier 3 sites.

  6. Open issues • Proxy service • Some sites have data servers on private networks need a Proxy machine route the files and communication • Solutions exist; need to test options • Authentication • Users can only read data • Write activity triggered by trusted servers • Is additional Authentication needed? • Lighter weight option - simple shared secret • Heavy weight option - X509 is also possible • Has performance implications.

  7. Schedule • By Friday trigger Tier 3 to Tier 3 transfer • Within 1 week, test Tier 3 user job trigger data transfer. • Within 2 weeks, make Charles NtoN libraries available to European Lustre and GPFS sites • During December test with 3-4 T2’s and up to 5 T3 sites (including European sites) limited tests • January – Mid February – ramp up testing, establish ganglia monitoring (see Andy’s talk for Xrootd monitoring) • Late Jan. – early February – ATLAS evaluation • 9 Feb – WLGC meeting.

  8. Conclusions • Xrootd is a good match for data deliver amongst Tier 3 sites while reducing load on rest of DDM system • Existence of ATLAS global name space is vital • Should help us keep on top of users data needs

More Related