XRootD Release 4 And Beyond

1. XRootD Release 4And Beyond GSI Seminar Stanford University/SLAC July15, 2015 Andrew Hanushevsky, SLAC http://xrootd.org

2. User Visible Changes I • New client library: libXrdCl.so • First available in 3.3.0 now is default • Old deprecated client library is libXrdClient.so • Copy command: xrdcopy • xrdcopy renamed to xrdcp • Xrdcopy is now a symlink to xrdcp • Original xrdcp renamed toxrdcp-old • All have same command line interface!

3. User Visible Changes II • New meta command: xrdfs • Logical replacement to xrd command • Different command line interface • More user friendly and intuitive • When installing Release 4 RPM • You must de-install previous XRootDrelease • They are incompatible

4. Root Support • New client is the default in ROOT • Since 5.34.14 (Dec 14, 2013) • Via TNetXNGFile plug-in • Setting envarXNet.UseOldClient to 1 • Loads TNetXFile instead (old client) • Root 6 does not support new client • Due to conflicts with PROOF • Expected to be resolved in release 6.1

5. Admin Visible Changes • IPv6 support • Public Private Networking • Disk Caching Proxy • HTTP Plug-in • Remote Debugging • Optional Client Plug-ins • Miscellaneous

6. IPv6 Support • The newclient supports IPv6 • It prefers IPv6 whenever possible • This corresponds to HEPIX request • Connects using mapped IPv4 addr if need be • The old client is deprecated • It will not support IPv6

7. IPv6 Configuration • Frankly none needed • Clients detect available network stacks • May force it to use various IP modes • Via configuration file or envars • Server detects interfaces and DNS settings • It’s important that DNS is setup correctly • May force server to use IPv4 only • Via command line option (-I v4)

8. IPv6 Considerations I • While servers and redirectors can accept both IPv6 and IPv4 clients. . . • IPv6 clients are assumed dual-stack • May be redirected to either IPv6 or IPv4 node • Client error recovery will resolve this if unworkable • This works if all the servers are dual-stack • Generally the preferred migration path • Since redirection is via hostname… • DNS entry must have no un-routable entries

9. IPv6 Considerations II • Client dual-stack assumption will be lifted in release 4.1 (3Q14) • Clients will be redirected to compatible nodes • If none, the client receives an error • Largely driven by public-private networks • And interactions with IPv6/4

10. Public-Private Network Support • Redirectors are now network cognizant • Servers inform redirectors of usable i/f’s • Clients always compatibly redirected • Private to private and public to public • Subject to configured network topology • Via new xrd.networkdirective option • Applies to servers and redirectors

11. Public-Private Net Topologies • xrd.network routes type [useif1[,if2]] • type: local | common | split • local (default) • No address differentiation (i.e. pre R4 mode) • common • Private incoming -> private (preferred) or public • Public incoming -> only public • split • Incoming address must match outgoing address • If use unspecified addresses come from DNS!

12. Private Addresses & DNS • Private addresses should never be in DNS • Unless • It’s a site local DNS server or • The address is zone registered (i.e. only locally available) • This is the assumption used by XRootD • To avoid DNS reverse translation timeouts • Hence, the use option might be needed • If a server connects using a private address

13. Public-Private Considerations • The available server interfaces • Must be uniform within a cluster • Must be compatible with configured topology • Restrictions relaxed in R 4.1 • Redirectors will match clients & server i/f’s • May lead to inaccessible files if no match exists • Still working through external access issues • May require a separate redirector for external access • Due to IPv6/4 and public-private network interactions

14. Networking Is Complex! • There are many combinations now • Client and server capabilities must now match • There are 4 basic combinations Public Private IPv6 IPv4 But things are not that simple! Dual stack clients add another 4 combinations

15. Disk Caching Proxy • New proxy server mode • Configured via pss.cachelib directive • Caches whole files or file segments • Mode is configurable • Cached content available for future access • Until LRU purged (configurable) • Many use cases to increase access speed

16. Typical Disk Caching Proxy Uses Speed up Remote Access Remote XRootD Clusters Caching Proxy Caching Proxy XRootD Cluster Caching Proxy XRootD Server Local Clients Speed up Random Access HDFS Speed up HD Access SSD FS

17. HTTP Plug-in • Basic http, https, WebDav access • Suitable for browsers, curl, wget, & davix • Provides another mode of well-known access • http is neither low latency nor high-performance • Google & Microsoft have proposed improvements • Changes submitted to W3C and IETF as http2 • Improvements are considered incremental • They only address the most vexing problems • Configured via xrd.protocol directive • And specialized http.xxx directives

18. XRootDMulti-Protocol Support • Always supported multiple protocols • Improved architecture makes it much easier • New protocols can leverage XRootDfeatures • Security, monitoring, file system plug-ins, etc XRootD Server Memory Based Protocol Conversion XRootD Bridge Clients Loadable Protocol

19. Remote Debugging via DigFS • XRootD provided pseudo file system • Provides restricted selectable R/O access to • Configuration file Log files • Core files /proc/self (Linux only) • Has authentication & authorization options • Including access control restrictions • View is standardized regardless of location • Configured via xrootd.diglib directive

20. The DigFS View conf /=/ conf/etc (site specific) Virtual exported path core/cmsd core/xrootd logs/cmsd logs/xrootd proc/cmsd proc/xrootd

21. DigFS Authorization • DigFS consults authorization file • Created by the site and specified in config file • xrootd.diglib * authfile + gsi host krb5 pwd sss unix + all [-]conf [-]core [-]logs [-]proc g=group h=host n=name o=org r=role Ö Ö Ö Ö Ö Ö Ö Ö Ö Ö Ö allow

22. XRootDClient Relationships • Plug-ins replace original implementation • All calls may be replaced • All layers above benefit with any code changes External Package XRootD- add on XRootD- core XRootD– plug-ins xrdcopy root Dirac FTS3 Athena XrdClFile XrdClPostMaster XrdClFilesystem xrdfs CopyProcess Gaudi PyXRootD CMSSW PROOF

23. Client Plug-ins • Plug-ins are loaded at run-time • Client looks for plug-in configuration files • 1st Locally: ~/.xrootd/client.plugins.d/ • 2nd Globally: /etc/xrootd/client.plugins.d/ • Both locations can be over-ridden via envar • XRD_PLUGINCONFDIR • Plug-ins are strictly version checked • Allows for independent development

24. Miscellaneous I • Readv proxy pass-through • Automatic & greatly improves performance • Manual log file rotation (e.g. logrotate) • Via extended –k command line option • High precision log file timestamps • New –z command line option • Log timestamp appears in microsecond format

25. Miscellaneous II • Special stat() plug-in for odd file systems • Configured via the oss.statlib directive • Includes plug-in for GPFS with tape backend • Cancellable third party copy • Used by transfer tools • Fast directory listings • Stat info can now be included w/ dir entry • xrdfs uses this to speed up long listings

26. Miscellaneous III • New query config options • Use the xrdfs command to display • Query cms - shows cmsd status • Query role - shows server’s cluster role • Query sitename - shows the site’s name • Query version - shows server’s version • Cluster node blacklisting • Via cms.blacklist directive & blacklist file • Useful in federated environments

27. Miscellaneous IV • New monitoring information • User login record now also includes • Name of the client’s executable • Contents of client’s XRD_MONINFO envar • Useful to tie external information to actual data usage • E.g. the Panda jobid to cross-reference I/O usage

28. Looking Beyond Release 4 • Cross Protocol Redirection • Meta-links • I/O Throttling Plug-in

29. Cross-Protocol Redirections I • XRootD protocol is capable of redirecting to a protocol other than xroot • On file open the server may tell the client that it’s more efficient to try something else • E.g. read the file locally from disk • Open(xroot://host/filename) -> redirect file://filename

30. Cross-Protocol Redirections II • New client already capable of processing out-of protocol redirections • Server needs some development to do so • Root’s TFile framework needs updating to handle a change in protocols • Changes already in development • Target for root 6.1 or 6.2

31. Meta-Link Files I • XML file that describes one or more data files available for access • Meta-link file identified by dot suffix • metalink (v3) or meta4 (v4 & incompatible w/ v3) • File is read and parsed by the client • Client picks one based on certain criteria • E.G. priority, location, etc • If access fails, client can pick another one

32. Metalink Files II • XRootDclient will support meta-link files • But the road is not straight-forward • Incompatible meta-link formats, which one? • Need to avoid encumbering installation • I.e. pre-reqs for sites that don’t care about meta-links • Targeting availability in 4Q14

33. I/O Throttling Plug-in • Allows site to limit I/O access to disk • Useful for throttling external access • Used in federated environments via proxy server • Configured via xrootd.fslib directive • And specific throttling directives • Plug-in is in code review phase • Already used by CMS in production • Targeting 3Q14

34. Acknowledgements • Current Software Contributors • ATLAS: Doug Benjamin, Patrick McGuigan, IlijaVukotic • CERN: Lukasz Janyst, Andreas Peters, Justin Salmon • Fermi: Tony Johnson • Root: Gerri Ganis, Bertrand Bellenot • SLAC: Andrew Hanushevsky,WilkoKroeger, Daniel Wang, Wei Yang • UCSD: MatevzTadel • UNL: Brian Bockelman • WLCG: MattiasEllert, FabrizioFurano, David Smith • US Department of Energy • Contract DE-AC02-76SF00515with Stanford University