1 / 13

DMLite GridFTP frontend

DMLite GridFTP frontend. Andrey Kiryanov IT/SDC 13/12/2013. How GridFTP works. Basically, it's a standard FTP with GSI on control channel and a bunch of extensions. A separate connection(s) is used for data transfer. Endpoint parameters are negotiated via control channel.

barth
Download Presentation

DMLite GridFTP frontend

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DMLiteGridFTP frontend AndreyKiryanov IT/SDC 13/12/2013

  2. How GridFTP works • Basically, it's a standard FTP with GSI on control channel and a bunch of extensions. • A separate connection(s) is used for data transfer. Endpoint parameters are negotiated via control channel. • Two standard FTP data connection modes: Passive(initiated by client) and Active (initiated by server). • GridFTP v2 adds many extensions, in particular a third data connection mode: Delayed Passive. DMLiteGridFTP frontend

  3. How it was integrated with DPM • Globus’ GridFTP server supports pluggable backend modules: DSIs (Data Storage Interfaces). • No need to implement a full FTP server from scratch, just provide some basic file I/O callbacks specific to your backend. • DPM interaction was implemented with a DSI. • Clients contact disk nodes directly. Advance SRM call is necessary to get a TURL with a hostname of specific disk node. • If a file is not available locally RFIO is used for transparent staging (slow). DMLiteGridFTP frontend

  4. GridFTP in DPM LFN Client SRM TURL Disk node 1 • • • RFIO GridFTP with TURL Disk node n Data connection DMLiteGridFTP frontend

  5. Transition from DPM to DMLite • GridFTP frontend is still based on Globus’ server and a DSI plugin. • Most parts of the plugin were completely rewritten using DMLite API (= expect new bugs). • DSI functionality was extended with support of redirection. • It is understood that there’s not much interest in SRM outside of HEP community. DMLiteGridFTP frontend

  6. Default deployment scenario • Almost no visible changes. • DPM DSI is replaced with DMLite DSI. • Clients still need to contact SRM prior to disk nodes. • Needs proper configuration of DMLite. • Namespace operations can go directly through DMLite MySQL plugin (faster, no need to contact DPNS). DMLiteGridFTP frontend

  7. GridFTP redirection • Data transfer channel is negotiated and established separately, not necessarily with the same server (third party transfers rely on this). • There’s a shortcoming in standard FTP finite-state machine: in Passive mode server address needs to be supplied to a client in advance, before the transfer command when file name is not yet known. • Globus’ GridFTP provides a third connection mode that works around this problem: Delayed Passive. It delays server address response until a file name is known to the server. DMLiteGridFTP frontend

  8. Standard Passive vs. Delayed Passive Client Server Client Server Connection Connection Capabilities Capabilities PASV command Delayed PASV Delayed Passive Standard Passive Transfer request IP address and port IP address and port Transfer request Data connection Data connection End of transfer End of transfer DMLiteGridFTP frontend

  9. Future deployment scenario • GridFTP is configured differently on a head node and on a disk servers. • Clients always contact the head node. Direct control connections to disk nodes are no longer supported. • Clients do not need to contact SRM, but if they do, a proper TURL with a head node is given. • “FTPHEAD” option in /etc/shift.conf should point to the head node. • Clients need to support Delayed Passive mode for optimal performance. • Older clients connecting to the head node without SRM will end up on a random disk node. A file transfer will be done with transparent RFIO staging to the right disk node (slow). DMLiteGridFTP frontend

  10. Redirecting GridFTP in DMLite Client GridFTP with LFN Head node Disk node 1 • • • RFIO Disk node n Data connection DMLiteGridFTP frontend

  11. Redirecting GridFTP with SRM LFN Client SRM TURL Head node GridFTP with TURL Disk node 1 • • • RFIO Disk node n Data connection DMLiteGridFTP frontend

  12. Interoperability • Current versions of GFAL2 and FTS3 already support Delayed Passive mode. • DPM already has all the necessary bits to support GridFTP redirection with SRM. • Clients that rely on Globus libraries can easily be modified to support Delayed Passive mode: • Use the globus_ftp_client_operationattr_set_delayed_pasv(), Luke! • Third party clients (e.g. uberftp) that do not support Delayed Passive mode will work with non-optimal performance involving RFIO fallback. • One can switch it off completely in DMLite configuration. DMLiteGridFTP frontend

  13. Thank you! DMLiteGridFTP frontend

More Related