1 / 5

GridFTP issues for SC3

GridFTP issues for SC3. standard GT2 gridftpd has problems under high load code is no longer supported should use new GT4 gridftpd instead: new, complete reimplementation of the server support for striping (using multiple hosts to move a single file)

gunnar
Download Presentation

GridFTP issues for SC3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GridFTP issues for SC3 • standard GT2 gridftpd has problems under high load • code is no longer supported • should use new GT4 gridftpd instead: • new, complete reimplementation of the server • support for striping (using multiple hosts to move a single file) • to come: automatic buffer size determination • ease new feature additions and modifications of the server: • new commands • new data sources/sinks such as MSS • transport protocols other than TCP • automatic data processing (e.g. tar) • resolves a commercial licensing issue with the WU-ftpd Maarten Litmaath (CERN/IT/GD), LCG Storage Management Workshop, 2005/04/05

  2. Remote data-nodes and striped operation • hostB> globus-gridftp-server -p 6000 -dn • hostC> globus-gridftp-server -p 7000 –dn • hostA> globus-gridftp-server -p 5000 -r hostB:6000,hostC:7000 • Client: globus-url-copy -stripe gsiftp://hostA:5000/file file:///dest globus-url-copy -stripe gsiftp://hostA:5000/file gsiftp://hostX/dest • Why “-stripe” option needed? Maarten Litmaath (CERN/IT/GD), LCG Storage Management Workshop, 2005/04/05

  3. Known problems • e-block mode: connect in data flow direction vs. firewalls • will be fixed in mode X • host and port must be given before file is known • also for mode X • data port range vs. firewalls • no plans yet • need for group-writable files and directories • no plans yet • others…? Maarten Litmaath (CERN/IT/GD), LCG Storage Management Workshop, 2005/04/05

  4. Tests • performance tests at Argonne: • 27/30 Gb/s for 32 nodes mem-mem, 2 GB/s disk-disk • U. of Chicago + PlanetLab scalability tests, 1 server at ISI: • 10 MB file/request from disk to /dev/null on server: • 363 GB/3h (37,200 files), 500 clients/100 nodes • 131 GB/1h (13,425 files), 1100 clients/130 nodes • 917 GB/6h (93,858 files), 1100 clients/100 nodes • 151 GB/1h (15,428 files), 1800 clients/130 nodes • limitiations from server 1 Gbit network link Maarten Litmaath (CERN/IT/GD), LCG Storage Management Workshop, 2005/04/05

  5. Practical issues • new gridftpd uses new Globus XIO, OpenSSL 0.9.7, ... • not yet available through VDT • DNs with "email=" in OpenSSL 0.9.7 vs. 0.9.6 • put hacks in grid-mapfile • new features can only be used when both ends implement them • does feature negotiation always work? • Globus gridftpd vs. dCache vs. Castor vs. ... • dCache vs. DCAU • Plan: evaluate and package new gridftpd well before SC3 Maarten Litmaath (CERN/IT/GD), LCG Storage Management Workshop, 2005/04/05

More Related