Reliable File Transfer: Lessons Learned

1. Reliable File Transfer:Lessons Learned Bill Allcock, ANL Ravi Madduri, ANL

2. Overview • Quick Overview of the Service • Design Issues • Lifetime Management • Virtualization • Service Data Elements • Implementation Issues • SOAP Processing Issues • Standardization • Avoid Language specific Types • Anyhelper API • Fault Tolerance

3. RFT in Action Grid Service Container Registry RFT Factory 1. A Grid Service Container is started up; It contains an RFT Factory service; The RFT Factory service registers itself * The scenarios in this presentation are offered as examples and are not prescriptive

4. RFT in Action Grid Service Container Registry RFT Factory 2. From a known registry, the client discovers a factory by querying the Service data of the registry Client * The scenarios in this presentation are offered as examples and are not prescriptive

5. RFT in Action Grid Service Container 3. The client calls the createService operation on the factory and passes in a TransferRequest RFT Factory Client * The scenarios in this presentation are offered as examples and are not prescriptive

6. RFT in Action Grid Service Container RFT Factory RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State 4. The instance is started, and the factory returns a locater Client * The scenarios in this presentation are offered as examples and are not prescriptive

7. RFT in Action Grid Service Container RFT Factory 5. Client calls Start(), subscribes to notificaitons, etc. RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State Client * The scenarios in this presentation are offered as examples and are not prescriptive

8. RFT in Action • Service is OGSI compliant • Uses existing GridFTP (non-OGSI) protocols and tools to execute 3rd Party Transfer for the user • Provides extensive state transition notification RFT ServiceInstance GridFTPServer GridFTPServer * The scenarios in this presentation are offered as examples and are not prescriptive

9. Lifetime Management • Lifetime management is a key aspect of OGSI • Was not intuitively clear how to handle this for “disconnected” services • Our (perhaps not optimal) solution is to give it an indefinite lifetime • Should there be an activity monitor? • Does that really solve the problem? • Any other ideas?

10. Virtualization • Another Key Design Issue in services • We virtualize data movement • It works, LBL and ANL have interoperable implementation • Need to standardize… • Data Services Virtualization from DAIS • Should we pass around GSHs of file services rather than URLs? • Granularity • Single file .vs. Multi-file .vs. service composition

11. Service Data Elements • A huge improvement over the non-OGSI services • Information Services are (should be) baked in to the services • Defined both push (event notification) and pull (full transfer status) SDEs • Need to be cognizant of size, frequency, and performance of notifications • Lots of interesting possibilities: bandwidth, errors, network status, etc..

12. Implementation Issues • SOAP Deserialization • Deserialization can be a HUGE issue. Our original (very simple) XML could take up to 30 minutes to process. • A straight forward change reduced that drastically. • SOAP engine also needs to be looked at. • Standardization • Critical for success • Will be moving to the OGSI-Agreement interface

13. Implementation Issues (cont.) • Language Specific Data Types • Avoid them (I.e. Java vector type) • A python based service would not be able to deserialize that. • AnyHelper API • Will deserialize any general XML blob that utilizes basic types.

14. Implementation Issues (cont.) • Fault Tolerance • Multiple levels of Fault tolerance • GridFTP will handle remote failures • RFT provides fault tolerance of request via JDBC compliant database • Service container provides instance fault tolerance • We write only the primary key into the wsdd file to avoid slowing down container restart.