1 / 17

Protocols and Services for Distributed Data-Intensive Science

Protocols and Services for Distributed Data-Intensive Science. Bill Allcock, ANL ACAT Conference 19 Oct 2000 Fermi National Accelerator Laboratory Contributors: Ian Foster, Carl Kesselman, Steve Tuecke, Ann Chervenak. The Globus Data Grid. Two major components: 1. Data Transport and Access

azriel
Download Presentation

Protocols and Services for Distributed Data-Intensive Science

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Protocols and Services for Distributed Data-Intensive Science Bill Allcock, ANL ACAT Conference 19 Oct 2000 Fermi National Accelerator Laboratory Contributors: Ian Foster, Carl Kesselman, Steve Tuecke, Ann Chervenak

  2. The Globus Data Grid Two major components: 1. Data Transport and Access • Common protocol • Secure, efficient, flexible, extensible data movement • Family of tools supporting this protocol 2. Replica Management Architecture • Simple scheme for managing: • multiple copies of files • collections of files APIs, white papers: http://www.globus.org

  3. GridFTP Data Transport and Access

  4. GridFTP: Basic Approach • FTP is defined by several IETF RFCs • Start with most commonly used subset • Standard FTP: get/put etc., 3rd-party transfer • Implement standard but often unused features • GSS binding, extended directory listing, simple restart • Extend in various ways, while preserving interoperability with existing servers • Parameter set/negotiate, parallel transfers (multiple TCP streams), striped transfers (multiple hosts), partial file transfers, automatic & manual TCP buffer setting, progress monitoring, extended restart

  5. The GridFTP Family of Tools • Patches to existing FTP code • GSI-enabled versions of existing FTP client and server, for high-quality production code • Custom-developed libraries • Implement full GridFTP protocol, targeting custom use, high-performance • Custom-developed tools • Servers and clients with specialized functionality and performance

  6. Family of ToolsPatches to Existing Code • Patches to standard FTP clients and servers • gsi-ncftp: Widely used client • gsi-wuftpd: Widely used server • GSI modified HPSS pftpd • GSI modified Unitree ftpd • Provides high-quality, production ready, FTP clients and servers • Integration with common mass storage systems • Do not support the full GridFTP protocol

  7. Family of ToolsCustom Developed Libraries • Custom developed libraries • globus_ftp_control: Low level FTP driver • Client & server protocol and connection management • globus_ftp_client: Simple, reliable FTP client • globus_gass_copy: Simple URL-to-URL copy library, supporting (Grid-)ftp, http(s), file URLs • Implement full GridFTP protocol • Various levels of libraries, allowing implementation of custom clients and servers • Tuned for high performance on WAN

  8. Family of ToolsCustom Developed Programs • Simple production client • globus-url-copy: Simple URL-to-URL copy • Experimental FTP servers • Modified WUFTPD with parallel channels • Striped FTP server (ala.DPSS) • Firewall FTP proxy: Securely and efficiently allow transfers through firewalls

  9. Replica Management Architecture

  10. Replica Management • Maintain a mapping between logical names for files and collections and one or more physical locations • we define a replica to be a “managed copy of a file”. • The replica management system controls where and when copies are created, and provides information about where copies are located. However, the system does not make any statements about file consistency. In other words, it is possible for copies to get out of date with respect to one another, if a user chooses to modify a copy. • Based on the LDAP Protocol

  11. Reliable Replication Reliable Transport Disk Cache TapeLibrary Disk Array A Model Architecture for Data Grids Attribute Specification Replica Catalog Metadata Catalog Application Multiple Locations NWS Logical Collection and Logical File Name Selected Replica Replica Selection Performance Information and Predictions MDS Disk Cache Replica Location 1 Replica Location 2 Replica Location 3

  12. Replica Manager Components • Replica catalog definition • LDAP object classes for representing logical-to-physical mappings in an LDAP catalog • Low-level replica catalog API • globus_replica_catalog library • Manipulates replica catalog: add, delete, etc. • URL: http://www.globus.org • High-level reliable replication API • globus_replica_manager library • Combines calls to file transfer operations and calls to low-level API functions: create, destroy, etc.

  13. Replica Catalog Logical Collection C02 measurements 1998 Logical Collection C02 measurements 1999 Filename: Jan 1998 Filename: Feb 1998 … Logical File Parent Location jupiter.isi.edu Location sprite.llnl.gov Filename: Mar 1998 Filename: Jun 1998 Filename: Oct 1998 Protocol: GridFTP UrlConstructor: GridFTP://jupiter.isi.edu/ nfs/v6/climate Filename: Jan 1998 … Filename: Dec 1998 Protocol: ftp UrlConstructor: ftp://sprite.llnl.gov/ pub/pcmdi Logical File Jan 1998 Logical File Feb 1998 Size: 1468762 Replica Catalog Structure: A Climate Modeling Example

  14. Outstanding Issues • What write consistency should we support? • Methodology for handling updates • Access Control • Intermediate feedback required (callbacks) • Timing • Replicating the replica catalog • Replication of partial files • Alternate catalog views: files belong to more than one logical collection

  15. Status • Grid FTP and Replica Catalog API and tools in alpha test • Applications with climate data, intended for production use. • Replica Management API under design • Grid based access control strategy under design

  16. CustomServers globus-url-copy globus_replica_manager CustomClients globus_gass globus_gass_copy globus_gass_transfer globus_ftp_client globus_ftp_control globus_replica_catalog Legend globus_io OpenLDAP client Program In Alpha globus_common GSI (security) Library Released Globus Data-Intensive Services Architecture ReplicaPrograms

  17. The End

More Related