1 / 35

Data Management

Data Management. Kelly Clynes Caitlin Minteer. Agenda. Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable File Transfer Data Replication Replication Location Service Logical File Data Replication Service OGSA. Globus Toolkit.

mick
Download Presentation

Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Management Kelly Clynes Caitlin Minteer

  2. Agenda • Globus Toolkit • Basic Data Management Systems • Overview of Data Management • Data Movement • Grid FTP • Reliable File Transfer • Data Replication • Replication Location Service • Logical File • Data Replication Service • OGSA

  3. Globus Toolkit • Fundamental enabling technology for the "Grid" • Does not provide interactive client • Globus Toolkit provides: • Server implementation • Scriptable command line client • Set of development libraries

  4. Basic Data Management Systems • GridFTP - A uniform, secure, high-performance interface to file-based storage systems on the Grid • OGSA-DAI - An OGSA interface for accessing XML and relational data stores

  5. Overview of Data Management • Two Basic Categories of Data Management • Data Movement • Data Replication

  6. Data Movement • Globus Reliable File Transfer (RFT) Service • Globus GridFTP Tools

  7. GridFTP • What is GridFTP? • Global Grid Forum • GFD.020, RFC 959, RFC 2228, RFC 2389, etc. • based upon the Internet FTP protocol • Why use GridFTP? • Secure • Robust • Fast • Efficient

  8. GridFTP • What does GridFTP Provide? • multiple data channels for parallel transfers • partial file transfers • third-party (direct server-to-server) transfers • reusable data channels • command pipelining.

  9. GridFTP • Wish to make data available to others • Install server on a host that can access the data • Make sure that there is an appropriate Data Storage Interface (DSI) • To access data that other have made available, you need a GridFTP • To add access to files stored behind the GridFTP servers, you need a custom client

  10. GridFTP in GT 4.0 • Globus Code • Backwards Compatible • Stripping support • IPV6 support included • Modular

  11. Reliable File Transfer • Third-party multi-file transfers • Exponential back-off on failure • Optional use of parallel streams and TCP buffer size tuning • Recursive directory transfer

  12. Reliable File Transfer Service • Not a web service protocol • Requires that the client maintain an open socket connection to the server throughout the transfer • Service interfaced based on web services protocols that persist the transfer state in reliable storage

  13. Reliable File Transfer • Web Service Resource Framework • “job – scheduler” • You simply: • Provide a list of source and destination URLS • The service writes your job destination in DB and the moved the files on your behalf • Service methods are provided for querying the transfer status

  14. RFT in Action Grid Service Container Registry RFT Factory

  15. RFT in Action Grid Service Container Registry RFT Factory Client

  16. RFT in Action Grid Service Container RFT Factory Client

  17. RFT in Action Grid Service Container RFT Factory RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State Client

  18. RFT in Action Grid Service Container RFT Factory RFT Service Instance - Start the Instance - Deserialize XML to Java - Write Request via JDBC - Persist Service State Client

  19. RFT in Action • Service is OGSI compliant • Uses existing GridFTP protocols and tools to execute 3rd Party Transfer for the user • Provides extensive state transition notification RFT ServiceInstance GridFTPServer GridFTPServer

  20. Data Replication • Replica Location Service (RLS) • Provides the ability to keep track of one or more copies or replicas • Helpful for users ore applications that need to find where existing files are located on grid • Services register files in RLS when files are created

  21. Replica Location Service • Provides a framework for tracking the physical locations of data that has been replicated. • RLS maps logical names to physical names. • Replication of data can: • reduce access latency • improve data locality • increase robustness, scalability and performance for distributed applications. • An RLS typically does not operate in isolation, but functions as one component of a data grid architecture.

  22. Replica Location Service • RLS may consist of multiple servers at different sites • Able to increase over all scale of system and store more mappings then if just on single, centralized catalog

  23. Replica Location Service • Logical File – unique identifier for the contents of a file • Physical File – location of a copy of the file on a storage system • User can provide a logical file name to an RLS • User can query an RLS server to find name associated with a particular physical file location

  24. Logical File Associations between a logical file name and three replicas on different storage sites

  25. RLS Example Laser Interferometer Gravitational Wave Observatory • Detect the existence of gravitational waves • Produces millions of data files • Eight other sites need to access files • Ten physical locations • RLS servers at each site

  26. RLS Example • Requests the file from LIGO’s Data Management System • Lightweight Data Replicator (LDR) • LDR queries Replica Location Service for local copy • If no local copy is found, returns where it is in the Grid • Request to copy file to local storage • Registers new copy to local RLS

  27. Higher Level Data Services • Combines two existing data management components • RFT • RLS

  28. Data ReplicationService (DRS) • Provides a pull – based replication capability • Built on top of two GT data management components • Reliable File Transfer • Replica Location Service

  29. Data ReplicationService • Function • To ensure that a specified set of files exist on a storage site • Begins by querying RLS to discover where the desired files exist • After files are located, creates a transfer request that is executed by RFT • DRS then registers the new replicas with RLS

  30. Data ReplicationService • Implemented as a web service and complies with the Web Service Resource Framework • When request is received • Creates a WS – resource • Used to maintain state about each file being replicated • Including which operations on the file have failed

  31. OGSA • Open Grid Services Architecture (OGSA) • Product of the Grid community • Service oriented • Provides a pure Java data service framework for accessing and integrating data resources

  32. OGSA • Defines a set of core capabilities and behaviors: • uniform exposed service semantics (the Grid service); • defines standard devices for creating, naming, and discovering temporary Grid service instances; • provides location transparency and multiple protocol bindings for service instances; • supports integration with underlying native platform facilities.

  33. OGSA and GT4 • Allows multiple data resources to be accessed through a single service. • The listResources() operation • The data service resource identifiers can then be used by a client to obtain metadata and other information • Access to data service resource metadata is provided by an implementation of the WS-ResourceProperties specification. • getVersion() operation. • A WSRF version of the OGSA-DAI GridDataTransport portType supports asynchronous data delivery between data services.

  34. Summary • GridFTP • RLS • RFT • DRS • OGSA

  35. Resources • http://www.globus.org/alliance/publications/papers/ogsa.pdf • http://www.globus.org/toolkit/data/ • http://www.globus.org/toolkit/data/gridftp/ • http://www.globus.org/toolkit/data/rft/ • http://www.globus.org/toolkit/data/rls/

More Related