1 / 42

GT4 GRAM: A Functionality and Performance Study

GT4 GRAM: A Functionality and Performance Study. Stuart Martin, Martin Feller Computational Institute, University of Chicago & Argonne National Lab. TeraGrid 2007 Madison, WI. Contributors / Collaborators. UC/ANL Ian Foster Peter Lane Jarek Gawor Ravi Madurri Rachana Ananthakrishnan.

lang
Download Presentation

GT4 GRAM: A Functionality and Performance Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GT4 GRAM: A Functionality and Performance Study Stuart Martin, Martin Feller Computational Institute, University of Chicago & Argonne National Lab TeraGrid 2007 Madison, WI

  2. Contributors / Collaborators • UC/ANL • Ian Foster • Peter Lane • Jarek Gawor • Ravi Madurri • Rachana Ananthakrishnan

  3. GRAM - Basic Job Submission and Control Service • A uniform service interface for remote job submission and control • Includes file staging and I/O management • Includes reliability features • Supports basic Grid security mechanisms • Asynchronous monitoring • Interfaces with local resource managers, simplifies the job of metaschedulers/brokers • GRAM is not a scheduler. • No scheduling • No metascheduling/brokering

  4. Comparison • Functionality • Security • File Staging • General • Performance • Concurrent jobs • Sequential jobs

  5. Security Functional Comparisons

  6. Privilege Limiting Model • GRAM must be able to start jobs submitted by remote users under different user ids. It must execute some code as “root” • GRAM2: Entire gatekeeper runs as root • GRAM4: Service with sudo privs • non-root container account requires sudo to invoke operations as other users

  7. Authentication • A client can authenticate with GRAM with a variety of protocols • GRAM2: TLS (only) • GRAM4: TLS, Message Level Security • Message-level WS-Security • Channel-level WS-SecureConversation • Choice for which to support in each deployment

  8. Credential Delegation • Needed by GRAM or the user’s applications to do file staging or other grid operations • GRAM2: Yes, Required • Clients must delegate from client to service on every request • GRAM4: Yes, Optional • Clients can choose and delegate when necessary

  9. Credential Refresh • Credentials have a lifetime and may expire before a job has completed execution • GRAM2: Yes • GRAM4: Yes • A client can query for information about the WS Resource of the delegated credential • Remaining lifetime

  10. Share credential delegationamong jobs • When repeatedly interacting with the same GRAM service, a client may want to delegate once and share the delegation among multiple jobs • GRAM2: No • GRAM4:Yes • Refreshing a credential in the delegation service that was shared among multiple job submission will results in a refresh for each job

  11. Authorization Callouts • Following authentication, GRAM checks to see if the request should be authorized.For example, a gridmap file acting as an access control list • GRAM2: Yes - single PDP callout • GRAM4:Yes - Multiple PDP callout chain • Allows for richer policies • Parse VOMS attributes • Use attributes in policy evaluations • Site level black lists

  12. File Management Functional Comparisons

  13. File Staging • Job staging before and after the user’s job is executed • GRAM2: Yes • GRAM4: Yes

  14. File staging retry policy • If a file staging operation fails, it may be non-fatal and retry may be desired • GRAM2: None • GRAM4:RFT Supported • Server defaults for all transfers can be configured • Defaults can be overridden for a specific transfer

  15. Incremental output staging “streaming” • It can be useful to obtain access to data produced by a program as it executes. • GRAM2: stdout/stderr only • GRAM4:stdout/stderr and any file • A client can stream files via the service-side GridFTP server. This is what globusrun-ws does for stdout and stderr streaming.

  16. Standard input access • The contents of a file can be passed to the job’s standard input • GRAM2: Yes • GRAM4: Yes

  17. Throttle staging work • A GRAM submission that specifies file staging imposes load on the service node executing the GRAM service. • GRAM2: No • GRAM4:Yes • GRAM is configured for a maximum number of “worker” threads and thus a maximum number of concurrent staging operations.

  18. Load balance staging work • Allow staging work to be load balanced among a set of service hosts • GRAM2: No • GRAM4: Yes • Staging work can be distributed over several “service nodes”. For example, a separate GridFTP server can be configured for each LRM type or file system paths.

  19. General Functional Comparisons

  20. Access protocol • Protocol used to interact with the service • GRAM2: proprietary HTTP • GRAM4:Web Service SOAP • Standards based • WSDL • Client tooling

  21. Job Description Language • The mechanism for specifying job directives. • GRAM2: RSL • Custom string-based language • GRAM4:JDD • Job description document (JDD) XML-based version • Initial prototype of OGF’s JSDL specification

  22. Extensible Job Description Language • A mechanism for passing “extensions” through GRAM to underlying local resource managers • GRAM2: Yes • GRAM4: Yes

  23. Local Resource Manager Interface • The GRAM interface to the LRM to submit, monitor, and cancel jobs. • GRAM2: Perl scripts • GRAM4:Perl scripts + SEG • Scheduler Event Generator (SEG) provides efficient monitoring between the GRAM service and the LRM for all jobs for all users

  24. Local Resource Managers • Supports a range of LRMs - PBS, LSF, Condor, Fork, … • GRAM2: Yes • GRAM4: Yes

  25. Fault Tolerance • GRAM can recover from a container or host crash. Upon restart, GRAM will resume processing of the users job submission • GRAM2: Yes - Client initiated • Processing resumes for a single job after the client has restarted the job manager service process • GRAM4:Yes - Service initiated • Processing resumes for all jobs once the service container has been restarted

  26. State Access: Push (subscription) • Allow clients to request notifications for state changes • GRAM2: Yes - callbacks • GRAM4:Yes - WS Notifications • Clients can subscribe for notifications to the “job status” resource property

  27. State Access: Pull • Allow clients to get the state for a previously submitted job • GRAM2: Yes • The service defines a proprietary operation to get the job state. • GRAM4:Yes • The service defines a WSRF resource property that contains the value of the job state. A client can then use the standard WSRF getResourceProperty operation.

  28. Audit Logging • Allow an audit records to be inserted into an audit DB when a job completes • GRAM2: Yes • GRAM4:Yes • An enhancement was contributed by Gerson Galang (APAC) to insert the record at the beginning of the job and to update the audit record after submission and again at job end.

  29. At Most Once Job Submission • A simple request-reply job submission protocol has the problem that if the reply message is lost, a client cannot know whether a job has been started. Measures need to be taken to ensure that the same job is not submitted twice. • GRAM2: Yes - 2-phase commit • Requires an extra round trip, plus a delay on the service to begin processing • GRAM4:Yes - UUID on create • The client supplies a client-created unique ID (UUID) and the GRAM4 service guarantees not to start a job with a duplicate ID

  30. Job Cancellation • Allow a job to be cancelled • GRAM2: Yes • Proprietary operation • GRAM4:Yes • WSRF standard “Destroy” operation

  31. Job Lifetime Management • Allow a client to control when a job’s state is cleaned up • GRAM2: Yes • Implements a set of job directives and operations • GRAM4:Yes • Standard WS-ResourceLifetime operations

  32. Maximum Active Jobs • The Maximum number of jobs that the service can manage • GRAM2: ~250 • Due to each job Job Manager process querying the LRM separately • GRAM4: 32,000 • Limited by the number of directories that can be created in a directory

  33. Parallel Job Support • Support for MPI jobs “jobtype = MPI” • GRAM2: Yes • GRAM4: Yes

  34. MPICH-G Support • Support for multi-site MPI • GRAM2: Yes • Client-side DUROC and service-side DUCT service • GRAM4:Yes • Multi-job and rendezvous Web Services • MPIg support coming soon

  35. Basic Execution Service (BES) Interface • Support for OGSA BES for job submission • GRAM2: No • GRAM4: Prototyped • Working on plans to initially support JSDL with the current GRAM4 port type, then add support for BES too

  36. Performance Comparisons

  37. Concurrent Jobs(as in paper) Average seconds per 1000 jobs Condor-g to GRAM to Condor LRM

  38. Concurrent Jobs(as will be in GT 4.0.5) Average seconds per 1000 jobs Condor-g to GRAM to Condor LRM

  39. Improving performance forstaging jobs • Adding local method call mechanism for general use in Java WS Core (4.0.5) • GRAM is doing this with RFT • Any service which calls another in-process service could make similar modifications for local calls and likely benefit from improved performance • Adding caching of the GridFTP server connections in RFT (4.0.6)

  40. Sequential Jobs Average seconds per job (Fork)

  41. Sequential Jobs Average seconds per job (Fork)

  42. For More Information • Stuart Martin - smartin@mcs.anl.gov • Martin Feller - feller@mcs.anl.gov

More Related