1 / 52

Condor-G: An Update

Condor-G: An Update. Outline. What is Condor-G Past Present Future. What Is Condor-G. Use Condor to run jobs on the Grid Uses Globus Toolkit GRAM (submit a remote job) GASS (transfer job’s files) Two components Globus Universe GlideIn. Globus Universe. Run a job on a Grid resource

ronia
Download Presentation

Condor-G: An Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Condor-G:An Update

  2. Outline • What is Condor-G • Past • Present • Future

  3. What Is Condor-G • Use Condor to run jobs on the Grid • Uses Globus Toolkit • GRAM (submit a remote job) • GASS (transfer job’s files) • Two components • Globus Universe • GlideIn

  4. Globus Universe • Run a job on a Grid resource • Features • Job management • Fault tolerance • Credential management • Disadvantages • No remote syscalls, checkpoint/migration, or dynamic resource selection

  5. How It Works Condor-G Grid Resource Schedd LSF

  6. 600 Globus jobs How It Works Condor-G Grid Resource Schedd LSF

  7. 600 Globus jobs How It Works Condor-G Grid Resource Schedd LSF GridManager

  8. 600 Globus jobs How It Works Condor-G Grid Resource JobManager Schedd LSF GridManager

  9. 600 Globus jobs How It Works Condor-G Grid Resource JobManager Schedd LSF GridManager User Job

  10. GlideIn • Create your own personal Condor pool from temporarily-acquired Grid resources • Brings the full power of Condor to the Grid • Run a Condor startd on a Grid resource • Startd reports back to your machine and runs Vanilla and Standard Universe jobs

  11. 600 Condor jobs How It Works Condor-G Grid Resource Schedd LSF Collector

  12. 600 Condor jobs glide-ins How It Works Condor-G Grid Resource Schedd LSF Collector

  13. 600 Condor jobs glide-ins How It Works Condor-G Grid Resource Schedd LSF GridManager Collector

  14. 600 Condor jobs glide-ins How It Works Condor-G Grid Resource JobManager Schedd LSF GridManager Collector

  15. 600 Condor jobs glide-ins How It Works Condor-G Grid Resource JobManager Schedd LSF GridManager Startd Collector

  16. 600 Condor jobs glide-ins How It Works Condor-G Grid Resource JobManager Schedd LSF GridManager Startd Collector

  17. 600 Condor jobs glide-ins How It Works Condor-G Grid Resource JobManager Schedd LSF GridManager Startd Collector User Job

  18. Globus Grid LSF PBS Condor Condor-G

  19. Globus Grid 600 Condor jobs LSF PBS Condor Condor-G

  20. Globus Grid Condor-G 600 Condor jobs LSF PBS Condor

  21. Globus Grid Condor-G 600 Condor jobs LSF PBS glide-ins Condor

  22. Globus Grid Condor-G 600 Condor jobs LSF PBS glide-ins Condor

  23. Globus Grid Condor-G 600 Condor jobs LSF PBS glide-ins Condor

  24. Globus Grid Condor-G 600 Condor jobs LSF PBS glide-ins Condor

  25. Past • GridManager daemon • Runs Grid jobs using GRAM protocol • Stages executable and standard I/O using GASS protocol • Globus GRAM 1.5 • We added fault-tolerance to the GRAM protocol • Changes included in Globus Toolkit 2.0 release

  26. Present • Updated Condor-G to Globus Toolkit 2.0 • Enhanced GridManager • GAHP

  27. Enhanced GridManager • Put problem jobs on hold • No more stuck jobs • Increase concurrency with GAHP • Almost ready

  28. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  29. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  30. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  31. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  32. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  33. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  34. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  35. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  36. Single-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  37. Multi-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  38. Multi-Threaded Execution Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  39. Globus Application Helper Protocol (GAHP) • Condor is non-threaded • Want to use multi-threaded libraries • Increased concurrency • Put libraries in external helper process • Simple interface over pipes/sockets

  40. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource Grid Resource

  41. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource GAHP Server GAHP Client Grid Resource

  42. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource GAHP Server GAHP Client Grid Resource

  43. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource GAHP Server GAHP Client Grid Resource

  44. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource GAHP Server GAHP Client Grid Resource

  45. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource GAHP Server GAHP Client Grid Resource

  46. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource GAHP Server GAHP Client Grid Resource

  47. Multi-Threaded Execution with GAHP Grid Resource GridManager Job 1 Grid Resource Job 2 Job 3 Job 4 Grid Resource GAHP Server GAHP Client Grid Resource

  48. Future • GRAM 1.6 • Condor-G on Windows • Condor-G Grid service

  49. Globus GRAM 1.6 • Working with Globus team to add additional features to GRAM protocol • Credential refresh • File staging • Scheduler-specific options

  50. Condor-G for Windows • Condor • Windows implementation available • GRAM and GASS APIs • No C implementation for Windows (yet) • Java implementation (Java CoG) • Condor-G • Windows version possible by writing GAHP server in Java

More Related