1 / 15

What is SAM-Grid?

What is SAM-Grid?. Job Handling Data Handling Monitoring and Information. Problems To Solve. How can a large, geographically distributed, dynamic , physics collaboration work together? How can this collaboration make use of available distributed computing resources?

garciarachel
Download Presentation

What is SAM-Grid?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What is SAM-Grid? Job Handling Data Handling Monitoring and Information

  2. Problems To Solve • How can a large, geographically distributed, dynamic, physics collaboration work together? • How can this collaboration make use of available distributed computing resources? • How can it handle the huge amount of data (PBs) generated by the experiment?

  3. Answers – The GRID & SAM-Grid • GRID • A network of middleware services that tie together distributed resources (Fabric – processors, storage). • SAM-Grid • Integrate the standard middleware to achieve a complete Job, Data, and Information management infrastructure thereby enabling fully distributed computing.

  4. SAM-Grid Architecture

  5. Job Management • Grid-level (global) job scheduling (selection of a cluster to run) distinguished from local scheduling (distribution of the job within the cluster) • We distinguish structured jobs from unstructured. • Structured jobs have their details known to Grid middleware • Unstructured jobs are mapped as a whole onto a cluster • Scheduler is interfaced with the data handling system. • For data-intensive jobs, sites are ranked by the amount of data cached at the site

  6. Job Handling User Interface User Interface Match Making Service Match Making Service Resource Selection external algorithm Submission Service Submission Service Information Collector Information Collector JOB Exec Site #1 Execution Site #n Grid/Fabric Interface Grid/Fabric Interface Grid/Fabric Interface Generic Service Generic Service Generic Service Computing Element Computing Element Computing Element Grid Sensors Grid Sensors Grid Sensors Grid Sensors Computing Element

  7. Data Handling - SAM • SAM is a distributed data movement and management service • SAM stations are resources pooled together to enable data management • Data replication is achieved by the use of disk caches during file routing. • SAM is a fully functional meta-data catalog. • A station can access a remote resource via the services offered by other connected stations Remote Station Cache2 MSS – Mass Storage System Control Flow Data Flow Local Station 1 Cache1 MSS2 Local Station 1 Cache2 Remote Station Cache1 MSS1 Local Station 2 Cache1

  8. Name Server Log server Global Resource Manager(s) services Data Handling Database Server(s) (Central Database) Shared Globally Local To Site Station 1 Servers Station 3 Servers Station 2 Servers Station n Servers Arrows indicate Control and data flow Mass Storage System(s) Shared Locally

  9. Monitoring and Information • This includes: • configuration framework • resource description for job brokering • infrastructure for monitoring • Main features • Sites (resources), services and jobs monitoring • Distributed knowledge about jobs etc. • Incremental knowledge building • Grid Monitoring Architecture for current state inquiries, Logging for recent history studies • All Web based

  10. Web Browser Web Browser Web Server Web Server 1 Web Server N Site N Information System Site 2 Information System Site 1 Information System IP IP IP IP Monitoring and Information

  11. Challenges with Grid/Fabric Interface • The Globus toolkit Grid/Fabric interfaces are not sufficiently… • …flexible: they expect a “standard” batch system configuration. • …scalable: a process per grid job is started up at the gateway machine. We want/need aggregation. • …comprehensive: they interface to the batch system only. How about data handling, local monitoring, databases, etc. • …robust: if the batch system forgets about the jobs, they cannot react.

  12. Flexibility • Addressing the peculiarity of the configuration of each batch system requires modification to the Globus toolkit job-manager • We address the problem by writing job-managers that use a level of abstraction on top of the batch systems. • Each batch system adapter can be locally configured to conform to the local batch system interface

  13. Scalability • The Globus gatekeeper starts up a process at the gateway node for every job entering the site • This limits the number of grid jobs at a site to around 300, for the typical commodity computer • We split single grid jobs into multiple batch processes in the SAM-Grid job-managers. Not only does this increase scalability, but it also increases the manageability of the job

  14. Comprehensiveness • The standard job-managers interface only to the local batch system • We notify other fabric services when a job enters a site • Data handling: for data pre-staging • Monitoring: to monitor a non-running job • Database: to aggregate queries

  15. Robustness • The standard job-managers cannot react to temporary failures of the local batch systems • In our experience, PBS, Condor and BQS have failed to report the status of a job • We write wrappers around the batch systems. These wrappers implement extra robustness. We call them “idealizers”

More Related