1 / 9

Intelligent Data Placement Scheduler for Heterogeneous Grid Environment

This system efficiently manages, schedules, and monitors Data Placement activities in a Grid environment, making intelligent decisions based on job requirements. It integrates with various middleware and storage systems, ensuring reliability and recoverability from failures. The user-friendly Data Placement Scheduler offers a single interface for streamlined operations. Supported by multiple protocols, it simplifies data transfer and enhances job scheduling productivity.

leonardevans
Download Presentation

Intelligent Data Placement Scheduler for Heterogeneous Grid Environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Managing and Scheduling Data Placement (DaP) Requests in GRID Tevfik Kosar & Miron Livny

  2. Motivation An intelligent data scheduling system is needed, which: • integrates/interacts with all different middleware and end-storage systems in the Grid • makes intelligent scheduling decisions depending on job requirements • is recoverable from storage system and network failures • easy to use, single interface

  3. Data Placement Scheduler (DaPS) • Intelligently & reliably manages, schedules and monitors Data Placement (DaP) activities in a heterogeneous Grid environment • What Condor means for computational jobs, DaPS means the same for DaP jobs • Just submit a bunch of DaP jobs and then relax..

  4. DaPS Client DaPS Client DaPS Client SRB Server SRM Server Sched. Exec. Accept DAPS Server Local Disk Req. Req. Req. Req. Req. Req. GridFTP Server GridFTP Server NeST Server DaPS Architecture Remote Local Get Queue Buffer Put Thirdparty transfer

  5. DaP Job ClassAds [ Type = Reserve; Server = nest://turkey.cs.wisc.edu; Size = 100MB; reservation_no = 1; ] [ Type = Transfer; Src_url = srb://ghidorac.sdsc.edu/kosart.condor/x.dat; Dst_url = nest://turkey.cs.wisc.edu/kosart/x.dat; ] [ Type = Release; Server = nest://turkey.cs.wisc.edu; reservation_no = 1; ]

  6. Supported Protocols • Currently supported: • FTP • GridFTP • NeST (chirp) • SRB (Storage Resource Broker) • SRM (Storage Resource Manager) • Very soon: • GDMP (Grid Data Management Pilot)

  7. Reserve In & out Transfer in Job B Release in Transfer out Release out Case Study: DAGMan Add DaP jobs to the DAG structure PRE Job B POST

  8. New DAGMan Architecture .dag File DAGMan DAGMan A DaPS Job Queue Condor Job Queue X X A C B Y D

  9. Future Work • More intelligent scheduling • Data Level Management instead of File Level Management • Enhanced interaction with DAGMan • Possible integration with Kangaroo to keep the network pipeline full

More Related