1 / 39

STORK & NeST: Making Data Placement a First Class Citizen in the Grid

STORK & NeST: Making Data Placement a First Class Citizen in the Grid. Need to move data around. While doing this. Locate the data Access heterogeneous resources Recover form all kinds of failures Allocate and de-allocate storage Move the data Clean-up everything.

sema
Download Presentation

STORK & NeST: Making Data Placement a First Class Citizen in the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. STORK & NeST: Making Data Placement a First Class Citizen in the Grid

  2. Need to move data around.. Reliable and Efficient Grid Data Placement

  3. While doing this.. • Locate the data • Access heterogeneous resources • Recover form all kinds of failures • Allocate and de-allocate storage • Move the data • Clean-up everything All of these need to be done reliably and efficiently! Reliable and Efficient Grid Data Placement

  4. Stork • A scheduler for data placement activities in the Grid • What Condor is for computational jobs, Stork is for data placement • Stork’s fundamental concept: “Make data placement a first class citizen in the Grid.” Reliable and Efficient Grid Data Placement

  5. Outline • Introduction • The Concept • Stork Features • Big Picture • Conclusions Reliable and Efficient Grid Data Placement

  6. Stage-in • Execute the Job • Stage-out Individual Jobs The Concept Reliable and Efficient Grid Data Placement

  7. Allocate space for input & output data Stage-in • Stage-in • Execute the Job • Stage-out Execute the job Release input space Stage-out Release output space Individual Jobs The Concept Reliable and Efficient Grid Data Placement

  8. Allocate space for input & output data Stage-in • Stage-in • Execute the Job • Stage-out Execute the job Release input space Stage-out Data Placement Jobs Release output space Computational Jobs The Concept Reliable and Efficient Grid Data Placement

  9. D F A B E The Concept Condor Job Queue Data A A.submit Data B B.submit Job C C.submit ….. Parent A child B Parent B child C Parent C child D, E ….. DAG specification C DAGMan Stork Job Queue C E Reliable and Efficient Grid Data Placement

  10. Why Stork? • Stork understands the characteristics and semantics of data placement jobs. • Can make smart scheduling decisions, for reliable and efficient data placement. Reliable and Efficient Grid Data Placement

  11. Understanding Job Characteristics & Semantics • Job_type = transfer, reserve, release? • Source and destination hosts, files, protocols to use? • Determine concurrency level • Can select alternate protocols • Can select alternate routes • Can tune network parameters (tcp buffer size, I/O block size, # of parallel streams) • … Reliable and Efficient Grid Data Placement

  12. Support for Heterogeneity Protocol translation using Stork memory buffer. Reliable and Efficient Grid Data Placement

  13. Support for Heterogeneity Protocol translation using Stork Disk Cache. Reliable and Efficient Grid Data Placement

  14. Flexible Job Representation [ Type = “Transfer”; Src_Url = “srb://ghidorac.sdsc.edu/kosart.condor/x.dat”; Dest_Url = “nest://turkey.cs.wisc.edu/kosart/x.dat”; …… …… ] Reliable and Efficient Grid Data Placement

  15. Failure Recovery and Efficient Resource Utilization • Fault tolerance • Just submit a bunch of data placement jobs, and then go away.. • Control number of concurrent transfers from/to any storage system • Prevents overloading • Space allocation and De-allocations • Make sure space is available Reliable and Efficient Grid Data Placement

  16. Outline • Introduction • The Concept • Stork Features • Big Picture • Conclusions Reliable and Efficient Grid Data Placement

  17. PLANNER USER JOB DESCRIPTIONS Abstract DAG

  18. RLS PLANNER WORKFLOW MANAGER USER JOB DESCRIPTIONS Abstract DAG Concrete DAG

  19. RLS / MCAT PLANNER WORKFLOW MANAGER GRID STORAGE SYSTEMS GRID COMPUTE NODES USER JOB DESCRIPTIONS Abstract DAG Concrete DAG DATA PLACEMENT SCHEDULER COMPUTATION SCHEDULER

  20. RLS / MCAT PLANNER WORKFLOW MANAGER GRID COMPUTE NODES GRID STORAGE SYSTEMS USER JOB DESCRIPTIONS Abstract DAG Concrete DAG POLICY ENFORCER DATA PLACEMENT SCHEDULER COMPUTATION SCHEDULER C. JOB LOG FILES D. JOB LOG FILES

  21. RLS / MCAT PLANNER WORKFLOW MANAGER GRID COMPUTE NODES GRID STORAGE SYSTEMS USER JOB DESCRIPTIONS Abstract DAG Concrete DAG POLICY ENFORCER DATA PLACEMENT SCHEDULER COMPUTATION SCHEDULER C. JOB LOG FILES D. JOB LOG FILES DATA MINER NETWORK MONITORING TOOLS FEEDBACK MECHANISM

  22. RLS / MCAT PLANNER GRID COMPUTE NODES GRID STORAGE SYSTEMS USER JOB DESCRIPTIONS Abstract DAG Concrete DAG DAGMAN MATCHMAKER STORK CONDOR/ CONDOR-G C. JOB LOG FILES D. JOB LOG FILES DATA MINER NETWORK MONITORING TOOLS FEEDBACK MECHANISM

  23. Conclusions • Regard data placement as individual jobs. • Treat computational and data placement jobs differently. • Introduce a specialized scheduler for data placement. • Provide end-to-end automation, fault tolerance, run-time adaptation, multilevel policy support, reliable and efficient transfers. Reliable and Efficient Grid Data Placement

  24. Future work • Enhanced interaction between Stork and higher level planners • better coordination of CPU and I/O • Interaction between multiple Stork servers and job delegation from one to another • Enhanced authentication mechanisms • More run-time adaptation Reliable and Efficient Grid Data Placement

  25. Related Publications • Tevfik Kosar and Miron Livny. “Stork: Making Data Placement a First Class Citizen in the Grid”. In Proceedings of 24th IEEE Int. Conference on Distributed Computing Systems (ICDCS 2004), Tokyo, Japan, March 2004. • George Kola, Tevfik Kosar and Miron Livny. “A Fully Automated Fault-tolerant System for Distributed Video Processing and Off-site Replication. To appear in Proceedings of 14th ACM Int. Workshop on etwork and Operating Systems Support for Digital Audio and Video (Nossdav 2004), Kinsale, Ireland, June 2004. • Tevfik Kosar, George Kola and Miron Livny. “A Framework for Self-optimizing, Fault-tolerant, High Performance Bulk Data Transfers in a Heterogeneous Grid Environment”. In Proceedings of 2nd Int. Symposium on Parallel and Distributed Computing (ISPDC 2003), Ljubljana, Slovenia, October 2003. • George Kola, Tevfik Kosar and Miron Livny. “Run-time Adaptation of Grid Data Placement Jobs”. In Proceedings of Int. Workshop on Adaptive Grid Middleware (AGridM 2003), New Orleans, LA, September 2003. Reliable and Efficient Grid Data Placement

  26. You don’t have to FedEx your data anymore.. Stork delivers it for you! • For more information: • Email: condor-admin@cs.wisc.edu • http://www.cs.wisc.edu/condor/stork Reliable and Efficient Grid Data Placement

  27. NeST • NeST (Network Storage Technology) • A lightweight, portable storage manager for data placement activities on the Grid • Allocation: NeST negotiates “mini storage contracts” between users and server. • Multi-protocol: Supports Chirp, GridFTP, NFS, HTTP • Chirp is NeST’s internal protocol. • Secure: GSI authentication • Lightweight: Configuration and installation can be performed in minutes, and does not require root. Reliable and Efficient Grid Data Placement

  28. Why storage allocations ? • Users need both temporary storage, and long-term guaranteed storage. • Administrators need a storage solution with configurable limits and policy. • Administrators will benefit from NeST’s automatic reclamations of expired storage allocations.

  29. Storage allocations in NeST • Lot – abstraction for storage allocation with an associated handle • Handle is used for all subsequent operations on this lot • Client requests lot of a specified size and duration. Server accepts or rejects client request.

  30. Lot types • User / Group • User: single user (user controls ACL) • Group: shared use (all members control ACL) • Best effort / Guaranteed • Best effort: server may purge data if necessary. Good fit for derived data. • Guaranteed: server honors request duration. • Hierarchical: Lots with lots (“sublots”)

  31. Lot operations • Create, Delete, Update • MoveFile • Moves files between lots • AddUser, RemoveUser • Lot level authorization • List of users allowed to request sub-lots • Attach / Detach • Performs NeST lot to path binding

  32. GridFTP Server globus-url-copy Functionality: GT4 GridFTP Sample Application (GSI-FTP) Disk Module Disk Storage

  33. NeST Server GridFTP Server NeST Client globus-url-copy Functionality: GridFTP +NeST (Lot operations, etc.) (File transfers) (GSI-FTP) (chirp) NeST Module (File transfer) (chirp) Chirp Handler Disk Storage

  34. NeST Server GridFTP Server Stork NeST with Stork GT4 NeST (File transfers) (GSI-FTP) (Lot operations, etc.) (chirp) NeST Module (File transfer) (chirp) Chirp Handler Disk Storage

  35. Xfer Out Allocate Xfer In Release Job Job Job NeST NeST Sample Work DAG Condor-G Stork Stork

  36. Connection Manager • Used to control connections to NeST • Allows connection reservations: • Reserve # of simultaneous connections • Reserve total bytes of transfer • Reservations have durations & expire • Reservations are persistent

  37. Release Status • http://www.cs.wisc.edu/condor/nest • v0.9.7 expected soon • v0.9.7 Pre 2 released • Just bug fixes; features frozen • v1.0 expected later this year • Currently supports Linux, will support other O/S’s in the future.

  38. Roadmap • Performance tests with Stork • Continue hardening code base • Expand supported platforms • Solaris & other UNIX-en • Bundle with Condor • Connection Manager

  39. Questions ? • More information available at http://www.cs.wisc.edu/condor/nest

More Related