GRID RESOUCE MANAGEMENT

1. Pham Thanh Toan Trinh Quoc Huy Pham The Anh GRID RESOUCE MANAGEMENT

2. Outline • Introduction to Grid Resource Management • Hierarchical Model • Abstract Owner Model • Economy/Market Model

3. Resource types Physical resource: computer, disk, database, networks, scientific instruments. Logical resource: Executing applications, Complex workflows…

4. Definition The term resource management refers to the operations used to control how capabilities provided by Grid resources and services are made available to other entities, whether users, applications or services.

5. Definition At the heart of the Grid is the ability to discover, allocate, and negotiate the use of network accessible capabilities, be they computational services offered by a computer, application services offered by a piece of software, bandwidth delivered on a network, or storage space provided by a storage system.

6. Requirements Establish a mutual agreement between resource provider and resource consumer. Encompass a wide range of different scenarios: task submission, workload management, on-demand access, co-scheduling, resource brokering,… Encompass not only a commitment to perform a task but also commitments to level of performance, or quality of services.

7. Issues Coordinate resource usage, the diversity of resource types and the variety of different management modes. The differing policy requirements of resource owners make difficulties in creating a consistent cross-organized policy domain that delivers the necessary capability to the end user while respecting the policy requirements of the resource owner. Grid Applications require the concurrently allocation of multiple resources.

8. Grid Resource Management Models Hierarchical model. Abstract Owner Computational Market/Economy Model.

9. Hierarchical Model

10. Hierarchical Resource Management The outcome of Grid Forum’s second meeting. GRAM, Legion, Ninf. The major components of this model is divided into passive and active components

11. Passive components • Resource: • things that can be used for a period of time, and may or may not be renewable • They have owners who may charge others for using resources. • They can be shared or exclusive.

12. Passive components • Task: • Are consumers of resources • Include both traditional tasks and non-computational tasks (such as file and communication). • Jobs: • Are hierarchical entities: can be composed of sub-jobs or tasks, and sub-jobs maybe themselves contain sub-jobs. The leave of this structure are tasks. • The simplest form of a job is one containing a single task

13. Passive components • Schedules: • Are mappings of tasks to resource overtime.

14. Active components • Schedulers: • Compute one or more schedules for input jobs • The unit of scheduling is the job. • The schedulers outside control domains cannot commit resources; these are known as metaschedulers or super schedulers. • Information Services: • Act as databases for describing items of interest to the resource management systems: resource, jobs, schedulers, agents, … • Could be a LDAP, a database,…

15. Active components • Domain Control Agents (Local resource manager): • Commit resource for use • The set of resources controlled by an agent is a control domain. • Are distinct from Schedulers, but control domains may contain internal Schedulers. • Can provide state information, either through publishing in an Information Service or via direct querying. • The actions outside are called requests. Action inside maybe called command.

16. Active components Deployment agents: implement schedules by negotiating with domain control agents to obtains resources and start tasks running. Users: submit job to the resource management system for execution. Admission Control Agents: determine whether the system can accommodate additional jobs, and reject or postpone job when the system are saturated.

17. Active components • Monitors: • Track the progress of jobs. • Obtain job status from the task comprising the job and from the Domain Control Agents where those tasks are running. • Perform outcalls to Job Control Agents and Schedulers to effect remapping, terminating extending lifetime, …

18. Active components • Job Control Agents: • Are responsible for shepherding a job through the system, and can act both as a proxy for the user and as a persistent point for a job. • Are responsible for coordinating between different components within the resource management system.

19. Interaction of Components A user submit a job to a Job Control Agent. User Job Control Agent

20. Interaction of Components Job Control Agent calls an Admission Agent to examines the resource demands of the job (perhaps consulting with a grid Information System). User Job Control Agent Admission Control Agent

21. Interaction of Components The Admission Agent pass the job to the Scheduler, which performs resource discovery (using the grid Information System). User Job Control Agent Admission Control Agent Scheduler

22. Interaction of Components Scheduler consults with Domain Control Agent to determine the current state and availability of resources. User Job Control Agent Admission Control Agent Domain Control Agent Scheduler

23. Interaction of Components The Scheduler computes a set of mappings and passes these mappings to a Deployment Agent. User Job Control Agent Admission Control Agent Deployment Domain Control Agent Scheduler

24. Interaction of Components The Deployment Agent negotiates with the Domain Control Agents for the resource indicated in schedules and obtains resource reservations for the resources User Job Control Agent Admission Control Agent Deployment Domain Control Agent Scheduler

25. Interaction of Components These reservations are passed to Job Control Agent. At the proper time, the Job Control Agent works with different Deployment Agent, and the Deployment Agent works with the appropriate Domain Control Agents to start the tasks running. User Job Control Agent Admission Control Agent Deployment Domain Control Agent Scheduler

26. Interaction of Components A Monitor tracks progress of the job and may later decide to reschedule if performance is lower than expected. User Job Control Agent Admission Control Agent Deployment Domain Control Agent Scheduler Monitor

27. Abstract Owner Model

28. Abstract Owner Model Introduction General Structure of AO Grid Resources Negotiating with an AO Job Shops

29. Abstract Owner When one makes a long distance phone call, who "owns" the resource being used?

30. Abstract Owner Each grid resource be represented by one or more “abstract owners” (AO) For complex resources, an AO will certainly be a broker for the actual owners or other brokers Pre-existing contract or protocol to acquire resource

31. General Structure of AO - Resource • Resource:any combination of hardware and software • A processor • Another Grid system • Resource will be regarded as object • Methods: initiating and controlling tasks • Attributes: customize the resource

32. General Structure of AO - Task • Task: problem/part of problem that need resource to do. • A message to be sent • A process to be executed • A complete multi-process program

33. General Structure of AO–External view Order Window Pickup Window Abstract Owner External View of AO model • Order Window: negotiate to acquire resource • how soon the resource may become available? • how much it might cost? • Pickup Window: deliver resource to client

34. General Structure of AO – External View Client send request and negotiate to AO successful negotiation? N Y Client place order to AO through Order Window CANCEL N Client wait for resource? Y AO return resource to Client through Pickup Window

35. General Structure of AO – Internal View Order Window Pickup Window Resource Manager Physical Resource AO is Resource Owner • InternalStructure • An owner of a physical resource

36. General Structure of AO – Internal View Order Window Pickup Window Manager Sales Rep. Delivery Rep. AO3 AO2 AO is Broker • Internal Structure • Higher level broker AO1

37. Grid Resources • Three type of resources: • Instruments • Logically exists at some location. • Creates, consumes, or transforms data or information. • Channels • Exists to facilitate the explicit transfer of data or information between two or more instruments • Connects to an Instrument through a Port • Complexes • A collection of (connected) Channel and Instrument resources.

38. Grid Resources • Instruments are divided into subclasses • Compute Instrument: a processor or set of processors along with associated memory, temp files, software, etc. • Archival Instrument: persistent storage of information. • Personal Instrument: interface directly to a human being, ranging from a simple terminal to a more complex

39. Negotiating with a AO • Client need to negotiate with AO before using resource • Negotiating process: • Client sends negotiation • AO replied with appropriate resource information • Client accept and place order, or reject the negotiation.

40. Negotiating with AO • Sample object attributes: • Client first effectively creates a "sample" resource object of the appropriate structure • Assigns each attribute either • (1) a constant value, • (2) a "don’t care" value, • (3) a variable name • Some variables can be “negotiable” • Variable constraint list: constraints for variables

41. Negotiating with AO • Negotiation Style: • Immediate: AO is to schedule the resource immediately • Pending: AO is to return a specified number of sets of proposed values for the negotiation variables • Confirmation: AO is to finish scheduling based on an earlier-returned set of negotiation variable values • Cancel: AO is to cancel earlier Pending negotiation • Pickup Approach: Protocol to be used at Pickup Window

42. Negotiating with AO Authorization: a capability or key which allows the AO to determine the authority of the client to access resources Bid: maximum price that the client is willing to pay for the Negotiation ID: like cookie

43. Job Shop Job Result Job Shop (Estimator & Executor) Resource Des. Resource AO for Grid Job Scheduling • Instead of only deal with Resource acquiring, “Job Shop” handle job scheduling. • Job Shop • Estimator • Executor

44. Job Shop – The Estimator Deals with the customer, acting as a liaison with the grid AO (through its order window) if necessary to help determine how soon the job might be done and how much it might cost. Requests the resources needed from the grid AO (through the order window). Records (in the Work To Do List) what needs to be done when the resources are ready.

45. Job shop – The Executor Takes ready resources from the AO delivery window Dequeues the associated work (tasks) from the Work To Do List Builds any necessary environment for those tasks based on the resource Initiates the tasks on the resource and collects the answer. Frees the resource upon completion, if necessary. Notifies and returns the answers to the user

46. Job Shop Work to do list Estimator Executor Job Shop

47. AO Summary AO model is still a proposal There are many remaining gaps in the model Even a potentially useful and well-defined (successfully prototyped) AO protocol will not be viable unless it can coexist with other contemporary approaches.

48. Economy/Market Model

49. Overview • Grid environment • Resources are geography distributed and belong to different organization • Has different resource management, policies, charges for different users • Need a market-driven resource management system to satisfy both resource owner and user • User : minimize expenses • Resource owner: maximize return-on-investment

50. Overview • Necessitating a grid resource management system: • Provides appropriate tools and services to allow both resource users and owners to express their requirements. • Users: specify their “QoS requirements” such as minimize the computational cost and yet meet the deadline for the results. • Resource owners: specify their charges, that can vary from time to time and users to users—and terms of use. • Likely that more and more systems are going to emerge based on this concept.