1 / 30

Architectural Models for Resource Management in the Grid

$grid Architectural Models for Resource Management in the Grid Rajkumar Buyya Monash University, Australia Steve Chapin, Syracuse University, USA David DiNucci Elepar.com, USA Outline Changes in Computing Landscape Resource Management Issues Architectural Models

mike_john
Download Presentation

Architectural Models for Resource Management in the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. $grid Architectural Models for Resource Management in the Grid Rajkumar Buyya Monash University, Australia Steve Chapin, Syracuse University, USA David DiNucci Elepar.com, USA

  2. Outline • Changes in Computing Landscape • Resource Management Issues • Architectural Models • Hierarchical Resource Management • Abstract Owner • Market Model • Economy Grid • Nimrod/G Resource Broker • Conclusions

  3. Computing Power (HPC) Drivers • Solving grand challenge applications using computer modeling, simulation and analysis Aerospace Internet & Ecommerce Life Sciences Digital Biology CAD/CAM Military Applications Military Applications Military Applications

  4. 2100 2100 2100 2100 2100 2100 2100 2100 2100 Computing Platforms ? PERFORMANCE Breaking Administrative Barriers • Individual • Group • Department • Campus • State • National • Globe • Inter Planet • Universe Desktop SMPs or SuperComputers Global Cluster/Grid Inter Planet Cluster/Grid ?? Local Cluster Enterprise Cluster/Grid

  5. Towards Grid Computing Unification of geographically distributed resources

  6. What is Grid ? • An infrastructure that couples: • Computers – PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc; • Software – e.g., ASPs renting expensive special purpose applications on demand; • Catalogued data and databases – e.g. transparent access to human genome database; • Special devices – e.g., radio telescope – SETI@Home searching for life in galaxy, Austrophysics@Swinburne for pulsars) • People/collaborators. • & offers dependable, consistent, & pervasive access to resources.

  7. A Example Grid Infrastructure

  8. Sources of Complexity in Grid Resource Management • No single administrative control. • No single ownership policy: • Each resource owner has their own policies or scheduling mechanisms; • Users must honour them (particularly external Grid users). • Heterogeneity of resources. • Dynamic availability – may appear and disappear…

  9. Sources of Complexity in Grid Resource Management • Unreliable resource – disappear from view! • No uniform cost model - varies from one user’s resource to another and from time of day. • No single access mechanism – Web, custom interfaces, command line…

  10. Grid Resource Management Issues • Authentication (once). • Specify (code, resources, etc.). • Discover resources. • Negotiate authorization, acceptable use, Cost, etc. • Acquire resources. • Schedule Jobs. • Initiate computation. • Steer computation. • Access remote data-sets. • Collaborate with results. • Account for usage. • Discover resources. • Negotiate authorisation, • acceptable use, Cost, etc. • Acquire resources. • Schedule jobs. • Initiate computation. • Steer computation. Domain 1 Domain 2 Ack: Globus..

  11. Architectural Models

  12. Global Scheduler Global Scheduler Global Scheduler Global Scheduler Global Scheduler Hierarchical Resource Management Access/Admission Control Agent User Grid Information Service Persistent Job Control Agent Connection Cloud Monitor Local Scheduler Deployment Agent Domain Resource Manager or Control Agent Control Domain - Task Resource

  13. Who owns the GRID? Talk to people Powerappliances Use GRID resources I want to: My interface is: I arrange serviceand payments with a: (may be many choices) Abstract Owner (AO) Phone co. Electric co. HPC Networks Instruments People Antennae Cable/fiber Switches Generators Power lines Transformers But resources I “get”may belong to others:

  14. AO is owner or broker User • User negotiates with AO through “order window” • That AO may own some resources, and/or it may broker with other AOs for those resources • After negotiation, resources are delivered through “pickup window” Requests Resources Order Window Pickup Window AO Order Order Pickup Pickup Manager ResourceManager Sales Delivery AO3 Physical Resource AO2 AO1

  15. AO Resources • Resources are objects • Classes are • Instrument • Data source, sink, transform • e.g. programs, people, files, data collection devices • Channel • Moves data among instruments • Complexes of above • Attributes define sizes, times, connections, etc. Instrument (File) Instrument (Program) Channels Instrument (File) Instrument (Program) Instrument (Telescope) Instrument (Person)

  16. Negotiating with an AO Make dummy resource (with attributes set to constants, variables, or “don’t care”) + bid + delivery plan+ variable constraints Pick one, Try again, Or give up Assign tasksto resource,use, relinquish User Perhapslater... Delivery Window Order Window Resource candidates (values for variables/attributes+ asking price for each) AO Resource

  17. $grid Many Testbeds ? & who pays ? GUSTO EcoGrid Legion Testbed NASA IPG

  18. Testbeds so far -- observations • Who contributed to resources & why ? • Volunteers: for fun, challenge, fame, public good like SETI@Home & distributed.net projects. • Collaborators: sharing resources while developing new technologies of common interest – Globus, Legion, Ecogrid. • How long ? • Short duration: GUSTO decommissioned. • What do we need ? Grid Marketplace! • Regulates demand and supply, offers incentive for being players, simple, scalable solution, quasi-deterministic – proven model in real-world.

  19. Users in Grid Economy & Strategy • Grid Consumers • Execute jobs for solving varying problem size and complexity • Benefit by selecting and aggregating resources wisely • Tradeoff timeframe and cost • Strategy: minimise expenses • Grid Providers • Contribute “idle” resource for executing consumer jobs • Benefit by maximizing resource utilisation • Tradeoff local requirements & market opportunity • Strategy: maximise returns on services

  20. Building of a Economy Grid “brokerage” system….. Who pays for that ??? Users! Foundation for the Grid Economy

  21. Grid Architecture for Computational Economy Information Server(s) Grid Market Services Sign-on Health Monitor Info ? Grid Node N … Grid Explorer … Application Secure Job Control Agent Grid Node1 Schedule Advisor QoS Pricing Algorithms Trade Server Trading Trade Manager Accounting Resource Reservation Misc. services … Deployment Agent JobExec Resource Allocation Storage Grid User Grid Resource Broker … R1 R2 Rm Grid Middleware Services Grid Service Providers

  22. Economic Models for Trading • Commodity Market Model • Posted Prices Models • Bargaining Model • Tendering (Contract Net) Model • Auction Model • English, first-price sealed-bid, second-price sealded-bid (Vickrey), and Dutch. • Proportional Resource Sharing Model • Shareholder Model • Partnership Model

  23. Economy Grid = Globus + GRACE Applications Grid Apps. … Science Engineering Commerce Portals ActiveSheet High-level Services and Tools GlobusView Grid Status Grid Tools Nimrod/G DUROC MPI-G MPI-IO CC++ globusrun Core Services Heartbeat Monitor Nexus GRACE-TS Grid Middleware GRAM Globus Security Interface GASS DUROC MDS GBank GARA GMD Grid Fabric Local Services GRD QBank JVM Condor TCP UDP eCash LSF PBS Linux Irix Solaris

  24. What is Nimrod/G ? • A resource broker for managing and steering task farming (parametric sweep) applications on computational Grids based on deadline and computational economy. • Key Features • A single window to manage & control experiment • Resource Discovery • Trade for Resources • Scheduling • Steering & data management • It allows to study the behaviour of some of the output variables against a range of different input scenarios.

  25. Nimrod/G Grid Broker Architecture Nimrod/G Client Nimrod/G Client Nimrod/G Client Nimrod/G Engine Schedule Advisor Trading Manager Grid Store Grid Dispatcher Grid Explorer Grid Middleware Globus,Legion, Condor-g,, Ninf,etc. TM TS GE GIS Grid Information Server(s) RM & TS RM & TS RM & TS G C L G Legion enabled node. Globus enabled node. L C RM: Local Resource Manager, TS: Trade Server Condor enabled node.

  26. Cost A Nimrod/G Client Deadline Legion hosts Globus Hosts Bezek is in both Globus and Legion Domains

  27. User Requirements: Deadline/Budget

  28. Global Economy Grid Australia North America Monash Uni.: ANL: SGI/Sun/SP2 USC-ISI: SGI UVa: Linux Cluster Manitoba: Cluster Nimrod/G Linux cluster Globus+Legion +Condor/G Globus/Legion GRACE_TS Solaris WS Internet Europe ZIB/FUB: T3E/Mosix Cardiff: Sun E6500 Paderborn: HPCLine Lecce: Compaq SC CNR: Cluster CERN: Cluster Asia/Japan Tokyo I-Tech.: ETL, Tuskuba Linux cluster Globus + GRACE_TS Globus + GRACE_TS

  29. Conclusions • Proposed three models for Grid resource management architecture • Hierarchical, AO, & Market-model • The future systems are likely follow a model that combines all these models. • The future computing (HPC) infrastructure is going to be a Grid of Clusters. • Peer-to-Peer/Grid has already become a darling of venture capitalists. • The impact of Grid on 21st century economy will be the same as electricity on 20th century economy.

More Related