Grid Computing

1. Grid Computing Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago http://www.mcs.anl.gov/~foster

2. Partial Acknowledgements • Open Grid Services Architecture design • Carl Kesselman, Karl Czajkowski @ USC/ISI • Steve Tuecke @ANL • Jeff Nick, Steve Graham, Jeff Frey @ IBM • Grid services collaborators at ANL • Kate Keahey, Gregor von Laszewski • Thomas Sandholm, Jarek Gawor, John Bresnahan • Globus Toolkit R&D also involves many fine scientists & engineers at ANL, USC/ISI, and elsewhere (see www.globus.org) • Strong links with many EU, UK, US Grid projects • Support from DOE, NASA, NSF, IBM, Microsoft

3. Goals • Communicate the purpose, significance, state, adoption, & future of Grid technology • Persuade you that Grid technology represents an opportunity • Grids aren’t (particularly) about science or servers—themes of virtualization, service discovery, service management, and QoS delivery are universal • Rapid uptake in industry & science represents an exceptional opportunity for impact

4. Overview • Origins: Resource sharing within scientific collaborations • Science drivers & science Grid projects • Globus Toolkit • Evolution: Resource virtualization • Commercial drivers • OGSA: Grid meets Web services

5. Overview • Origins: Resource sharing within scientific collaborations • Science drivers & science Grid projects • Globus Toolkit • Evolution: Resource virtualization • Commercial drivers • OGSA: Grid meets Web services

6. E-Science: The Original Grid Driver • Pre-electronic science • Theorize &/or experiment, in small teams • Post-electronic science • Construct and mine very large databases • Develop computer simulations & analyses • Access specialized devices remotely • Exchange information within distributed multidisciplinary teams • Need to manage dynamic, distributed infrastructures, services, and applications

7. Galaxy cluster size distribution Chimera Virtual Data System + GriPhyN Virtual Data Toolkit + iVDGL Data Grid (many CPUs) eScience Application:Sloan Digital Sky Survey Analysis Size distribution of galaxy clusters? www.griphyn.org/chimera

8. Human Models NASA’s Information Power Grid: Aviation Safety Wing Models • Lift Capabilities • Drag Capabilities • Responsiveness Stabilizer Models Airframe Models • Deflection capabilities • Responsiveness Crew Capabilities - accuracy - perception - stamina - re-action times - SOPs Engine Models • Braking performance • Steering capabilities • Traction • Dampening capabilities • Thrust performance • Reverse Thrust performance • Responsiveness • Fuel Consumption Landing Gear Models

9. Life Sciences: Telemicroscopy DATA ACQUISITION PROCESSING,ANALYSIS ADVANCEDVISUALIZATION NETWORK COMPUTATIONALRESOURCES IMAGING INSTRUMENTS LARGE DATABASES

10. And Thus: The Grid “Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”

11. Underlying Technical Requirements • Dynamic formation and management of virtual organizations • Online negotiation of access to services: who, what, why, when, how • Configuration of applications and systems able to deliver multiple qualities of service • Autonomic management of distributed infrastructures, services, and applications

12. The Grid World: Current Status • Dozens of major Grid projects in scientific & technical computing/research & education • Open source Globus Toolkit™ a de facto standard for major protocols & services • Simple protocols & APIs for authentication, discovery, access, etc.: infrastructure • Information-centric design • Large user and developer base • Multiple commercial support providers • Enabler of numerous tools and applications • Global Grid Forum: community & standards

13. Overview • Origins: Resource sharing within scientific collaborations • Science drivers & science Grid projects • Globus Toolkit • Evolution: Resource virtualization • Commercial drivers • OGSA: Grid meets Web services

14. Resource Sharing within “VOs” is Not Unique to Science! • Fragmentation of enterprise infrastructure • Driven by cheap servers, fast nets, ubiquitous Internet, eBusiness workloads • Need to configure distributed collections of services to deliver specified QoS • Virtualization • Emerging service infrastructure, utility computing models, economies of scale • Services dynamically instantiated across device spectrum • B2B, B2C, C2C interactions

15. Distributed service management Resource & service aggregation Delivery of virtualized services with QoS guarantees Dynamic, secure service discovery & composition Virtualization andDistributed Service Management Larger, more integrated More connected Dynamically provisioned Less capable, integrated Less connected User service locus Device Continuum

16. Realizing the PromiseRequires Significant Innovation • Automation of infrastructure operation to achieve economies of scale • Management and component models for service discovery, composition, provisioning • New applications and tools powered by distributed services and resources • Business and service models to support specialization of function

17. Grid Evolution:Open Grid Services Architecture • Refactor Globus protocol suite to enable common base and expose key capabilities • Service orientation to virtualize resources and unify resources/services/information • Embrace key Web services technologies: WSDL as IDL, leverage commercial efforts • Result: standard interfaces & behaviors for distributed system management

18. OGSA System Structure • A standard substrate: the Grid service • Standard interfaces and behaviors that address key distributed system issues • The “Grid Service Specification” • … supports standard service specifications • Resource management, databases, workflow, security, diagnostics, etc., etc. • Target of current & planned GGF efforts • … and arbitrary application-specific services based on these & other definitions

19. Transient Service Instances • “Web services” address discovery & invocation of persistent services • Interface to persistent state of entire enterprise • In Grids, must also support transient service instances, created/destroyed dynamically • Interfaces to the states of distributed activities • E.g. workflow, video conferencing, distributed data analysis, workload management • Significant implications for how services are named, discovered, managed, and used

20. OGSI, OGSA, and Web Services • OGSI (I = Infrastructure) • Small extensions to WSDL • Nested serviceType & serviceDataDescription • Conventions for naming service instances • Handles and references • portTypes for common behavior • Instance creation, lifetime management, introspection and monitoring, registration, notification, … • OGSA (A = Architecture) built on OGSI • A collection of Grid service interfaces • Resource description & provisioning • Higher-level services: messaging services, logging, etc.

21. GridService (required) … other interfaces … (optional) Service data access Explicit destruction Soft-state lifetime Standard: - Notification - Authorization - Service creation - Service registry - Manageability - Concurrency + application-specific interfaces The Grid Service =Interfaces/Behaviors + Service Data Service data element Service data element Service data element Binding properties: - Reliable invocation - Authentication Implementation Hosting environment/runtime (“C”, J2EE, .NET, …)

22. Service Data • A Grid service instance maintains a set of service data elements • Described in WSDL extension • XML element encapsulated in standard container: name, type, lifetime, etc. • Includes basic introspection information, interface-specific data, and application state • Pull and push models for information query • GridService::FindServiceData operation • Pull: queries this information via extensible query language • NotificationSource::SubscribeServiceData • Push: Subscribe to notification of changes to information

23. Notification Interfaces • NotificationSource for client subscription • Subscription expression describes which service data element changes are of interest • Creates a subscription manager service • Manages the lifetime and properties of subscription • NotificationSink for asynchronous delivery of notification messages • Simple, flexible base with wide variety of uses • Dynamic discovery/registry services, monitoring, application error notification, etc. • Intermediaries: filter, aggregate, archive, et.c • Can integrate commercial messaging services

24. Grid Service Example:Database Service • A DBaccess Grid service will support at least two portTypes • GridService • DBaccess • Each has service data • GridService: basic introspection information, lifetime, … • DBaccess: database type, query languages supported, current load, …, … • Maybe other portTypes as well • E.g., NotificationSource Grid Service DBaccess Name, lifetime, etc. DB info

25. Lifetime Management • GS instances created by factory or manually; destroyed explicitly or via soft state • Negotiation of initial lifetime with a factory (=service supporting Factory interface) • GridService interface supports • Destroy operation for explicit destruction • SetTerminationTime operation for keepalive • Soft state lifetime management avoids • Explicit client teardown of complex state • Resource “leaks” in hosting environments

26. Factory • Factory interface’s CreateService operation creates a new Grid service instance • Reliable creation (once-and-only-once) • CreateService operation can be extended to accept service-specific creation parameters • Returns a Grid Service Handle (GSH) • A globally unique URL, resolves to GSR • Uniquely identifies the instance for all time • Based on name of a handle resolver • Or Grid Service Reference (GSR)

27. “Create a database service” “What services can you create?” Grid Service DBaccess Name, lifetime, etc. DB info “What database services exist?” Grid Service DBaccess Name, lifetime, etc. DB info Example:Transient Database Services Grid Service DBaccess Factory Instance name, etc. Factory info Grid Service Registry Instance name, etc. Registry info

28. OGSA Design & Implementation • OGSI (I=Infrastructure) WG in GGF defining core Grid service specification • (At least) three implementation efforts • Globus Toolkit => GT3 (alpha end 2002) • GT3 Core: Grid service specification • GT3 Base: Globus Toolkit behaviors • CIM resource model, GRAM-2 SLA negotiation, database services, … • Other GGF WGs address OGSA security, OGSA-compliant database services, etc.

29. Recap: Goals • Communicate the purpose, significance, state, adoption, & future of Grid technology • Persuade you that Grid technology represents a significant opportunity • Grids aren’t only (or particularly) about science and servers—themes of virtualization, service discovery, service management, and QoS delivery are universal • Rapid uptake in industry & science represents an exceptional opportunity for impact

30. For More Information • The Globus Project™ • www.globus.org • Context & research articles • www.mcs.anl.gov/~foster • Open Grid Services Architecture • www.globus.org/ogsa • Global Grid Forum • www.gridforum.org • Edinburgh, July 22-24 • Chicago, Oct 15-17

31. 1) OGSA builds on infrastructure • Plumbing: WSDL, WS-Security, WS-Routing/Referral, reliable messaging, transactions, etc. • Hosting environments OGSI/OGSA Interfaces service description, service provisioning, … Standard OGSI container Web services various 2) to enable virtualization via • Service description • Service provisioning Hosting Environment Resource virtualization and QoS support 3) Standard container avoids implementing OGSI features in every service instance OGSA Implementation OGSI/OGSA Interfaces service description, service provisioning, … Standard OGSI container Web services various Hosting Environment Resource virtualization and QoS support

32. Building an OGSI Container • Service data mgmt, query, subscription • Container should provide simple interface for interacting with an instance’s implementation to get and manage dynamic service data • Service instance = CLR object • Container should handle query processing • .NET support for XPath & Xquery allows for rich functionality • Container manages notification subscriptions, and drives asynchronous notification messages • Soft-state lifetime management • Soft-state registration