1 / 41

Grid Computing and Scaling Up the Internet

Grid Computing and Scaling Up the Internet. Ian Foster Argonne National Laboratory University of Chicago Globus Project Brian E Carpenter IBM and 6NET IPv6 Forum, San Diego, June 27, 2003. Why Grids and IPv6?.

lyre
Download Presentation

Grid Computing and Scaling Up the Internet

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Computingand Scaling Up the Internet Ian Foster Argonne National Laboratory University of ChicagoGlobus Project Brian E Carpenter IBM and 6NET IPv6 Forum, San Diego, June 27, 2003

  2. Why Grids and IPv6? • Grid computing represents a fundamental shift in how we approach distributed computing, like the fundamental shift in information access introduced by the Web • IPv6 represents a major step function in the Internet’s ability to scale, like the introduction of IPv4 twenty years ago • Inevitably there is synergy between these two game changers • Let’s share a common goal of reaching 10 billion Internet nodes

  3. Topics • Brief introduction to Grid computing • Why Grids and IPv6 need each other • Practical aspects of enabling GT3 for IPv6 • Future directions and summary

  4. The Grid Is … • A collaboration & resource sharing infrastructure with origins in the sciences • A distributed service integration and management technology • A disruptive technology that enables a virtualized, collaborative, distributed world • An open source technology & community • A marketing slogan • All of the above

  5. The (Power) Grid:On-Demand Access to Electricity Quality, economies of scale Time

  6. By Analogy, A Computing Grid • Decouple production and consumption • Enable on-demand access • Achieve economies of scale • Enhance consumer flexibility • Enable new devices • On a variety of scales • Department • Campus • Enterprise • Internet

  7. Not Exactly a New Idea … • “The time-sharing computer system can unite a group of investigators …. one can conceive of such a facility as an … intellectual public utility.” • Fernando Corbato and Robert Fano , 1966 • “We will perhaps see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.” • Len Kleinrock, 1967

  8. But Things are Different Now …

  9. But Wait A Minute—Computing isn’t Really Like Electricity! • I import electricity but must export data • “Computing” is not interchangeable but highly heterogeneous • Computers, data, sensors, services, … • So the story is more complicated • But more significantly, the sum can be greater than the parts • Real opportunity: Construct new capabilities dynamically from distributed services • Virtualization & distributed service mgmt

  10. Distributed service management Resource & service aggregation Delivery of virtualized services with QoS guarantees Dynamic, secure service discovery & composition Virtualization &Distributed Service Management Larger, more integrated More connected Dynamically provisioned Less capable, integrated Less connected User service locus Device Continuum

  11. Why the Grid?Origins: Revolution in Science • Pre-Internet • Theorize &/or experiment, aloneor in small teams; publish paper • Post-Internet • Construct and mine large databases of observational or simulation data • Develop simulations & analyses • Access specialized devices remotely • Exchange information within distributed multidisciplinary teams

  12. Edinburgh Glasgow DL Newcastle Belfast Manchester Cambridge Oxford Hinxton RAL Cardiff London Soton Tier0/1 facility Tier2 facility Tier3 facility 10 Gbps link 2.5 Gbps link 622 Mbps link Other link ExampleScience Grids

  13. The Grid/eScience World: Status • Dozens of major Grid projects in scientific & technical computing/research & education • Deployment, application, technology • www.mcs.anl.gov/~foster/grid-projects • Globus Toolkit™ broadly adopted as de facto standard for major protocols & services • Global Grid Forum a significant force for community building and standardization • GGF9: Seattle, June 2003, 800 people • www.ggf.org; 200+ organizations; Boeing, Merck, Ford, J&J, IBM, Platform, …

  14. Why the Grid?(2) Revolution in Business • Pre-Internet • Central data processing facility • Post-Internet • Enterprise computing is highly distributed, heterogeneous, loosely coupled, inter-enterprise (B2B) • Business processes increasingly computing- & data-rich • Outsourcing becomes feasible => on-demand service providers of various sorts

  15. CommoneScience/eBusiness Vision • Link dynamically acquired resources • From collaborators, customers, eUtilities, … (members of evolving “virtual organization”) • Into a “virtual computing system” • Dynamic, multi-faceted system spanning institutions and industries • Loose coupling of heterogeneous systems • Configured on demand to meet instantaneous needs, for: • Multi-faceted QoS for demanding workloads • Security, performance, reliability, …

  16. Managed shared virtual systems Computer science research Open Grid Services Arch Web services, etc. Real standards Multiple implementations Globus Toolkit Internet standards Defacto standard Single implementation The Emergence ofOpen Grid Standards Increased functionality, standardization Custom solutions 1990 1995 2000 2005 2010

  17. Open Grid Services Architecture • Service-oriented architecture • Key to virtualization, discovery, composition, local-remote transparency • Leverage industry standards • Internet, Web services • Distributed service management • A “component model for Web services” • A framework for the definition of composable, interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, Nick, Tuecke, 2002

  18. Client • Introspection: • What port types? • What policy? • What state? GridService (required) Other standard interfaces: factory, notification, collections Grid Service Handle Service data element Service data element Service data element handle resolution Grid Service Reference Open Grid Services Infrastructure • Lifetime management • Explicit destruction • Soft-state lifetime Data access Implementation Hosting environment/runtime (“C”, J2EE, .NET, …)

  19. Client • Introspection: • What port types? • What policy? • What state? GridService (required) Other standard interfaces: factory, notification, collections Grid Service Handle Service data element Service data element Service data element handle resolution Grid Service Reference Open Grid Services Infrastructure • Lifetime management • Explicit destruction • Soft-state lifetime GWD-R (draft-ggf-ogsi- gridservice-29) Editors: Open Grid Services Infrastructure (OGSI) S. Tuecke, ANL http://www.ggf.org/ogsi-wg K. Czajkowski, USC/ISI I. Foster, ANL J. Frey, IBM S. Graham, IBM C. Kesselman, USC/ISI T. McGuire, IBM T. Sandholm, ANL D. Snelling, Fujitsu Labs P. Vanderbilt, NASA April 5, 2003 Open Grid Services Infrastructure (OGSI) Version 1.0 Data access Implementation Hosting environment/runtime (“C”, J2EE, .NET, …)

  20. Resource allocation OGSA Interactions Create Service Authentication & Authorization are applied to all requests Grid Service Handle Service factory Service requestor (e.g. user application) Service data Keep-alives Notifications Service invocation Service discovery Register Service Service instances Service registry Interactions standardized using WSDL and SOAP

  21. Open Grid Services Architecture • Data access and integration • Security • Agreement and SLA negotiation • Manageability • … More specialized & domain-specific services Other schemas OGSA services: registry, authorization, monitoring, data access, management, etc., etc. OGSA schemas Open Grid Services Infrastructure Web Services Host. Env. & Protocol Bindings Hosting Environment Transport Hosting Environment Protocol GWD-R (draft-ggf-ogsa-platform-3) Editors: Open Grid Services Architecture Platform I. Foster, Argonne & U.Chicago http://www.ggf.org/ogsa-wg D. Gannon, Indiana U.

  22. Globus Toolkit v3 (GT3)Open Source OGSA Technology • Implements OGSI interfaces • Supports primary GT2 interfaces • High degree of backward compatibility • Multiple platforms & hosting environments • J2EE, Java, C, .NET, Python • New services • SLA negotiation, service registry, community authorization, data management, … • Rapidly growing adoption and contributions: “Linux for the Grid”

  23. Topics • General Introduction • Brief introduction to Grid computing • Why Grids and IPv6 need each other • Practical aspects of enabling GT3 for IPv6 • Future directions and summary

  24. Grid Deployment Options A function of business need, technology and organizational flexibility Inter-Grids Extra-Grids Intra-Grids Cactus (SF) Express Project NTG Grid Grid Grid VPN NAS/SAN NAS/SAN NAS/SAN Fin. Services MFG

  25. Virtual Organisation VirtualOrganisation Virtual Organisation Physical Organisation Physical Organisation Virtual Organization View of Deployment Physical Organisation Physical Organisation

  26. Global Knowledge Communities

  27. Virtual Organizations Look Like Dynamic Mergers & Acquisitions • The effect of a Grid VO on networks is like a temporary partial merger of the organizations • Merging two networks is painful today • “Private” IPv4 address space becomes ambiguous • Worst case: forced to renumber both networks • Temporary partial mergers of an arbitrary number of IPv4 networks is unthinkable • IPv4-based Grids are forced to rely on HTTP proxying between organizations: inefficient, and cannot exploit network-level security

  28. VO VO VO Overlapping Virtual Organizations • Any system can be in any number of VOs with any number of other systems • Needs uniform address space to avoid proxies & allow IPSEC • Addressing ambiguities unacceptable • Security boundaries ≠ organization boundaries • Not achievable at massive scale with IPv4

  29. Critical Advantages of IPv6for OGSA • Potential for massive scaling • Uniform global address space eliminates the problem of ambiguous “private” addresses and network address translation • Wasteful proxies can be avoided • Network level security can be used • Autoconfiguration is a big plus for infrastructure configuration

  30. There’s No Such Thing as an IPv6 Killer App, But ... • It would be nice to find the killer app that only works on IPv6 • OGSA won’t be that, but there is a good chance that it will be the first major middleware suite to be IPv6-capable out of the box almost from Day One • The IPv6 community should make the most of it

  31. Topics • Brief introduction to Grid computing • Why Grids and IPv6 need each other • Practical aspects of enabling GT3 for IPv6 • Future directions and summary

  32. Testing, Testing • 6NET is a three-year EU-funded project to demonstrate that continued growth of the Internet can be met using IPv6. • Includes a work package for IPv6 Middleware and User Application Trials (led by IBM) • Globus is the subject of a trial (lead site UCL) • Target is Globus Toolkit 3, i.e. OGSA • GT3 (OGSA) alpha code is now available and being tested on IPv6/Linux at UCL • Credits: Sheng Jiang, Piers O’Hanlon, Peter Kirstein

  33. Status (Evolving Daily) • Background: GT3 runs mainly over Java; Java2 Development Kit 1.4 supports IPv6, although GT3 alpha ships with JDK 1.3 • In principle, all GT3 Java components will “just work” with IPv6 via switch to JDK 1.4 • In practice, it is not so simple • Jim Bound of HP is also looking into this • UCL has begun testing with JDK 1.4. • GT3 Master Host Environment listens on both IPv6 and IPv4. Simple test from IPv6 completes, but some IPv4 packets are observed • Exact environment is GT3 alpha code with Java SDK 1.4.1. on Redhat Linux 7.3 and 8.0

  34. Status (update May 6th) • Tested the postgresql IPv6 patch; found one bug there: after removing IPv6 items from the configuration file, the IPv6 address was still enabled • Working to deploy GT3 core on Apache Tomcat Java servlet container (IPv6 enabled) • Waiting for IBM Websphere to do the same. • Starting to port the OGSA stand-alone web container to be IPv6-enabled • Sheng Jiang

  35. GridFTP (evolving daily) • GridFTP (striped FTP) as shipped with GT3 alpha is C code that does not support IPv6 sockets. • Globus is developing a new generic I/O module called XIO that does support IPv6 sockets • GridFTP is being rewritten, still in C, to exploit XIO • Any other components of GT3 that remain in C can also support IPv6 via XIO

  36. Further Plans (Evolving Daily) • Plan is to make more extensive tests with about 10 nodes • Issues with IPv6 will be reported into the Globus bug-tracking system • Good relations established between 6NET and Globus teams • Also need to consider what is required to operate GT3 in the cases of • IPv6 only • IPv6 and IPv4 coexistence • Final goal is a realistic systematic trial between 6NET sites

  37. Topics • General Introduction • Brief introduction to Grid computing • Why Grids and IPv6 need each other • Practical aspects of enabling GT3 for IPv6 • Future directions and summary

  38. Grid Past, Present, Future • Past • Origins and broad adoption in eScience, fueled by open source Globus Toolkit • Present • Rapidly growing commercial adoption • Open Grid Services Architecture (OGSA) • Future • Key enabler of new applications & industries based on resource virtualization and distributed service integration

  39. Future Directions • GT3 will become fully functional for IPv6 • Global Grid Forum must chase down any IPv4 dependencies in its standards • Grid computing will become a key enabler of new applications based on resource virtualization and loosely coupled distributed service integration • IPv6 will enable Grid Virtual Organizations to span existing network boundaries smoothly and securely

  40. Summary • Grid computing is the new model for sharing networked IT resources efficiently & securely. • Transforming the Internet into a computing platform for e-business on demand • The key toolkit is the Globus Toolkit open source package based on the Open Grid Services Architecture • A marriage between OGSA and IPv6 is the key to massive scaling in a fully connected but secure network environment.

  41. Pointers • www.globus.org • www.ggf.org • www.ipv6forum.org • www.6net.org brian@hursley.ibm.com foster@mcs.anl.gov

More Related