Introduction to GRID Computing

1. Introduction to GRID Computing Bebo White bebo@slac.stanford.edu New Directions in Information Technology Series Contra Costa College Fall 2005

2. Today’s Goals • To provide an introduction to key Grid computing and Web services issues, techniques, and technologies • To provide a substantial background and vocabulary to support future studies in Grid computing and Web services • To describe some of the current applications of Grid computing • To describe some of the current Grid computing initiatives

3. Grid Hype

4. The Power Grid -On-Demand Access to Electricity Decouple production & consumption, enabling • On-demand access • Economies of scale • Consumer flexibility • New devices Quality, economies of scale Time

5. The Shape of Grids to Come? Energy Internet

6. A Grid Checklist (#1) • A system that coordinates resources that are not subject to centralized control • Integrates and coordinates resources and users that live within different control domains – for example, the user’s desktop vs. central computing; different administrative units of the same company; or different companies; and addresses the issues of security, policy, payment, membership, and so forth that arise in these settings. • Otherwise we are dealing with a local management system (Ian Foster)

7. A Grid Checklist (#2) • A system that uses standard, open, general-purpose protocols and interfaces • Is built from multi-purpose protocols and interfaces that address such fundamental issues as authentication, authorization, resource discovery, and resource access. • It is important that these protocols and interfaces be standard and open. • Otherwise, we are dealing with an application-specific system. (Ian Foster)

8. A Grid Checklist (#3) • A system that delivers nontrivial qualities of service. • Allows its constituent resources to be used in a coordinated fashion to deliver various qualities of service, relating, for example, to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands, so that the utility of the combined system is significantly greater than the sum of its parts. (Ian Foster)

9. What is Grid Computing ? • Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations[ I.Foster] • A VO is a collection of userssharing similar needs and requirements in their access to processing, data and distributed resources and pursuing similar goals. • Key concept : • Ability to negotiate resource-sharing arrangements among a set of participating parties (providers and consumers) and then to use the resulting resource pool for some purpose[I.Foster]

10. The Grid Problem • Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations” • Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of… • central location, • central control, • omniscience, • existing trust relationships.

11. Elements of the Problem • Resource sharing • Computers, storage, sensors, networks, … • Sharing always conditional: issues of trust, policy, negotiation, payment, … • Coordinated problem solving • Beyond client-server: distributed data analysis, computation, collaboration, … • Dynamic, multi-institutional virtual orgs • Community overlays on classic org structures • Large or small, static or dynamic

12. The Grid Information Problem • There is a need for different views of the information depending upon • VO membership • Security constraints • Intended purpose • Etc.

13. Why Grids ? • Scale of the problems/applications • Solving problems that are bigger than any one data center can hold • Size of user communities • Leading research in many different fields today require collaborations that span research centers and countries (i.e. multi-domain access to distributed resources) • Need to provide access to large data processing power and huge data storage

14. What Kinds of Applications? • Computation intensive • Interactive simulation (climate modeling) • Large-scale simulation (galaxy formation, gravity waves, battlefield simulation) • Engineering (parameter studies, linked models) • Data intensive • Experimental data analysis (high energy physics) • Image, sensor analysis (astronomy, climate) • Distributed collaboration • Online instruments (microscopes, x-ray devices) • Remote visualization (climate studies, biology) • Engineering (structural testing, chemical)

15. Online Access to Scientific Instruments Advanced Photon Source wide-area dissemination desktop & VR clients with shared controls real-time collection archival storage tomographic reconstruction DOE X-ray grand challenge: ANL, USC/ISI, NIST, U.Chicago

16. Mathematicians Solve NUG30 • Looking for the solution to the NUG30 quadratic assignment problem • The problem involves assigning 30 facilities to 30 fixed locations so as to minimize the total cost of transferring material between the facilities. • An informal collaboration of mathematicians and computer scientists • Condor-G delivered 3.46E8 CPU seconds in 7 days (peak 1009 processors) in U.S. and Italy (8 sites) • 14,5,28,24,1,3,16,15, • 10,9,21,2,4,29,25,22, • 13,26,17,30,6,20,19, • 8,18,7,27,12,11,23 MetaNEOS: Argonne, Iowa, Northwestern, Wisconsin

17. Home Computers Evaluate AIDS Drugs • Community = • 1000s of home computer users • Philanthropic computing vendor (Entropia) • Research group (Scripps) • Common goal= advance AIDS research

18. Network for Earthquake Engineering Simulation • NEESgrid: national infrastructure to couple earthquake engineers with experimental facilities, databases, computers, & each other • On-demand access to experiments, data streams, computing, archives, collaboration NEESgrid: Argonne, Michigan, NCSA, UIUC, USC

19. High Energy Physics The LHC Detectors CMS ATLAS ~6-8 PetaBytes / year ~108 events/year ~103 batch and interactive users LHCb Federico.carminati , EU review presentation

20. ~PBytes/sec ~100 MBytes/sec Offline Processor Farm ~20 TIPS There is a “bunch crossing” every 25 nsecs. There are 100 “triggers” per second Each triggered event is ~1 MByte in size ~100 MBytes/sec Online System Tier 0 CERN Computer Centre ~622 Mbits/sec or Air Freight (deprecated) Tier 1 FermiLab ~4 TIPS France Regional Centre Germany Regional Centre Italy Regional Centre ~622 Mbits/sec Tier 2 Tier2 Centre ~1 TIPS Caltech ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS HPSS HPSS HPSS HPSS HPSS ~622 Mbits/sec Institute ~0.25TIPS Institute Institute Institute Physics data cache ~1 MBytes/sec 1 TIPS is approximately 25,000 SpecInt95 equivalents Physicists work on analysis “channels”. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz Tier 4 Physicist workstations Data Grids for High Energy Physics Image courtesy Harvey Newman, Caltech

21. Solving Large Problems – Pre-Grid Once upon a time…….. mainframe Microcomputer Mini Computer Cluster (by Christophe Jacquet)

22. The Grid Distributed Computing Idea …and today (by Christophe Jacquet)

23. Differences Between Grids andDistributed Applications • Huge distributed applications already exist, but they tend to be specialized systems intended for a single purpose or user group • e.g., SETI@Home, FightAIDS@Home • Grids go further and take into account: • Different kinds ofresources • Not always the same hardware, data and applications • No parallelization required • Different kinds of interactions • User groups or applications want to interact with Grids in different ways • Dynamic nature • Resources and users added/removed/changed frequently

24. The Grid: networked data processing centers and ”middleware” software as the “glue” of resources. Researchers perform their activities regardless geographical location, interact with colleagues, share and access data Scientific instruments and experiments provide huge amount of data The Grid Vision

25. Broader Context • “Grid Computing” has much in common with major industrial thrusts • Business-to-business, Peer-to-peer, Application Service Providers, Storage Service Providers, Distributed Computing, Internet Computing… • Sharing issues not adequately addressed by existing technologies • Complicated requirements: “run program X at site Y subject to community policy P, providing access to data at Z according to policy Q” • High performance: unique demands of advanced and high-performance systems

26. Grid Types - Physical Cluster GridEnterprise GridGlobal Grid

27. Grid Types - Logical • Data Grid responds to requests for computers and data stores; similar to (but more secure and auditable than) today's research grids • Information Grid responds to requests for computational processes, that may require several data sources and processing stages to deliver a desired result • Knowledge Grid responds to high-level questionsand finds the appropriate processes to deliver answers in the required form

28. The Classical (early) Grid • Focused on applications where data was stored in files • little support for transactions, relational database access or distributed query processing • Exploits a range of protocols such as: • LDAP for directory services and file store queries, • GridFTP for large-scale reliable data transfer • SSL for security

29. Why Now? • Moore’s law improvements in computing produce highly functional end systems • The Internet and burgeoning wired and wireless provide universal connectivity • Changing modes of working and problem solving emphasize teamwork, computation • Network exponentials produce dramatic changes in geometry and geography

30. Network Exponentials • Network vs. computer performance • Computer speed doubles every 18 months • Network speed doubles every 9 months • Difference = order of magnitude per 5 years • 1986 to 2000 • Computers: x 500 • Networks: x 340,000 • 2001 to 2010 • Computers: x 60 • Networks: x 4000 Moore’s Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan-2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.

31. The 13.6 TF TeraGrid:Computing at 40 Gb/s Site Resources Site Resources 26 HPSS HPSS 4 24 External Networks External Networks 8 5 Caltech Argonne External Networks External Networks NCSA/PACI 8 TF 240 TB SDSC 4.1 TF 225 TB Site Resources Site Resources HPSS UniTree TeraGrid/DTF: NCSA, SDSC, Caltech, Argonne www.teragrid.org

32. Tier0/1 facility Tier2 facility Tier3 facility 10 Gbps link 2.5 Gbps link 622 Mbps link Other link iVDGL:International Virtual Data Grid Laboratory U.S. PIs: Avery, Foster, Gardner, Newman, Szalay www.ivdgl.org

33. Main Services of a Grid Architecture • Service providers • Publish the availability of their services via information systems • Such services may come-and-go or change dynamically • E.g. a testbed site that offers x CPUs and y GB of storage • Service brokers • Register and categorize published services and provide search capabilities • E.g. 1) SLAC ResourceBroker selects the best site for a “job” 2)Catalogues of data held at each testbed site • Service requesters • Single sign-on: log into the Grid once • Use brokering services to find a needed service and employ it • E.g. CMS physicists submit a simulation job that needs 12 CPUs for 6 hours and 15 GB which gets scheduled, via the Resource Broker, on the CERN testbed site

34. Grid Security • Resource providers are essentially “opening themselves up” to itinerant users • Secure access to resources is required • X.509 Public Key Infrastructure • User’s identity has to be certified by (mutually recognized) national Certification Authorities (CAs) • Resources (node machines) have to be certified by CAs • Temporary delegation from users to processes to be executed “in user’s name” ( proxy certificates ) • Common agreed policies for accessing resource and handling user’s rights across different domains within VOs

35. The Globus Project™Making Grid computing a reality • Close collaboration with real Grid projects in science and industry • Development and promotion of standard Grid protocols to enable interoperability and shared infrastructure • Development and promotion of standard Grid software APIs and SDKs to enable portability and code sharing • The Globus Toolkit™: Open source, reference software base for building grid infrastructure and applications • Global Grid Forum: Development of standard protocols and APIs for Grid computing

36. g g g g g g Selected Major Grid Projects New New

37. g g g g g g Selected Major Grid Projects New New New New New

38. g g g g g g Selected Major Grid Projects New New

39. g g Selected Major Grid Projects New New Also many technology R&D projects: e.g., Condor, NetSolve, Ninf, NWS See also www.gridforum.org

40. GT1 GT2 OGSI Started far apart in apps & tech Have been converging WSRF WSDL2, WSDM WSDL, WS-* HTTP Where is Development of the Grid Going ? Grid Web The definition of WSRF means that Grid and Web communities can move forward on a common base

41. Standards • Grid and Web Services are merging • Grid is an aggressive use case of Web Services • WSRF completes common infrastructure • Web Services standards landscape is in flux • Uncertain status of security and policy standards continues to be a big source of concern • Grid services standards landscape heating up • Agreement, management, data access, … • Open source software important for adoption

42. Standards (cont) • Open, standard protocols • Enable interoperability • Avoid product/vendor lock-in • Enable innovation/competition on end points • Enable ubiquity • In Grid space, must address how to • Describe, discover, and access resources • Monitor, manage, and coordinate, resources • Account and charge for resources For many different types of resource

43. Standards (cont) • SSL/TLS v1 (from OpenSSL) (IETF) • LDAP v3 (from OpenLDAP) (IETF) • X.509 Proxy Certificates (IETF) • GridFTP v1.0 (GGF) • WSDL 1.1, XML, SOAP (W3C) • WS-Security (OASIS) • OGSI v1.0 (GGF) • And others on the road to standardization • WSRF (OASIS), DAIS (GGF), WS-Agreement (GGF), WSDL 2.0, WSDM, SAML, XACML

44. WSRF Specifications • List is still changing, but basically includes.. • Core: • WS-Resource Framework (WSRF) • WS-ResourceProperties (WSRF-RP) • WS-ResourceLifetime (WSRF-RL) • WS-ServiceGroup (WSRF-SG) • WS-Base Faults(WSRF-BF) • Related: • WS-Notifications • WS-Addressing

45. WSRF WSRF is a framework consisting of a number of specifications. • WS-Resource Properties • WS-Resource Lifetime • WS-Service Groups • WS-Notification • WS-BaseFaults • WS-Renewable References (unpublished) Other WS specifications such as: • WS-Addressing

46. How WSRF Fits in With Other Standards, Specifications and Protocols. Grid stuff Globus (GRAM, MDS) WSRF Web services WSDL, SOAP Internet protocols HTTP, TCP/IP

47. Describing Web Services • Web Services Description Language (WSDL) 2.0 • Status: W3C Last Call Working Draft • http://www.w3.org/TR/wsdl • WSDL is for describing Web Services • Defines XML-based grammar for describing network services as a set of endpoints • Describes their methods, arguments, return values and how to use • Approach: Service Oriented Architecture (SOA) • Service-Provider: • Develop a Web Service and publish its description as WSDL • Publish a link to it in a Service-Registry • Service-Consumer: • Service discovery, i.e. find WSDL, e.g. via Service-Registry • Use endpoint definition (WSDL) to communicate with service

48. Web Services Addressing • URIs (Uniform Resource Identifiers). • Look like URLs: • http://webservices.mysite.com/weather/us/WeatherService • When you have a Web Service URI, you will usually need to give that URI to a program • If you typed a Web Service URI into your web browser, you would probably get an error message or some unintelligible code • Some services include a polite response page

49. Service-Oriented Architecture ServiceProvider Publish Endpoint Definition Bind Registry: ServiceBroker ServiceConsumer Discovery

50. Web Services Architecture • WSDL: Core element of the Web Service Architecture stack (Endpoint definition language) Simplified Web Service Stack(WS-I Basic Profile 1.0 compliant) UDDI (service discovery) Web Service WSDL (service description) Listener XSD (service description) WSDL SOAP (messaging) Responder XML 1.0 + Namespaces (messaging)