Introduction to Grid Technology

1. Introduction to Grid Technology Adam Belloum Computer Architecture & Parallel Systems group University of Amsterdam adam@science.uva.nl

2. The Grid Problem • Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource • Enable communities (“Virtual Organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of : central location, central control, omniscience, existing trust relationships. • From the Grid tutorials available at : http:.//www.globus.org

3. Online Access to Scientific Instruments Advanced Photon Source wide-area dissemination desktop & VR clients with shared controls real-time collection archival storage tomographic reconstruction DOE X-ray grand challenge: ANL, USC/ISI, NIST, U.Chicago • From the Grid tutorials available at : http:.//www.globus.org

4. ~PBytes/sec ~100 MBytes/sec Offline Processor Farm ~20 TIPS There is a “bunch crossing” every 25 nsecs. There are 100 “triggers” per second Each triggered event is ~1 MByte in size ~100 MBytes/sec Online System Tier 0 CERN Computer Centre ~622 Mbits/sec or Air Freight (deprecated) Tier 1 France Regional Centre Germany Regional Centre Italy Regional Centre FermiLab ~4 TIPS ~622 Mbits/sec Tier 2 Tier2 Centre ~1 TIPS Caltech ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS HPSS HPSS HPSS HPSS HPSS ~622 Mbits/sec Institute ~0.25TIPS Institute Institute Institute Physics data cache ~1 MBytes/sec 1 TIPS is approximately 25,000 SpecInt95 equivalents Physicists work on analysis “channels”. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz Tier 4 Physicist workstations Data Grids forHigh Energy Physics Image courtesy Harvey Newman, Caltech • From the Grid tutorials available at : http:.//www.globus.org

5. Mathematicians Solve NUG30 • Looking for the solution to the NUG30 quadratic assignment problem • An informal collaboration of mathematicians and computer scientists • Condor-G delivered 3.46E8 CPU seconds in 7 days (peak 1009 processors) in U.S. and Italy (8 sites) • 14,5,28,24,1,3,16,15, • 10,9,21,2,4,29,25,22, • 13,26,17,30,6,20,19, • 8,18,7,27,12,11,23 MetaNEOS: Argonne, Iowa, Northwestern, Wisconsin • From the Grid tutorials available at : http:.//www.globus.org

6. Some Definitions of the Grid? “A Computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities”. Karl Kesselman & Ian Foster. “The overall motivation for Grids is to enable the routine interactions of resources geographically and organizationally dispersed to facilitate Large-scale Science and engineering” The Vision for a DOE Science Grid, William Johnston, Lawrence Berkeley Nat. Lab. “Making possible a shared large wide-area Computational infrastructure a concept which has been named the Grid” Peter Dinda, Gorgia Tech, 2001.

7. What people think of the Grid? “The Grid is the next evolutionary step for supercomputing.” Jim Gray, Microsoft Research “The Grid represents the first wave of Computing in the well-connected world into which we are heading” David Culler, University of California, Berkeley “It is clear that distributed information processing will lie at the heart of many of the technology of the 21st century” William J. Feiereisen, Program Manager High Performance Computing and Communications, NASA etc.

8. Misuse of the term Grid? “then term Grid has been conflated, at least in the popular perception, to embrace everything from advanced networking to artificial intelligence”. “One may wonder whether the term has any real substance and meaning” From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations” Foster et al

9. Some Facts • Grid concepts are driven by real and specific problems • Grid is distinct from Internet, enterprise, distributed, and peer to peer computing

10. Grid vs Distributed Computing • The two either overlap or distributed computing is a subset of grid computing • Large scale resource sharing, innovative applications and high performance • Grid architecture: protocols, APIs, SDKs

11. The real Grid Problem • The real and specific problem that underlies the Grid concept is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations • direct access to computers • software, • data, and files

12. Coordinated Sharing • The sharing is highly controlled by the providers and consumers • what is shared • who is allowed to share • and the conditions under which sharing occurs • sharing relationships • client-server, peer-to-peer, and brokered • access control: fine AC, delegation, local/global policies

13. Internet technologies: communication and information exchange and but not the coordinated use of resources at multiple sites Business-to-business information sharing via centralized servers Enterprise distributed computing (CORBA, Java Ent.) resource sharing within a single enterprise SSP and ASP outsource storage and computing requirements to other parties in constrained way ( via virtual private Network) Grid complete rather than compete Other Technologies

14. Common Object Request Broker Architecture (CORBA) • Vendor-independent architecture and infrastructure that computer applications use to work together over networks. • Using the standard protocol IIOP, a CORBA-based program from any vendor, on almost any computer, operating system, programming language, and network, can interoperate with a CORBA-based program from the same or another vendor, on almost any other computer, operating system, programming language, and network. http://www.omg.org/gettingstarted/corbafaq.htm

15. Common Object Request Broker Architecture (CORBA) • Defines an architecture for distributed objects. • The basic CORBA paradigm is that of a request for services of a distributed object. • The services provided by an object are given by its interface. Distributed objects are identified by object references typed by IDL interfaces.

16. Remote Method Invocation (RMI) • Enables to create distributed Java technology-based to Java technology-based applications • the methods of remote Java objects can be invoked from other JVM*, possibly on different hosts. • A Java technology-based program • can make a call on a remote object • once it obtains a reference to the remote object, • either by looking up the remote object in the bootstrap naming service provided by RMI • or by receiving the reference as an argument or a return value http://java.sun.com/products/jdk/rmi/

17. Remote Method Invocation (RMI) The three layers of the RMI system consist of the following: • stub/skeletons • client-side stubs (proxies) • server-side skeletons (dispatchers) • remote reference layer invocation behavior and reference semantics (e.g., unicast, multicast) • transport connection set up and management and remote object tracking http://www.usenix.org/publications/library/proceedings/coots96/full_papers/wollrath/

18. Jini • Jini network technology is an open architecture that enables developers to create network-centric services that are highly adaptive to change. • Jini technology can be used to build adaptive networks that are scalable, evolvable and flexible as typically required in dynamic computing environments. http://wwws.sun.com/software/jini/

19. Jini • Components • Lookup service • Service Provider • Client • Service Registration • Client lookup • Proxy http://pandonia.canberra.edu.au/java/jini/tutorial/Jini.xml

20. Enterprise JavaBean • The JavaBeans component architecture is the platform-neutral architecture for the Java application environment. • developing or assembling network-aware solutions for heterogeneous hardware and operating system environments--within the enterprise or across the Internet • The JavaBeans architecture takes interoperability a major step forward--your code runs on every OS and also within any application environment. • The JavaBeans specification defines a set of standard component software APIs for the Java platform. http://java.sun.com/docs/books/tutorial/javabeans/

21. Enterprise JavaBean • JavaBeans Components are self-contained, reusable software units that can be visually composed into composite components, applets, applications, and servlets using visual application builder tools. • Beansexpose their features (public methods and events) to builder tools for visual manipulation. • JB-enabled builder tool can then • examine the Bean's patterns, discern its features, and expose those features for visual manipulation. • maintains Beans in a palette or toolbox. http://www.stlwebdev.org/meetings/past/1999/archives/19991116/

22. The Grid is an emerging technology • The Grid is not yet mature • The fundamentals of the Grid are still being discussed within the Global Grid Forum (http://www.gridforum.org) • Current State • Web service oriented architecture (OGSA)

23. Who needs to interact with the Grid? Grid developers: Implement the basic services exp:Globus toolkit Tool developers: Implement the tools, compiler and libraries exp: Condor-G, MPI-G, … Application developers: Implement grid-enabled application exp:VLAM-G End user: Use grid-enabled applications, as most of the users today use the Internet-enabled applications

24. How the Grid is going to help? • The tool and application developers • dynamic resource allocation • resource co-allocation • heterogeneous and dynamic computations • heterogeneous and dynamic communications • data and information management • collaboration • security • The End users • single sign-on for all the resources • uniform interface to various local and distributed resource • support for staging executables

25. References • Grid Information Services for Distributed Resource Sharing. K. Czajkowski, S. Fitzgerald, I. Foster, C. Kesselman. Proceedings of the Tenth IEEE Int. Symposium on High-Performance Distributed Computing (HPDC-10), August 2001 • The Anatomy of the Grid: Enabling Scalable Virtual Organizations. I. Foster, C. Kesselman, S. Tuecke. International J. Supercomputer Applications, 15(3), 2001. • The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration. I. Foster, C. Kesselman, J. Nick, S. Tuecke, Open Grid Service Infrastructure WG, Global Grid Forum, June 22, 2002. • The Grid: Blueprint for a New ComputingInfrastructure, I. Foster, C. Kesselman , Morgan-Kaufman, 1999. • Data Management and Transfer in High Performance Computational Grid Environments. B. Allcock, et al. Parallel Computing Journal, Vol. 28 (5), May 2002, pp. 749-771.

26. Grid Architecture***A few years ago ***

27. Application Application Internet Protocol Architecture “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services Collective “Sharing single resources”: negotiating access, controlling use Resource “Talking to things”: communication (Internet protocols) & security Connectivity Transport Internet “Controlling things locally”: Access to, & control of, resources Fabric Link Layered Grid Architecture(By Analogy to Internet Architecture)

28. Fabric layer • Provides the resources to which shared access is mediated by the Grid protocols • resource may be a logical entity (exp. File system, …) in this case internal protocols are not part of the Grid architecture • Implements the local resource specific operations • Resource-specific characterization of capabilities • Starting programs, monitoring, and controlling the execution of the resulting processes. • Mechanisms for putting and getting files or executing remote data selection or reduction function • Enquiry functions for determining software and hardware characteristics • Mechanisms that provide control over the resources allocated to network transfers

29. Connectivity layer • Defines the core communication and authentication protocols required for the Grid-specification network transactions. • Communication protocols enable the exchange of data between Fabric layer resources. • Authentication protocols build on communication services to provide cryptographically secure mechanisms for verifying the identity of users and resources. • Flexible support for communication protection • control over degree of protection • independent data protection for unreliable protocol • support for reliable transport protocol other than TCP • Enable stakeholder control over authorisation decisions

30. Resource layer • Defines protocols (APIs and SDKs) for the secure initiation, monitoring, and control of sharing operations on individual resources • information protocols obtain information about the structure and the state of resources (local policy, usage, …) • management protocols negotiate access to a shared resource • The resource layer protocols call fabric layer function to access and control local resources. • The resource layer protocols are concerned with individual resources and ignore issues of global state and atomic actions across distributed actions

31. Collective layer • Coordinates multiple resources • Contains protocol and services (and APIs and SDKs) concerned with the global state and captures interactions across collections of resources • They implement a wide variety of sharing behaviors without placing new requirements on the resources: • Directory services • co-allocation • Monitoring and diagnosis • data replication services • Software discovery • community authorization • collaboratory services

32. Protocols, Services,and APIs Occur at Each Level Applications Languages/Frameworks Collective Service APIs and SDKs Collective Service Protocols Collective Services Resource APIs and SDKs Resource Service Protocols Resource Services Connectivity APIs Connectivity Protocols Local Access APIs and Protocols Fabric Layer

33. Open Grid Services Architecture

34. “… Grid as an extensible set of Grid services that may be aggregated in various ways to meet the needs of V.O. ….” “the term architecture denoting here a well defined set of basic interfaces from which can be constructed interesting systems” Open Grid Services Architecture From “The Physiology of the Grid: an Open Architecture for Distributed Systems Integration” Foster et al

35. Open Grid Services Architecture • Defines a uniform exposed service semantics • Defines standard mechanisms for creating, naming, and discovering Grid Services instances • Provides location transparency and multiple protocol bindings for services instances

36. Open Grid Services Architecture • In terms of Web Services Description language (WSDL) • Defines the interfaces and the associated conventions • Mechanisms required for creating composing sophisticated distributed systems • The Grid technology is being aligned to Web services technology • Web service describes a distributed paradigm using Internet-based Standards to address heterogeneous distributed computing

37. Components relevant to OGSA • Grid Information System (GIS) • information discovery, data modeling, local registry • Grid Resource Allocation and Management (GRAM) • secure and reliable service creation and management of arbitrary computations “transient service instances” • Grid Security Infrastructure (GSI) • single sign on, delegation, and credential mapping

38. Conventions and Transport protocols in OGSA • Service must be upgradeable without disturbing the client operation • OGSA defines mechanisms for refreshing the client’s knowledge of a service (operation supported, network protocol …) • reliable service invocation • Authentication

39. OGSA Grid Service interfaces operation • Find Service Data • setTerminationTime • Destroy • subscribeToNotificationTopic • DeliverNotification • RegisterService • UnregisterService • CreateService • FindByHandel Interface or portType (WSDL) GridService NotificationSource NotificationSink Registry Factory HandelMap

40. OGSA Interfaces • Discovery • Standard presentation for service data, operation FindServiceData, and interfaces for registering information. • Dynamic service creation • Lifetime management • Notification • Other Interfaces

41. The Hosting Environment • OGSA defines the Semantics of a Grid service (how it is created, how to communicate whith, …) • OGSA doesn’t address implementation issues (programming model, programming language, …) • It’s the hosting env. Which address the implementation issues as well as how the Grid service meets its obligations with respect to Grid service semantics • Web services are build on containers hosting env. Such as J2EE, websphere, .NET, Sun one.