GRID Neue Netzdienste ZKI Arbeitskreis Netzdienste Würzburg , den 08. März 2004

1. GRIDNeue NetzdiensteZKI Arbeitskreis NetzdiensteWürzburg, den 08. März 2004 Dipl.-Inform. Markus Hillenbrand TU Kaiserslautern - Kaiserslautern University of Technology, Dept. of Computer ScienceICSY – Integrated Communication Systemshillenbr@informatik.uni-kl.de Tel.+49 631 205 4173, Fax. +49 631 205 3056

2. Evolution of the Internet • 1st generation (since the beginning of the Internet): • permanent IP addresses, always connected • static domain name system (DNS) mapping • limited specialized applications, protocols: Telnet, FTP, Gopher, World Wide Access • 2nd generation (since 90ties): • WWW & graphical browsers • dynamic IP addresses / NAT* (network address translation) / roaming users • heterogeneous applications, asymmetric server based services • protocol: HTTP, .. World Wide Web • 3rd generation (since 2000): • more collaboration and personalized applications • powerful edge devices (peers), instant networking • protocols/applications: • Napster, Gnutella,… • Grid World Wide Peering

3. Grid Vision • Ideas (and name) started with power grid generation • I don’t care where the power used to run my fridge is generated • unless You live in California • I just consume • and pay for my consumption • I could have a computer terminal Plug it in and pay for the computation as I go • But: • what application do I want to use? where is it? • what data is input? where is it? what results are produced? • where are they and how do I see them? • Are these transactions all really secure? • So it is not that simple

4. Grid Definition • Recall the following definition: • The transparent, systematic and effective utilisation of geographically distributed heterogeneous resources (both hardware and software) for applications in science and commerce

5. The Grid: Another Definition • “Dependable, consistent, pervasive access to[high-end] resources” • Dependable: Can provide performance and functionality guarantees • Consistent: Uniform interfaces to a wide variety of resources • Pervasive: Ability to “plug in” from anywhere

6. Grid Resources • Computational hardware • Storage (secondary, tertiary) • Networks • Databases, large datasets • Remote instruments • Applications • Libraries • Compilers and tools

7. So what´s a Grid? • Grids are large • in terms of potentially available resources • Grids are distributed • substantial latencies in moving data, may dominate application runtime • Grids are dynamic • resources may change during the lifespan of an application • Grids are heterogeneous • form and properties of sites (nodes) may differ significantly • Grids are across boundaries of organizations • access policies differ at different sites

8. Why Grid ? Why Grid? • large-scale, flexible, secure, coordinated resource sharing among dynamic collections of high-performance computers Characteristics of Grid • Inherited from Client/Server model • unrestricted registration and exit are NOT allowed • QoS ensured service • name the Grid node as Server Entity

9. Grid Applications

10. Grid Computing • GRID Objectives • Resource sharing among individuals, institutions and resources • Bandwidth, processing capacity, storage • Globally distributed heterogeneous resources • Grid • “Infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities” (1998) • “A system that coordinates resources not subject to centralized control, using open, general-purpose protocols to deliver nontrivial QoS” (2002) • good legal applications without communities • P2P • “Applications that takes advantage of resources at the edges of the Internet” (2000) • “Decentralized, self-organizing distributed systems, in which all or most communication is symmetric” (2002) • clever designs without good, legal applications

11. Grid and P2P

12. Grid and P2P • 1) Both are concerned with the same general problem • Resource sharing within virtual communities • 2) Both take the same general approach • Creation of overlays that need not correspond to underlying structures • Networks • Organizations • 3) Each has made genuine technical advances, but in complementary directions • “Grid addresses infrastructure but not yet failure” • “P2P addresses failure but not yet infrastructure” • 4) Complementary strengths and weaknesses => room for collaboration (Ian Foster)

13. From Web-Services to Grid-Services • Definition: • "An interface that describes a collection of operations that are network accessible through standardized XML messaging." • Self-contained, modular applications that can be described, published, located, and invoked over the Web. • Use open standards and common infrastructure for description, discovery and invocation (XML over HTTP plus: SOAP, WSDL, UDDI) • In the web… • The key was a universal server-to-client model based on standards • In Web Services … • The key will be a universal program-to-program communication model based on standards and industry support

14. Web-Services

15. Web-Services / Grid-Services • Web Services address discovery and invocation of persistent services • Interface to persistent state of entire enterprise • Definition: A Web service that adheres to OGSI standards is called a Grid service. • Grid Services must also support transient service instances that are dynamically created/destroyed • need interfaces to the states of distributed activities • Significant implications for how services are managed, named, discovered, and used! • OGSA: Open Grid Service Architecture • Within OGSA, everything is represented as a Grid service, that is, a (potentially transient) service that conforms to a set of conventions (expressed using WSDL) for such purposes as lifetime management, discovery of characteristics, and notification. • An important merit of this service-oriented model is that all components of the environment can be virtualized (and some of them can be created by a service factory on demand)

16. Infrastructure and Architecture • Global Grid Forum GGF • community-initiated forum of 5000+ individual researchers and practitioners • working on distributed computing ("grid" technologies). • primary objective • to promote and support the development, deployment, and implementation of Grid technologies and applications • via the creation and documentation of "best practices" - technical specifications, user experiences, and implementation guidelines. • Grid service • OGSI = Open Grid Service Infrastructure • Web services interfaces and behaviors • address key distributed system issues • Standard service specifications • OGSA = Open Grid Services Architecture • Resource mgt, db, workflow, security, … • OGSA defines “OGSA compliance” • … and arbitrary application-specific services based on these & other definitions

17. OGSI and Stateful Services • Sometimes you can send a message to a service, get a result and that’s the end • This is a statefree service • However most non-trivial services need state to allow persistent asynchronous interactions • OGSI is designed to support Stateful services through two mechanisms • Information Port: where you can query for SDE (Service Definition Elements) • “Factories” that allow one to view a Service as a “class” (in an object-oriented language sense) and create separate instances for each Service invocation • There are several interesting issues here • Difference between Stateful interactions and Stateful services • System or Service managed instances

18. OGSI = Open Grid Service Infrastructure Client • Introspection: • What port types? • What policy? • What state? • Lifetime management • Explicit destruction • Soft-state lifetime Data access GridService (required) Other standard interfaces: factory, notification, collections Grid Service Handle Service data element Service data element Service data element handle resolution Grid Service Reference Implementation Hosting environment/runtime (“C”, J2EE, .NET, …)

19. Open GRID Service Architecture - OGSA • OGSA Objectives • Integrating • GRID Technology • Web Services • Creation of • Virtual infrastructure • Service based architecture • Support of virtual organisations • OGSA Characteristics • Openness • Service interaction via published interfaces • Platform independent • Virtual service definition • Multiple interworking implementations • Portability • Design portability & code portability on hosting environments

20. OGSA / Web-Services • What is OGSA: • „One of the major distinctions between a Grid service and an ordinary Web-Service is the underlying assumption that Grid-Services may be transient, created as part of the normal operation of the infrastructure.“ [Ian Foster] • This leads to • Factories creating services • Lifetime management of services • Service groups

21. OGSA Services • OGSA defines broadly applicable services upon OGSI: • Core Services • Name resolution and discovery • Service domains • Security • Policy • Messaging, queuing, and logging • Events • Metering and accounting • Data and Information Services • Data naming and access • Replication • Metadata and provenance • Resource and Service Management • Provisioning and resource management • Service orchestration • Transactions • Administration and deployment

22. OGSA Future directions • Services and tools: • the set of OGSA services needs to be expanded concerning security, resource management, data and knowledge management, instrumentation and troubleshooting • Implementation: • high performance protocols, lightweight service instances, effective sandboxing techniques • Semantics: • formal mechanisms for expressing the behavior of individual services • Scalability: • multiple dimensions: large numbers of entities, service economies, embedded and mobile systems

23. WS-Resource Framework • WSRF is a family of Web services specifications (proposals) that clarify: • how stateful resources are addressed • how to access resource properties using XML • how to organize groups of resources and services • WSRF and WS-Notification are an evolution of OGSI • Ian Foster: • “The definition of WSRF means that Grid and Web communities move forward to a common base.” • Why is WSRF important? • WSRF completes Grid/Web convergence • How does WSRF relate to OGSI? • WSRF restates OGSI concepts in WS terms • How does WSRF relate to OGSA? • WSRF mechanisms will enable OGSA • What will Globus Alliance do with WSRF? • WSRF-based GT4.0 planned for Q3 2004 • What does WSRF mean for GT3.0 users? • For the most only minor changes

24. Application Areas • Targeted Application Areas • eScience • Distributed supercomputing • Resource sharing, e.g. CPU, bandwidth • Data intensive computing • Data sharing & collaborative computing • eLearning • eBusiness • eHealth • Globus Toolkit • Large set of tools • High level, pragmatic implementations • OGSA • Universal services architecture concept • Combining GRID and Web services

25. Important Things • Single sign-on: collective job set should require once-only user authentication • Mapping to local security mechanisms: some sites use Kerberos, others using Unix • Delegation: credentials to access resources inherited by subcomputations, e.g., job 0 to job 1 • Community authorization: e.g., third-party authentication

26. Sources of Grid Technology • Grids support distributed collaboratories or virtual organizations integrating concepts from • Web-Services, Semantic Web • W3C standard WSDL defines IDL (Interface standard) for Web Services • Agents • Distributed Objects (CORBA Java/Jini COM) • High Performance Computing activities • Peer-to-peer Networks • With perhaps the Web and P2P networks being the most important for “Information Grids” and Globus for “Compute Grids”

27. Activities • Look at different styles of Grids such as Autonomic (Robust Reliable Resilient) • New Grid architectures hard due to investment required • Critical Services Such as • Security – e-security • Notification – event services • Metadata – Use Semantic Web • Databases and repositories – instruments, sensors • Computing – Submit job, scheduling, distributed file systems • Visualization, Computational Steering • Fabric and Service Management • Network performance • Program the Grid – Workflow • Access the Grid – Portals, Grid Computing Environments

28. Conclusions • Gridsare inevitable and pervasive • Can expect Web Servicesand Grids to merge with a common set of general principles but different implementations with different scaling and functionality trade-offs • We will be flooded with data, information and purported knowledge • One should be preparing Grid strategies; understanding relevant Web and Grid standardsand developing new domain specific standards • Note many existing (standards) efforts assume client-server and not a brokered service model; these will need to change!

29. D-Grid Initiative

30. GRID Konzept • Komplexe, verteilte Arbeitsform in variablen, virtuellen Organisationen • Nutzung von im Netz verteilt verfügbaren Ressourcen wie Bandbreite, Rechner, Speicher, Information, Werkzeuge • Kopplung heterogener technischer und organisatorischer Umgebung über Netze  Anforderungen an Netze  Anforderungen an Middleware  administrative Anforderungen

31. GRID Aktivitäten • Bündelung der GRID Aktivitäten in: • USA: Cyberinfrastructure • UK: e-Science • NL: Virtual Lab • GGF • EU: 6th Framework Projekte (EGEE) • GRID Aktivitäten in Deutschland: • in verschiedenen Wissenschaftscommunities erste Ansätze: Teilchenphysik, Klimaforschung, Biowissenschaften • unkoordinierte, sporadische Förderung dieser Aktivitäten durch unterschiedliche Referate im BMBF • erste Koordinierungsversuche durch DFN (G3-Group) bis 2002 • unkoordinierte Mitarbeit in internationalen Gremien (z.B. GGF) nur durch Einzelaktivitäten

32. D-GRID: Entstehung • 2 Workshops (auf Initiative des FZ KA u. der HGF) • beteiligt alle relevanten Einrichtungen aus dem Wissenschaftsbereich + DFN + BMBF + Industrie • Konsens über Bündelung der GRID Aktivitäten • D-GRID: Ziele der Initiative • Bündeln der Aktivitäten, um Synergiepotenziale für globale,verteilte und neuartige Wissenschaftskollaborationen auf der Basis internetgestützter Dienste freizusetzen • Etablierung eines e-Science-Frameworks • D-GRID: Organisation • Bildung eines Lenkungsausschusses (LA) • Wahl der Herren Hegering (LRZ), Hiller (AWI), Maschuw (FZK, GridKa), Reinefeld (ZIB), Resch (HLRZS) • Aufgabe: Strategiepapier erstellen

33. D-GRID: Weiteres Vorgehen • Einrichtung von 5 AKs (horiz. Struktur) Unterzeichner der Grundsatzerklärung können mitarbeiten • Aktuelles Ziel aller AKs: Mitarbeit an der Erstellung des Arbeitsprogrammes der Initiative • Für jeden AK gilt: Einhaltung der terminlichen und inhaltlichen Vorgaben des LA ist zwingend für den Erfolg • Arbeitskreise: • AK1: Kooperationsmodelle für den Betrieb • AK2: Middleware und Services • AK3: Management Methoden und Autonomic Computing • AK4: Netz • AK5: Datenmanagement

34. Arbeitskreise • AK1: Kooperationsmodelle für den Betrieb • Bestandsaufnahme verfügbarer Ressourcen • Abrechnungsmodelle für Nutzung von Ressourcen, Ressourcenbewertung • Rechte für Ressourcennutzung • Weitere rechtliche und org. Fragestellungen • Bestandsaufnahme von Organisationsformen in Communities (im Sinne VO) • AK2: Middleware und Services • Bestandsaufnahme vorhandener Middleware • Interop.anforderungen an Middleware Komponenten • Bestimmung des Bedarfs (Security, AAA, VC, Directory, Roaming) • Ressourcebroker und weitere Dienste • Beschreibungssprachen, Modelling

35. Arbeitskreise • AK3: Management Methoden und AutonomicComputing • Rahmenbetriebsstrukturen der Middleware- und Serviceebene (SLAs, Support- und Operationszentren, Einbindung RZs), auch Forschungsaufgaben dazu • Qualitätsmanagement • Methoden für Self-Management, Self Healing... von Ressourcen und Diensten im Netz • AK4: Netz • Bestandsaufnahme existierender Netz-Dienste und -Protokolle (G-WiN u.a.) • Bestimmung neuer, Grid spezifischer Netzdienste • Bestimmung der Volumenbedarfe (Grids) • Performance Monitoring • AK5: Daten-und Informationsmanagement • Langzeitarchivierung • Metadaten, semantische Methoden und Informationsdienste • verteilte gemeinsame Dateisysteme • Massendatenverwaltung

36. D-GRID und DFN-Verein • Aufgaben des DFN-Vereins: • Bereitstellung der Netzinfrastruktur G-WiN • Aufbau und Betrieb von Basisdiensten der Middleware-Ebene, wie z.B. PKI, AAA • Aufgaben für Rechenzentren: • Aufbau von GRID-Know How vor Ort • Support für Nutzer bei Einführung von Middlewarediensten • Bereitstellen von Bausteinen für Middlewaredienste (z.B.)Nutzerverzeichnisse • Stärkung der Rolle der RZ als Anbieter für zentrale IT-Dienste