1 / 35

Grids and Grid Technologies

Grids and Grid Technologies. Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago Member, Executive Committee, Global Grid Forum http://www.mcs.anl.gov/~foster. Overview. The Grid problem and concept

luka
Download Presentation

Grids and Grid Technologies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grids and Grid Technologies Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago Member, Executive Committee, Global Grid Forum http://www.mcs.anl.gov/~foster

  2. Overview • The Grid problem and concept • Review of major Grid projects • State of the art in Grid technologies • Global Grid Forum • Opportunities for integration

  3. Grid Concept Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals—in the absence of central control, omniscience, trust relationships

  4. Universal Nature of the Grid Problem • Original motivation, and support, from high-end science and engineering • But “sharing” fundamental in many settings • Application Service Providers, Storage Service Providers, etc.; Peer-to-peer computing; Distributed computing; B2B; … • Not adequately addressed by existing tech • Sharing at a deep level, across broad ranges of resources and in a general way • E.g., user provides ASP with controlled access to their data on an SSP: how??

  5. A Little History(U.S. Perspective) • Early 90s • Gigabit testbeds, metacomputing • Mid to late 90s • Early experiments (e.g., I-WAY), software projects (e.g., Globus), application experiments • 2001 • Major application communities emerging • Major infrastructure deployments are underway • Rich technology base has been constructed • Global Grid Forum: >1000 people on mailing lists, 192 orgs at last meeting, 28 countries

  6. “Major Application Communities are Emerging” • Intellectual buy-in, commitment • Earthquake engineering: NEESgrid • Exp. Physics, etc.: GriPhyN, PPDG • Simulation: Earth System Grid, Astrophysical Sim. Collaboratory • Emerging, e.g. • Natl. Ecological Obs. Network • Bioinformatics Grids • Fusion collaboratory • National Virtual Observatory

  7. ~PBytes/sec ~100 MBytes/sec Offline Processor Farm ~20 TIPS There is a “bunch crossing” every 25 nsecs. There are 100 “triggers” per second Each triggered event is ~1 MByte in size ~100 MBytes/sec Online System Tier 0 CERN Computer Centre ~622 Mbits/sec or Air Freight (deprecated) Tier 1 FermiLab ~4 TIPS France Regional Centre Germany Regional Centre Italy Regional Centre ~622 Mbits/sec Tier 2 Tier2 Centre ~1 TIPS Caltech ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS Tier2 Centre ~1 TIPS HPSS HPSS HPSS HPSS HPSS ~622 Mbits/sec Institute ~0.25TIPS Institute Institute Institute Physics data cache ~1 MBytes/sec 1 TIPS is approximately 25,000 SpecInt95 equivalents Physicists work on analysis “channels”. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz Pentium II 300 MHz Tier 4 Physicist workstations Grid Communities & Applications:Data Grids for High Energy Physics Image courtesy Harvey Newman, Caltech

  8. Grid Communities and Applications:Network for Earthquake Eng. Simulation • NEESgrid: national infrastructure to couple earthquake engineers with experimental facilities, databases, computers, & each other • On-demand access to experiments, data streams, computing, archives, collaboration NEESgrid: Argonne, Michigan, NCSA, UIUC, USC

  9. “Major Infrastructure Deployments are Underway” • Projects well under way • NSF “National Technology Grid” • NASA “Information Power Grid” • DOE ASCI DISCOM Grid • Starting/proposed • DOE Science Grid • NSF Dist. Terascale Facility • DOD MOD Grid • GRIDS Center: National Middleware Infrastructure

  10. “A Rich Technology Basehas been Constructed” • 6+ years of R&D have produced a substantial code base based on open architecture principles: esp. the Globus Toolkit, including • Grid Security Infrastructure • Resource directory and discovery services • Secure remote resource access • Data Grid protocols, services, and tools • Essentially all projects have adopted this as a common suite of protocols & services • Enabling wide range of higher-level services

  11. Large U.S. Grid Projects And in Europe: EU DataGrid EuroGrid GridLab AstroGrid GridPP Etc. • DOE ASCI DISCOM • DOE Particle Physics Data Grid • DOE Earth Systems Grid • DOE Science Grid • DOE Fusion Collaboratory • NASA Information Power Grid • NSF National Technology Grid • NSF Network for Earthquake Eng Simulation • NSF Grid Application Development Software • NSF Grid Physics Network • + smaller computer science research: portals, security, resource management, discovery, monitoring, programming models, etc., etc.

  12. Grid Challenges and Technologies • New applications enabled by the coordinated use of geographically distributed resources • E.g., distributed collaboration, data access and analysis, distributed computing • Persistent infrastructure for large-scale resource sharing [“systems”] • E.g., authentication, authorization, policy; protocols for resource discovery/access, etc. • Developing programs [“programming”] • Abstractions, tools

  13. Application Application Internet Protocol Architecture “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services Collective “Sharing single resources”: negotiating access, controlling use Resource “Talking to things”: communication (Internet protocols) & security Connectivity Transport Internet “Controlling things locally”: Access to, & control of, resources Fabric Link Layered Grid Architecture(By Analogy to Internet Architecture)

  14. Grid Services Architecture:Connectivity Layer Protocols & Services • Communication • Internet protocols: IP, DNS, routing, etc. • Security: Grid Security Infrastructure (GSI) • Uniform authentication & authorization mechanisms in multi-institutional setting • Standards: SSL/TLS, X.509 + CA, GSS-API • Extensions for single sign-on, delegation • Credential management: Login, logout, etc.; smartcards; online credential repositories for Web portal login and delegation; K5cert for automatic X.509 certificate creation GSI: www.globus.org

  15. GSI Working Group Documents(GGF -> IETF) • Grid Security Infrastructure (GSI) Roadmap • Informational draft overview of working group activities and documents • Grid Security Protocols & Syntax • X.509 Proxy Certificates • X.509 Proxy Delegation Protocol • The GSI GSS-API Mechanism • Grid Security APIs • GSS-API Extensions for the Grid • GSI Shell API

  16. Current and Future Work • Ease of use • CA operation, credential mgt, account mgt • Authorization • Policy languages, community authorization • Protection (despite compromised resources) • Restricted delegation, smartcards • Flexible communication support • GSS-API extensions • Independent Data Units (UDP, IP multicast) • Apply GSI to new areas • Personal Area Networks, PDAs, wireless, etc.

  17. Grid Services Architecture:Resource Layer Protocols & Services • Grid Resource Access & Mgmt (GRAM) • Remote allocation, reservation, monitoring, control of compute resources • GridFTP protocol (FTP extensions) • High-performance data access & transport • Grid Information Service protocols • Registration, information access/monitoring • Network reservation, monitoring, control • All integrated with GSI: authentication, authorization, policy, delegation GRAM, GridFTP, GRIS: www.globus.org

  18. Resource Management Problem • Enabling secure, controlled remote access to computational resources and management of remote computation • Authentication and authorization • Resource discovery & characterization • Reservation and allocation • Computation monitoring and control • Addressed by new protocols & services • GRAM protocol as a basic building block • Resource brokering & co-allocation services • GSI for security, MDS for discovery

  19. GRAM Protocol • Simple HTTP-based RPC • Job request • Returns a “job contact”: Opaque string that can be passed between clients, for access to job • Job cancel • Job status • Job signal • Event notification (callbacks) for state changes • Pending, active, done, failed, suspended • Moving to SOAP (more later…)

  20. GridFTP: Basic Approach • FTP is defined by several IETF RFCs • Start with most commonly used subset • Standard FTP: get/put etc., 3rd-party transfer • Implement standard but often unused features • GSS binding, extended directory listing, simple restart • Extend in various ways, while preserving interoperability with existing servers • Parameter set/negotiate, parallel transfers (multiple TCP streams), striped transfers (multiple hosts), partial file transfers, automatic & manual TCP buffer setting, progress monitoring, extended restart (via plug-ins)

  21. Grid Information Service • Provide access to static and dynamic information regarding system components • Large numbers of “sensors”, in resources, services, applications, etc. • A basis for configuration and adaptation in heterogeneous, dynamic environments • Requirements and characteristics • Uniform, flexible access to information • Scalable, efficient access to dynamic data • Access to multiple information sources • Decentralized maintenance

  22. The GIS Problem: Many Information Sources, Many Views

  23. Globus MDS:Registration and Inquiry Protocols • Registration (GRRP) and inquiry (GRIP) protocols • Both currently based on LDAP protocols & models • Support creation of aggregate directories providing application-specific views of resources sets

  24. Grid Services Architecture:Collective Layer Protocols & Services • Index servers aka metadirectory services • Custom views on dynamic resource collections assembled by a community • Resource brokers (e.g., Condor Matchmaker) • Resource discovery and allocation • Replica catalogs • Co-reservation, co-allocation (choreography) • Etc., etc. Metadirectory: www.globus.org; Condor: www.cs.wisc.edu/condor

  25. Grid Protocols in Action • Compute server • GRAM: reservation, job submit/monitor/control • MDS: discovery/monitoring of availability • GridFTP: staging data in/out • Storage server • GRAM: disk space and bandwidth reservations • MDS: discovery of available space, utilization • GridFTP: client-server, server-to-server transfers • Bandwidth broker • GRAM: bandwidth reservation and allocation • MDS: discovery/monitoring of availability • Online instrument • GRAM: reservation; MDS: discover use and features • GridFTP for transferring data from the device

  26. The Programming Problem • Protocols and APIs are necessary but not sufficient for application development • Programming needs abstractions and tools that reduce complexity, enable reuse, facilitate difficult and/or common tasks • Presumably can occur at multiple levels • Low-level programming models and tools • Higher-level domain-specific libraries • Workflow, GUI tools • No reason to expect one “right” answer

  27. Examples ofProgramming Technologies • MPICH-G2: Grid-enabled message passing • CoG Kits, GridPort: Portal construction, based on N-tier architectures • GDMP, Data Grid Tools, SRB: replica management, collection management • Condor-G: simple workflow management • Legion: object models for Grid computing • Cactus: Grid-aware numerical solver framework

  28. Grid Forum: Initial Motivation (1998) • Diverse efforts building “Grids” • 12+ multi-institutional projects from most major US Federal Agencies (NASA, NSF, DOE, DOD, etc.) • Several dozen Grid “frameworks” or components • Hypothesis—Sufficient common interest that coordination will be useful to: • Discuss common interests, problems, solutions • Identify/define standards to promote code sharing/interoperability • Promote Grid Technology • No obvious fit with existing groups (e.g. IETF, ISOC, W3C, OMG, etc.)

  29. “Global Grid Forum”: History 1988 1999 2000 GF BOF (Orlando) GGF-1 GF1 (San Jose — NASA Ames) GF2 (Chicago — iCAIR) eGrid and GF BOFs (Portland) GF3(San Diego — SDSC) Global GF 1 (Amsterdam) eGrid1(Posnan — PSNC) GF4 (Redmond — Microsoft) Asia-Pacific GF Planning (Yokohama) eGrid2 (Munich — at Europar) GF5 (Boston — Sun) Global GF BOF (Dallas) www.gridforum.org Chair = Charlie Catlett, Argonne

  30. Global Grid Forum 1 (March 2000)U.S. and European GFs Merge • 325 Participants • Capped by facilities • 60 late-registrants turned away • 192 Organizations • Previous high: 110 at GF-5 • 28 Countries • Previous high: 11 at GF-5 • 85 Document Authors Next meeting: GGF2, Washington DC, July 2001: www.gridforum.org

  31. Grid Forum Areas and Selected Groups Working groups Research groups • Grid Information Services • Modeling, Schema, JINI • Scheduling and Resource Management • Advance reservation, resource management, NPI • Security • Security Infrastructure, CA Policy • Grid Performance • Performance Monitoring Architecture • Applications • Various • Architectures and Frameworks • GridFTP, Data Arch., Protocol Arch., Economic Models • Programming and Run Time Environments • Computing Envs., Prog. Models, Collaborative Envs.

  32. Relationships • Grid technologies are complementary to other distributed computing technologies • Additive, not competitive • To date, have addressed primarily systems issues of interoperability and sharing • Need to integrate with tools that address programming, workflow, modeling issues • Ideally, also integrate with other “systems” technologies • Integration with other technologies critical

  33. Future Directions: Grid Services? • Core Grid protocols predate Web Services • Various substrates: HTTP, LDAP, FTP • We are currently exploring retargeting to Web Services, as part of ongoing redesign • SOAP for all protocols, WSDL, UDDI • “Grid Services” layer provides soft-state, secure management of remote context • Other Grid protocols build on this substrate • Grid Services enhance Web Services with key Grid (“peer-to-peer”) capabilities

  34. Summary • Grids are being developed by a substantial community facing challenging problems • Considerable experience with large-scale Grids, & some useful technology • Emerging “standards” are being adopted within community, also starting in IETF • We see commonality of interest, and perhaps opportunities for joint work, with W3C and OMG

  35. For More Information • Book (Morgan Kaufman) • www.mkp.com/grids • Globus • www.globus.org • “The Anatomy of the Grid: Enabling Scalable Virtual Organizations” • Global Grid Forum • www.gridforum.org • GriPhyN • www.griphyn.org

More Related