1 / 42

Grid Dynamics

Grid Dynamics. Ian Foster Argonne National Laboratory University of Chicago Univa Corporation. Acknowledgements. Carl Kesselman, with whom I developed many ideas (& slides) Bill Allcock, Charlie Catlett, Kate Keahey, Jennifer Schopf, Frank Siebenlist, Mike Wilde @ ANL/UC

coye
Download Presentation

Grid Dynamics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Dynamics Ian Foster Argonne National Laboratory University of Chicago Univa Corporation

  2. Acknowledgements • Carl Kesselman, with whom I developed many ideas (& slides) • Bill Allcock, Charlie Catlett, Kate Keahey, Jennifer Schopf, Frank Siebenlist, Mike Wilde @ ANL/UC • Ann Chervenak, Ewa Deelman, Laura Pearlman @ USC/ISI • Karl Czajkowski, Steve Tuecke @ Univa • Numerous other fine colleagues in NESC, EGEE, OSG, TeraGrid, etc. • NSF & DOE for research support

  3. ? What is the Grid? “Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations” “When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances” (George Gilder) “The Anatomy of the Grid”, Foster, Kesselman, Tuecke, 2001

  4. Decomposition Implementation Facilities Computers Storage Networks Services Software People U. Colorado Experimental Model UIUC Experimental Model COORD. NCSA ComputationalModel System-Level Problem Grid technology

  5. BI Server 2 Dispatcher Grid backend Grid-enabled Business Intelligence Application Provision New Worker Process ManagedPool of Shared Resources BI server applications started and decommissioned by a Grid-enabled dispatcher

  6. Grid Dynamics:Vision vs. Reality • Vision: On-demand access to computing • New communities form easily • On-demand resources from providers • Adapt easily to new missions, requirements • Reality: Much manual configuration, e.g.: • Manually deployed services on dedicated hardware • Manually maintained access control lists • Sysadmin-maintained allocation policies • Human-mediated resource reservation

  7. Users Discovery tools Analysis tools Data Archives Fig: S. G. Djorgovski Grid Dynamics:A Two-Dimensional Problem • Decompose across network Clients integrate dynamically • Select & compose services • Select “best of breed” providers • Publish result as new services • Decouple resource & service providers Function Resource

  8. Provisioning Service-Oriented Systems:The Role of Grid Infrastructure Users • Service-oriented Gridinfrastructure • Provision physicalresources to support application workloads • Service-oriented applications • Wrap applications as services • Compose applicationsinto workflows Composition Workflows Invocation ApplnService ApplnService “The Many Faces of IT as Service”, ACM Queue, Foster, Tuecke, 2005

  9. Grid Dynamics:Forming & Operating Communities • Define membership & roles; enforce laws & community standards • I.e., policy for service-oriented architecture • Addressing dynamic membership & policy • Build, buy, operate, & share infrastructure • Decouple consumer & provider • For data, programs, services, computing, storage, instruments • Address dynamics of community demand

  10. Grid Dynamics:Forming & Operating Communities • Define membership & roles; enforce laws & community standards • I.e., policy for service-oriented architecture • Addressing dynamic membership & policy • Build, buy, operate, & share infrastructure • Decouple consumer & provider • For data, programs, services, computing, storage, instruments • Address dynamics of community demand

  11. A B 1 1 10 10 1 A B 1 2 1 2 16 Defining Community: Membership and Laws • Identify VO participants and roles • For people and services • Specify and control actions of members • Empower members  delegation • Enforce restrictions  federate policy Effective Access Policy of site to community Access granted by community to user Site admission-control policies

  12. Policy Challenges in VOs • Restrict VO operations based on requestor characteristics • VO dynamics create challenges • Intra-VO • VO-specific roles • Mechanisms to specify/enforce VO-level policy • Inter-VO • Different VOs define different entities/roles • Different sorts of policy need to be enforced • Access, usage, accounting, audit, …

  13. Evolution of Grid Security & Policy 1) Grid security infrastructure • Public key authentication & delegation • Access control lists (“gridmap” files)  Limited set of policies can be expressed 2) Utilities to simplify operational use, e.g. • MyProxy: online credential repository • VOMS, ACL/gridmap management  Broader set of policies, but still ad-hoc 3) General, standards-based framework for authorization & attribute management

  14. Core Security Mechanisms • Attribute Assertions • C asserts that S has attribute A with value V • Authentication and digital signature • Allows signer to assert attributes • Delegation • C asserts that S can perform O on behalf of C • Attribute mapping • {A1, A2… An}vo1  {A’1, A’2… A’m}vo2 • Policy • Entity with attributes A asserted by C may perform operation O on resource R

  15. Security Services for VO Policy • Attribute Authority (ATA) • Issue signed attribute assertions (incl. identity, delegation & mapping) • Authorization Authority (AZA) • Decisions based on assertions & policy VOUser A Delegation Assertion User B can use Service A Resource Admin Attribute VO AZA VO ATA VO-A Attr  VO-B Attr Mapping ATA VO Member Attribute VOUser B VO Member Attribute VO A Service VO B Service

  16. SSL/WS-Security with Proxy Certificates Authz Callout: SAML, XACML Services (running on user’s behalf) Access ComputeCenter Rights CAS or VOMS issuing SAML or X.509 ACs Rights VO Local policy on VO identity or attribute authority Rights’ Closing the Loop:GT4 Security Toolkit Users MyProxy KCA Shib

  17. Grid Dynamics:Forming & Operating Communities • Define membership & roles; enforce laws & community standards • I.e., policy for service-oriented architecture • Addressing dynamics of membership & policy • Build, buy, operate, & share infrastructure • Decouple consumer & provider • For data, programs, services, computing, storage, instruments • Address dynamics of community demand

  18. Bootstrapping a VOby Assembling Services 1) Integrate services from other sources • Virtualize external services as VO services 2) Coordinate & compose • Create new services from existing ones Community Content Services Provider Services Capacity Provider Capacity “Service-Oriented Science”, Science, Foster, 2005

  19. Community A Community Z … Providing VO Services:(1) Integration from Other Sources • Negotiate servicelevel agreements • Delegate and deploy capabilities/services • Provision to deliver defined capability • Configure environment • Host layered functions

  20. Virtualizing Existing Services into a VO • Establish service agreement with service • E.g., WS-Agreement • Delegate use to VO user User B User A VO User VO Admin Existing Services

  21. Deploying New Services Policy Allocate/provision Configure Initiate activity Monitor activity Control activity Activity Client Environment Resource provider Interface WSRF (or WS-Transfer/WS-Man, etc.), Globus GRAM, Virtual Workspaces

  22. Activities Can Be Nested Client Policy Client Client Environment Resource provider Interface

  23. Embedded Resource Management:E.g., EGEE & OSG Client-side VO Admin Deleg Deleg GRAM GRAM Cluster Resource Manager Headnode Resource Manager VOUser VOUser Monitoring and control VO Job Deleg GRAM Cluster Resource Manager Other Services VO Scheduler . . . • VO admin delegates credentials to be used by downstream VO services. • VO admin starts the required services. • VO jobs comes in directly from the upstream VO Users • VO job gets forwarded to the appropriate resource using the VO credentials • Computational job started for VO VO Job

  24. Virtual Workspaces(Kate Keahey et al.) • GT4 service for the creation, monitoring, & management of virtual workspaces • High-level workspace description • WSRF mechanisms to monitor & manage • Multiple implementations • Dynamic accounts • Xen virtual machines • (VMware virtual machines) • … • Virtual clusters as a higher-level construct

  25. request use existing VM image deploy, suspend How do Grids and VMs Play Together? VM Factory create new VM image VM EPR Create VM image VM Repository inspect & manage Client Resource VM Manager VM start program

  26. OSG cluster Xen hypervisors TeraGrid cluster Virtual OSG Clusters OSG “Virtual Clusters for Grid Communities,” Zhang et al., CCGrid 2006

  27. Providing VO Services:(2) Coordination & Composition • Take a set of provisioned services … … & compose to synthesize new behaviors • This is traditional service composition • But must also be concerned with emergent behaviors, autonomous interactions • See the work of the agent & PlanetLab communities “Brain vs. Brawn: Why Grids and Agents Need Each Other," Foster, Kesselman, Jennings, 2004.

  28. Cardiff AEI/Golm The Globus-BasedLIGO Data Grid LIGO Gravitational Wave Observatory Birmingham• Replicating >1 Terabyte/day to 8 sites >40 million replicas so far MTBF = 1 month www.globus.org/solutions

  29. Data Replication Service • Pull “missing” files to a storage system Data Location Data Movement GridFTP Local ReplicaCatalog Replica LocationIndex Reliable File Transfer Service GridFTP Local Replica Catalog Replica LocationIndex Data Replication List of required Files Data Replication Service “Design and Implementation of a Data Replication Service Based on the Lightweight Data Replicator System,” Chervenak et al., 2005

  30. Deploy hypervisor/OS Hypervisor/OS Composing Resources …Composing Services LRC GridFTP GridFTP Deploy service DRS Deploy container VO Services JVM Deploy virtual machine VM VM Procure hardware Physical machine State exposed & access uniformly at all levels Provisioning, management, and monitoring at all levels

  31. Dynamic Service Deployment(Argonne + China Grid) • Interface • Upload-push • Upload-pull • Deploy • Undeploy • Reload “HAND: Highly Available Dynamic Deployment Infrastructure for GT4,” Li Qi et al., 2006

  32. “Provide access to data D at S1, S2, S3 with performance P” S1 S2 D ServiceProvider S3 Replica catalog, User-level multicast, … “Provide storage with performance P1, network with P2, …” S1 D S2 ResourceProvider S3 Decomposition EnablesSeparation of Concerns & Roles S1 User S2 D S3

  33. Community Commons • What capabilities are available to VO? • Membership changes, state changes • Require mechanisms to aggregate and update VO information MORE The age of information A A A VO-specific indexes S Information FRESH S S S

  34. adapter Custom protocols for non-WSRF entities Automated registration in container GridFTP GRAM User GT4 Monitoring and Discovery Services(Uniform Treatment of State is Wonderful!) Clients (e.g., WebMDS) GT4 Container WS-ServiceGroup MDS-Index Registration & WSRF/WSN Access GT4 Cont. GT4 Container MDS-Index MDS-Index RFT

  35. Decomposition Implementation Facilities Computers Storage Networks Services Software People U. Colorado Experimental Model UIUC Experimental Model COORD. NCSA ComputationalModel System-Level Problem Grid technology

  36. BI Server 2 Dispatcher Grid backend Grid-enabled Business Intelligence Application Provision New Worker Process ManagedPool of Shared Resources BI server applications started and decommissioned by a Grid-enabled dispatcher

  37. Multiple applications and workload types Coarse Grained Fine Grained Data Driven Workflow Dev / Test Consistent & open management interface GridInfrastructure End-to-end Quality of Service Consistent & open enactment interface Multiple resource types and instances The Integrating Role of Grid Infrastructure

  38. Summary: Grid Dynamics and You • Grid = dynamic behaviors & environments • Dynamic communities & activities • Decoupling of service consumption from service production • Dynamic provisioning of services • We have tools to realize dynamic scenarios • Uniform state representation & access • Flexible security & policy framework • Virtual machines, dynamic services, & other building blocks • We now need much experimentation

  39. For More Information • Globus Alliance • www.globus.org • Background • www.mcs.anl.gov/~foster Come to GT4 workshop,8:30-12:00 Wednesday • Overview of features • User experiences • Future directions 2nd Edition www.mkp.com/grid2

  40. Python Runtime C Runtime Java Runtime Available in High-Quality Open Source Software … Globus Toolkit v4 www.globus.org Data Replication CredentialMgmt Replica Location Grid Telecontrol Protocol Delegation Data Access & Integration Community Scheduling Framework WebMDS Reliable File Transfer CommunityAuthorization Workspace Management Trigger Authentication Authorization GridFTP Grid Resource Allocation & Management Index Security Data Mgmt Execution Mgmt Info Services CommonRuntime I. Foster, Globus Toolkit Version 4: Software for Service-Oriented Systems, LNCS 3779, 2-13, 2005

  41. User Applications GT4 & Web Services:Uniform State, Security, Mgmt Custom Services Custom WSRF Services GT4WSRF Web Services Registry & Admin GT4 Container(e.g., Apache Axis) WS-A, WSRF, WS-Notification WSDL, SOAP, WS-Security

  42. http://dev.globus.org GlobDev Guidelines(Apache) Infrastructure(CVS, email,bugzilla, Wiki) Projects Include …

More Related