The Globus Toolkit: Description and Applications Review

1. The Globus Toolkit:Description and Applications Review Steve Tuecke & Ian Foster Argonne National Laboratory The University of Chicago Globus Co-PI: Carl Kesselman, USC/ISI

2. Overview • The need for Grid services • The Globus toolkit • Globus application case studies • Microtomography: on-line instrumentation • SF-Express and Overflow: distributed supercomputing • CAVERNsoft: collaborative engineering • Nimrod-G: high-throughput computing • ECCE’: problem solving environment • Summary

3. Creating a Usable Grid :Grid Services (“Middleware”) • Standard grid services that • Provide uniform, high-level access to a wide range of resources (including networks) • Address interdomain issues of security, policy, etc. • Permit application-level management and monitoring of end-to-end performance • Middleware-level and higher-level APIs and tools targeted at application programmers • Map between application and Grid

4. Distributed computing Collab. design Remote control Application Toolkit Layer Data- intensive Remote viz Information Resource mgmt . . . Grid Services Layer Security Data access Fault detection Grid Services Architecture High-energy physics data analysis Collaborative engineering On-line instrumentation Applications Regional climate studies Parameter studies Grid Fabric Layer Transport . . . Multicast Instrumentation Control interfaces QoS mechanisms

5. The Globus Project: Argonne, USC/ISI, NCSA, Aerospace, NASA Ames, LBNL, others • Basic research in grid-related technologies • Resource management, security, adaptation, etc. • Development of Globus toolkit • Core services for grid-enabled tools & applns • Construction of large grid testbed: GUSTO • Largest grid testbed in terms of sites & apps • Application experiments • Tele-immersion, distributed computing, etc.

6. GUSTO Testbed Map

7. Globus Grid Services • The Globus toolkit provides a range of basic Grid services • Security, information, fault detection, communication, resource management, ... • These services are simple and orthogonal • Can be used independently, mix and match • Programming model independent • For each there are well-defined APIs • Standards are used extensively • E.g., LDAP, GSS-API, X.509, ...

8. Grid Services Layer (1) • Grid Security Infrastructure • Single-sign on, run anywhere [if authorized] • PKI, X.509 certificates • Identity/credential mapping at each resource • Allows programs to act as user for limited period: delegation of rights

9. Grid Services Layer (2) • Grid Information Service • Currently an LDAP-based directory service • Publish structure and state info, dynamic performance info, software info, etc., etc. • Resource discovery: “find me an X with property Y available at time T” • Auto-configuration: “tell me what I need to know to use A efficiently/securely/...” • Gateways to other data sources required • Example of integrating “middleware” service

10. Grid Services Layer (3) • Access to remote data (GASS) • Uniform access to diverse storage management systems • Cache management • Integration with SRB, DPSS, HPSS • Communication (Nexus) • Application-level interfaces to comm services • Multiple methods: reliable/unreliable, IP/other, unicast/multicast • QoS interfaces

11. Grid Services Layer (4) • Globus Resource Allocation Manager (GRAM) • Uniform interface to resource management • Globus Arch. for Reservation and Allocation • Co-allocation of compute resources • Immediate and advance reservation of network and computers in prototype form • Fault detection service • Network measurement tools • Code management and distribution infrastructure

12. Application Toolkit Layer: e.g. • Message Passing Interface • Multi-method communication, specialized • CAVERNsoft • Shared state for collaborative environments • Condor, Nimrod-G • High-throughput computing • Parallel Application Workspace (PAWS) • High-speed parallel transfers for coupled apps

13. Globus Progress • Selected “Grid Services” are being migrated into the infrastructure • Grid information service • Grid security infrastructure • Grid resource management services • Simultaneously these and other Globus services are being applied to develop • Grid-enabled tools • Grid-enabled applications • An ongoing iterative refinement process

14. Case Study 1:Online Instrumentation Advanced Photon Source wide-area dissemination desktop & VR clients with shared controls real-time collection archival storage tomographic reconstruction DOE X-ray source grand challenge: ANL, USC/ISI, NIST, U.Chicago

15. CMT Processing Now

16. Additional Opportunities • End-to-end advance reservation of network, storage, computers • Dynamic discovery and allocation of supercomputers, networks, etc. • Adaptive determination of display resolution, reconstruction fidelity, etc., etc. • Reliable multicast for data, control, video • Access control and discovery for collaborative sessions • Integration with mass storage systems

17. Case Study 2:Distributed Supercomputing • Starting point: SF-Express parallel simulation code • Globus mechanisms for • Resource allocation • Distributed startup • I/O and configuration • Fault detection • 100K vehicles (2002 goal) using 13 computers, 1386 nodes, 9 sites NCSA Origin Caltech Exemplar CEWES SP Maui SP SF-Express Distributed Interactive Simulation: Caltech, USC/ISI

18. OVERFLOW with latency-tolerant algorithms MPICH-G “Grid-enabled” message passing Globus services Security Directory Scheduling Process mgmt Communication ARC SGI O2000 (California) Argonne SGI O2000 (Illinois) OVERFLOW simulation: NASA Ames

19. Case Study 3:Collaborative Engineering • Manipulate shared virtual space, with • Simulation components • Multiple flows: Control, Text, Video, Audio, Database, Simulation, Tracking, Haptics, Rendering • Uses Globus comms: (un)reliable uni/multicast • Future: Security, QoS, allocation, reservation CAVERNsoft: UIC Electronic Visualization Laboratory

20. Case Study 4:High-Throughput Computing • Schedule many independent tasks (e.g., parameter study) • Uses Globus security, discovery, data access, scheduling • Future: Reservation, accounting, code management, etc. Deadline Cost Available Machines Nimrod-G: Monash University

21. Case Study 5:Problem Solving Environment • Problem solving environment for comp. chemistry • Globus services used for authentication, remote job submission, monitoring, and control • Future: distributed data archive, resource discovery, charging ECCE’: Pacific Northwest National Laboratory

22. Summary • Grids require Grid services that make resources accessible and usable and Grid toolkits for application development • The Globus project is building essential services and partnering with tool developers • Significant success stories in a range of problem classes • We’re looking forward to working with applications throughout the community!