The Grid: Past, Present, and Future

1. The Grid:Past, Present, and Future Tom DeFanti University of Illinois at Chicago as Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago

2. http:// Web: Uniform access to HTML documents http:// Software catalogs Computers Sensor nets Colleagues Data archives The Grid: The Web on Steroids Grid: Flexible, high-perf access to all significant resources On-demand creation of powerful virtual computing systems

3. Why Now? • The Internet as infrastructure • Increasing bandwidth, advanced services • Advances in storage capacity • You can buy a Terabyte for < $15,000 • Increased availability of compute resources • Clusters, supercomputers, etc. • Advances in application concepts • Simulation-based design, advanced scientific instruments, collaborative engineering, ...

4. Talk in a Nutshell … • Where this all started • Special purpose networks, stunts • Where we are • Grid services as unifying concept • Limited but growing application set • Persistent testbeds • Where we’re going • Production grids • New user communities and applications • Convergence of Grid and Info Infrastructure • High-level services and development tools

5. Disclaimers • Describing work of many people • ANL, ISI, NCSA, CIT, SDSC, NASA, EVL, etc. • In particular, Carl Kesselman, Globus Co-PI • In U.S., Europe, Asia, Australia • Much important work by others is left out • I have picked just a representative sample

6. Where This All Started

7. High-Performance Networks:E.g., CASA Gigabit Testbed • One of four U.S. testbeds in early 90s • Dedicated network • HIPPI over SONET, custom hardware • Caltech, SDSC, LANL • Focus on distributed supercomputing applications • Large distributed memory & vector supercomputers CASA: Paul Messina et al.

8. The I-Way @ SC’95 Conference • Focus on application demonstrations • 60+ groups • OC-3 backbone • Large-scale use of immersive displays • CAVE and I-Desk • I-Soft programming environment • Pioneered security, scheduling ideas I-WAY leaders: Tom DeFanti, Tim Kuhfuss, Rick Stevens, Maxine Brown

9. Resource 1 ATM Switch ATM Resource N I-POP Internet AFS Kerberos Scheduler Possible Firewall The I-Soft SoftwareEnvironment • Kerberos authentication • I-POP initiated rsh to local resources • Andrew File System to distribute software & state • Central scheduler • Dedicated I-WAY nodes on resource • Interface to local scheduler • Nexus-based comm libraries • MPI, CAVEcomm, CC++ I-Soft development: Warren Smith, Jonathon Geisler, Steve Tuecke

10. Combustion System Modeling • A shared collaborative space • Link people at multiple locations • Share and steer scientific simulations on supercomputer • Prototypical of new applns • Traditional HPC an element of a more complex whole • Substantial development effort • Yet insecure, no QoS Chicago San Diego DC Lori Freitag et al., Argonne National Laboratory and NALCO

11. Where We Are Now

12. Collaboration Tools Data Mgmt Tools Distributed simulation . . . Information services Resource mgmt Fault detection . . . Remote access Remote monitor The Need for Grid Services net

13. The Grid Services Concept • Standard services that • Provide uniform, high-level access to a wide range of resources (including networks) • Address interdomain issues: security, policy • Permit application-level management and monitoring of end-to-end performance • Broadly deployed, like Internet Protocols • Enabler of application-specific tools as well as applications themselves • Integrate across domains to support the creation of scalable virtual organizations

14. The Globus Project: Developinga Grid Services Architecture • Developed as a key enabling mechanism the Grid Security Infrastructure • Uniform authentication & authorization mechanisms in multi-institutional setting • Single sign-on, delegation, identity mapping • Public key technology, SSL, X.509, GSS-API • Used to construct Grid resource managers that provide secure remote access to • Computers: GRAM server (HTTP), secure shell • Storage: storage resource managers, GSIFTP • Networks: differentiated service mechanisms Globus project Co-PI: Carl Kesselman

15. Developing a Grid Services Architecture • Developed as a 2nd key enabling mechanism the Grid Information Service (aka MDS) • Lightweight Directory Access Protocol (LDAP) used to access info about specific resources • Index servers provide for flexible indexing • Uniform resource discovery & characterization mechanisms in highly distributed systems • A basis for resource brokering, adaptation, autoconfiguration • Other services include fault detection and communication GIS: Steve Tuecke, Steve Fitzgerald, Gregor von Laszewski

16. Status of Grid Services • Core Grid services & others (e.g., Network Weather Service) have been deployed in large-scale testbeds • Availability of these services is enabling tool & application development projects • Major challenges certainly remain: e.g. • Advance reservation, policy, accounting • End-to-end application adaptation (events?) • Integration with commodity technology • Grid Forum: http://www.gridforum.org

17. Examples of Grid-Enabled Tools • Message Passing • MPICH-G2 • Distributed collaboration • CAVERNsoft, Access Grid • High-throughput computing • Condor-G, Nimrod-G • Distributed data management & analysis • Data Grid toolkits • Desktop access to Grid resources • Commodity Grid Toolkits (CoG Kits)

18. Online Instrumentation Advanced Photon Source wide-area dissemination desktop & VR clients with shared controls real-time collection archival storage tomographic reconstruction DOE X-ray grand challenge: ANL, USC/ISI, NIST, U.Chicago

19. Distributed Supercomputing • Starting point: SF-Express parallel simulation code • Globus mechanisms for • Resource allocation • Distributed startup • I/O and configuration • Fault detection • 100K vehicles using 13 computers, 1386 nodes, 9 sites (5 years early!) NCSA Origin Caltech Exemplar CEWES SP Maui SP SF-Express Distributed Interactive Simulation: Caltech, USC/ISI

20. Problem Solving Environments • Problem solving environment for comp. chemistry • Globus services used for authentication, remote job submission, monitoring, and control • Future: distributed data archive, resource discovery, charging ECCE’: Pacific Northwest National Laboratory

21. Where We Are Going

22. NSF National Technology Grid EmergingProductionGrids NASA Information Power Grid

23. Emerging User Communities • NSF Network for Earthquake Engineering Simulation (NEES) • Integrated instrumentation, collaboration, simulation • Grid Physics Network (GriPhyN) • ATLAS, CMS, LIGO, SDSS • World-wide distributed analysis of Petascale data • Access Grid: supporting group based collaboration

24. Today’s Information Infrastructure • Network-centric: simple, fixed end systems; few embedded capabilities; few services; no user-level quality of service O(106) nodes

25. Tomorrow’s Information Infrastructure:Not Just “Faster and More Reliable” • Application-centric: heterogeneous, mobile end-systems; many embedded capabilities; rich services; user-level quality of service O(109) nodes Caching Resource Discovery QoS

26. UC MREN Purdue IIT UIPUI UIC STAR LIGHT IU Bloom-ington ANL Franklin CO UIUC Optical MREN (OM) AADS switch OC-3 Bell Nexxia to CA*net4 BN OC-3 OC-3 (2) OC-3 NU Evanston 1 pr GigE 2 pr NU Chicago ? 6 pr GigE 3 pr Optical STAR TAP (AUP free) 3 pr I-WIRE Central Office

27. Topology I-WIRE: Computing Argonne Chicago • Research Areas • Latency-Tolerant Algorithms • Interaction of SAN/LAN/WAN technologies • Cluster Architectures Urbana

28. Scalable Global Services:The InterGrid • Do for the Grid what routers do for the network • Wide variety of services • Resource trading • Policy enforcement • Agent based scheduling • Agent based monitoring • Virtual data • Exploit emerging network services, e.g. QoS Scheduling Fault Recovery Data Caching Routing

29. “Megacomputing” (Larry Smarr) • Very large numbers of computers • Opportunistic, dynamic behaviors • Bandwidths variable, ultimately to Gb/s • SETI@home, distributed.net, entropia.com

30. The Emergence of theApplication Service Provider • ASP model: “rent” access to applications that execute on remote computers • Decouples interface, application, computing, hardware • Each can be optimized & sold separately • Highly disruptive impact on all aspects of the computer industry • May also be key to a dramatic increase in the penetration of HPC • E.g., see NetSolve (U.Tenn), PUNCH (Purdue)

31. Summary • Grids will change the way we do science and engineering • Key services and concepts have been identified, although major challenges remain • Transition of services and applications to production use is starting to occur • Future will see increased sophistication and scope of services, tools, and applications • Potentially disruptive trends: ASPs, megacomputing

32. For More Information • Book published by Morgan Kaufman • www.mkp.com/grids • Globus • www.globus.org • Grid Forum • www.gridforum.org

33. We are Hungry for the Grid www.evl.uic.edu