Grid Technology: The Rough Guide

1. Grid Technology: The Rough Guide Ashok Adiga (TACC) & Victor Bolet (Georgia State University)

2. Outline • Introduction • Workshop Objectives • Grids 101 • Grids in action • Workshop Outline • Open Discussions

3. Introduction • Workshop announcements • Wireless access available • Dinner on Thursday • Ride back to hotel after Workshop on Friday? • Introductions • Speaker Introductions • Attendee introductions • What would you like to get out of this workshop?

4. Workshop Objectives • Grid Technology is gaining acceptance • Still evolving; but many success stories • New middleware/tools are becoming available all the time • Target audience is technical people who are interested in building grids but not sure how best to begin • Objective is to provide an overview of current grid technology landscape via • Presentations & demonstrations • Hands-on session building “classroom grid” • Detailed coverage of selected topics, overview of other topics with links to details

5. What is a Grid? “Resource sharing & coordinated problem solving in dynamic … virtual organizations” • Enable integration of distributed service & resources • Using general-purpose protocols & infrastructure • To achieve useful qualities of service “The Anatomy of the Grid”, Foster, Kesselman, Tuecke, 2001

6. What is a Grid? (cont’d) • Integrate and coordinate resources not subject to centralized control … • addresses the issues of security, policy, payment, membership, etc. • … using standard, open, general-purpose protocols and interfaces … • for authentication, authorization, resource discovery, and resource access. • … to deliver nontrivial qualities of service. • relating to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands,

7. DB Access portal App Scheduler integration PDB portal PSE Grid Technologies for Resource Integration & Management interoperability Grid Applications User-level Middleware and Tools System-level Common Infrastructure Grid Resources

8. DB Federation Chem portal App Scheduler PDB portal PSE Grid Services Architecture • Agree on interfaces, services • Common infrastructure services act like a “grid OS” • Users interact with the Grid through higher-level, user-friendly middleware layer Grid Applications User-focusedmiddleware & tools (commercial opportunities) Authentication, information, resource access, resource mgmt, negotiation, scheduling, monitoring, data transfer, etc., etc. Common infrastructureservices (many open source) Grid Resources

9. Available Grid Technologies • Several Grid solutions & toolkits exist • Globus Toolkit (www.globus.org) • A toolkit of grid services and tools jointly developed by the Globus Alliance in the US. • Gridlab (www.gridlab.org) • A set of application-oriented Grid services and toolkits jointly developed by academic & commercial companies primarily based in Europe • Gridbus (www.gridbus) • Grid computing and Business technologies developed at the University of Melbourne. • Unicore (www.unicore.org) • Grid solution created by the Unicore Forum, a developed by leading European Computing Centers and supporting hardware vendors • Open Middleware Infrastructure Institute (OMII) (www.omii.ac.uk) • UK e-Science Grid solution • Condor (www.cs.wisc.edu/condor) • Complete grid solution developed at the University of Wisconsin • Grid MP (www.ud.com) • Commercial grid solution developed by United Devices • We will primarily discuss the use of the Globus Toolkit & Condor

10. What Is the Globus Toolkit? • A Grid development environment • Develop new OGSA-compliant Web Services • Develop applications using Java or C/C++ Grid APIs • Secure applications using basic security mechanisms • A set of basic Grid services • Job submission/management • File transfer (individual, queued) • Database access • Data management (replication, metadata) • Monitoring/Indexing system information • Tools and Examples • The prerequisite for many Grid community tools

11. “Standard Plumbing” for the Grid • Not turnkey solutions, but building blocks and tools for application developers and system integrators. • Some components (e.g., file transfer) go farther than others (e.g., remote job submission) toward end-user relevance. • Since these solutions exist and others are already using them (and they’re free), it’s easier to reuse than to reinvent. • And compatibility with other Grid systems comes for free!

12. G T 4 Delegation Service Community Scheduler Framework [contribution] Python WS Core [contribution] C WS Core G T 3 CAS OGSA-DAI [Tech Preview] Web ServicesComponents WS Authentication Authorization Reliable File Transfer (RFT) Java WS Core Grid Resource Allocation Mgmt (WS GRAM) Monitoring & Discovery System (MDS4) G T 2 Pre-WS Authentication Authorization GridFTP Grid Resource Allocation Mgmt (Pre-WS GRAM) Monitoring & Discovery System (MDS2) C Common Libraries Non-WS Components G T 3 Replica Location Service (RLS) XIO G T 4 Credential Management Security Data Mgmt Execution Mgmt Information Services CommonRuntime Globus Toolkit Components

13. Areas of Competence • “Connectivity Layer” Solutions • Service Management (WSRF) • Monitoring/Discovery (WSRF and MDS) • Security (GSI and WS-Security) • Communication (XIO) • “Resource Layer” Solutions • Computing / Processing Power (GRAM) • Data Access/Movement (GridFTP, OGSA-DAI) • In development: Telecontrol (NTCP/GTCP) • “Collective Layer” Solutions • Data Management (RLS, MCS, OGSA-DAI) • Monitoring/Discovery (MDS) • Security (CAS)

14. Evolution of the Grid App-specific Services Open Grid Services Arch Web services Increased functionality, standardization GGF: OGSI, WSRF, … (leveraging OASIS, W3C, IETF) Multiple implementations, including Globus Toolkit X.509, LDAP, FTP, … Globus Toolkit Defacto standards GGF: GridFTP, GSI (leveraging IETF) Custom solutions Time

15. Standards: Open Grid Services Architecture • Define a service-oriented architecture… • the key to effective virtualization • …to address vital Grid requirements • AKA utility, on-demand, system management, collaborative computing, etc. • …building on Web service standards. • extending those standards when needed

16. Standards Compliance • Web services: WS-I compliance • All interfaces support WS-I Basic Profile, modulo use of WS-Addressing • Security a) WS-I Basic Security Profile (plaintext) b) IETF RFC 3820 Proxy Certificate • GridFTP • GGF GFD 020 • Others in progress & being tracked • WSRF (OASIS), WS-Addressing (W3C), OGSA-DAI (GGF), RLS (GGF)

17. Grid and Web Services Convergence The definition of WSRF means that the Grid and Web services communities can move forward on a common base.

18. Grid Planning Considerations • All Grid technology is evolving rapidly. • Web services standards • Grid interfaces • Grid implementations • Grid hosting services (ASP, SSP, etc.) • Community is important! • Best practices (GGF, OASIS, etc.) • Open source (Linux, Axis, Globus, etc.) • Applying community standards is vital. • Increases leverage • Mitigates (a bit) effects of rapid evolution • Paves the way for future integration/partnership

19. What End Users Need Secure, reliable, on-demand access to data, software, people, and other resources (ideally all via a Web Browser!)

20. How it Really Happens ComputeServer SimulationTool ComputeServer WebBrowser WebPortal RegistrationService Camera TelepresenceMonitor DataViewerTool Camera Database service ChatTool DataCatalog Database service CredentialRepository Database service Certificate authority Users work with client applications Application services organize VOs & enable access to other services Collective services aggregate &/or virtualize resources Resources implement standard access & management interfaces

21. How it Really Happens • Implementations are provided by a mix of • Application-specific code • “Off the shelf” tools and services • Tools and services from the Globus Toolkit • Tools and services from the Grid community (compatible with GT) • Glued together by… • Application development • System integration

22. How it Really Happens (without the Grid) A ComputeServer SimulationTool B ComputeServer WebBrowser WebPortal RegistrationService Camera TelepresenceMonitor DataViewerTool Camera C Database service ChatTool DataCatalog D Database service CredentialRepository E Database service Certificate authority Users work with client applications Application services organize VOs & enable access to other services Collective services aggregate &/or virtualize resources Resources implement standard access & management interfaces

23. How it Really Happens (with the Grid) GlobusGRAM ComputeServer SimulationTool GlobusGRAM ComputeServer WebBrowser CHEF Globus IndexService Camera TelepresenceMonitor DataViewerTool Camera GlobusDAI Database service CHEF ChatTeamlet GlobusMCS/RLS GlobusDAI Database service MyProxy GlobusDAI Database service CertificateAuthority Users work with client applications Application services organize VOs & enable access to other services Collective services aggregate &/or virtualize resources Resources implement standard access & management interfaces

24. Examples of Grids in Action • Campus level grid projects • UT Grid (University of Texas) • MGrid (University of Michigan) • … • Regional initiatives • SURAgrid • National/International Grids • NSF Teragrid • OSG (Open Sciences Grid) • EGEE (Enabling Grids for E-Science in Europe) • Volunteer grids • SETI@home • World Community Grid • Grid.org

25. NSF TeraGrid • 40+ teraflops compute • 1+ petabyte online storage • 10-40Gb/s networking • Heterogeneous compute, storage, visualization resources • Globus-based grid middleware • Any US researcher can apply for allocations • http://www.teragrid.org for more information

26. Creating a unified user environment… Single user support resources. Single authentication point Common software functionality Common job management infrastructure Globally-accessible data storage …across heterogeneous resources 7+ computing architectures 5+ visualization resources diverse storage technologies Create a unified national HPC infrastructure that is both heterogeneous and extensible The TeraGrid Strategy

27. Current TeraGrid Usage Scenarios • “Traditional” massively parallel jobs • Tightly-coupled interprocessor communication • storing vast amounts of data remotely • remote visualization • Thousands of independent jobs • Automatically scheduled amongst many TeraGrid machines • Use data from a distributed data collection • Multi-site parallel jobs • Compute upon many TeraGrid sites simultaneously TeraGrid is working to enable more!

28. SURAgrid • A “beyond regional” initiative in support of SURA regional strategy “Mini-About” SURA • SURA region: 16 states & DC, from Delaware to Texas • SURA membership: 62 research universities, mostly within the region • SURA mission: Foster excellence in scientific research, strengthen capabilities, provide training opportunities • Evolved from the NMI Testbed Grid project, an outgrowth of SURA’s management of the NMI Integration Testbed Program • http://www1.sura.org/3000/NMI-Testbed.html

29. SURAgrid Goals SURAgrid: Organizations collaborating to bring grids to the level of seamless, shared infrastructure Goals: • To develop scalable infrastructure that leverages local institutional identity and authorization while managing access to shared resources • To promote the use of this infrastructure for the broad research and education community • To provide a forum for participants to gain additional experience with grid technology, and participate in collaborative project development

30. University of Alabama at Birmingham* University of Alabama in Huntsville* University of Arkansas* University of Florida* George Mason University* Georgia State University* Great Plains Network University of Kentucky* University of Louisiana at Lafayette* Louisiana State University* University of Michigan Mississippi Center for SuperComputing Research* University of North Carolina, Charlotte North Carolina State University* Old Dominion University* University of South Carolina* University of Southern California Southeastern Universities Research Association (SURA) Texas A&M University* Texas Advanced Computing Center (TACC)* Texas Tech Tulane University* Vanderbilt University* University of Virginia* SURAgrid Participants *SURA member

31. SURAgrid Resources

32. SURAgrid Applications • Multiple Genome Alignment (GSU, UAB, UVA) • Task Farming (LSU) • Muon Detector Grid (GSU) • BLAST (UAB) • ENDYNE (TTU) • SCOOP/ADCIRC (UNC, RENCI, MCNC, SCOOP partners, SURAgrid partners) • … Potential applications…

33. Workshop Agenda • Introduction & Overview • Configuring Resources for the Grid • Authentication, Authorization, & Identity Issues in Grids • Level Grid Services • High Level Grid Services • Grid Packages • User Interfaces • Grid Application Toolkits • Hands-on Session

34. Hands-on Session • Build Classroom Grid during workshop • Using some of the technologies described in the presentations • Grid services pre-installed or installed during sessions • Laptop “servers” installed & configured during hands-on session • Try out some basic grid operations on this grid • Security, data & job management, resource monitoring, grid portals

35. Questions?