1 / 39

The Grid

The Grid. 2006.11.8. 분산시스템연구실 최형준. Table of Contents. Introduction to Grid Scientists ’ Dream What is Grid How does it work More about Grid OGSA, WSRF, GT4 VM and the Grid Implementing the Grid on Classical Virtual Machine Systems. Introduction.

keith-jensen
Download Presentation

The Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Grid 2006.11.8. 분산시스템연구실 최형준

  2. Table of Contents • Introduction to Grid • Scientists’ Dream • What is Grid • How does it work • More about Grid • OGSA, WSRF, GT4 • VM and the Grid • Implementing the Grid on Classical Virtual Machine Systems DCSLab

  3. Introduction • Rapid increase in total amount of computation power • Qualitywise (Moore’s Law) • Quantitywise (Reduced cost, wide availability of easy-to-use programs) • Underutilization of compute power! • Waste of energy + resource DCSLab

  4. Scientists’ Need • Scientists, on the other hand, considerably lack in computation power… • Scientists face increasingly complicated problems, which require heavy computing power • Weather prediction • Development of new drugs • Earth science • High energy physics DCSLab

  5. Scientists’ Need Computers can not keep up with what scientists demand of them!!! Difficult, expensive… and sometimes impossible to achieve certain scientific goals • Example of compute intensive application: CEPAR - 3D protein structure comparison • Similar structure  Similar function • 30,000 proteins in Protein Data Bank • 30 sec to compare each pair on a typical desktop processor • 30,000*30,000/2*30 sec = 428 CPU years on one processor DCSLab

  6. LCG/EGEE • “Large Hadron Collider Computing Grid and Enabling Grid for e-Science in Europe Project” • 27km of circumference • Largest machine ever built by human! • Will generate 15 Petabytes of data per year • Requires 10,000 PCs • CERN only has 1,000 computers and 1 Petabyte of storage • Impossible to process and store this much in a single site • Grid computing is the only solution DCSLab

  7. Scientists’ Dream • I want infinite storage space… • I want infinite computing power… • I want to share data, results, resources with my colleagues… • But how?! • The dream… DCSLab

  8. Solution? • Computer Rentals? • Analogous to vehicle rental model… • Short-term rental of computers fell out of favor Increased complexity of configuring S/W DCSLab

  9. Emergence of New Computing Environment • Emergence of ubiquitous connectivity & Internet • Emerging model for the future • Seek resources over a communication network • Have the task performed remotely • “Computation world is in a state analogous to the electricity world in 1910” – Foster, Kesselman • Utility computing • A business model whereby computer resources are provided on an on-demand and pay-per-use basis DCSLab

  10. What is Grid? Desktop PCs Workstations Data Storage Super-computers World Wide Web Internet Electron-microscopes Satellites Meteorological Sensors DCSLab

  11. What is Grid? • Computers and instruments connected to the Grid share not only information… but also computing power and resources like disk storage and S/W applications • Grid is a service for sharing computer power and data storage capacity over the Internet GRID DCSLab

  12. What is Grid? • The Grid goes well beyond simple communication between computers, and aims ultimately to turn the global network of computers into one vast computational resource • The reality is that today, the Grid is a "work in progress", with the underlying technology still in a prototype phase DCSLab

  13. What is Grid? • Evolution of computing that led to the concept of Grid • Distributed computing • Metacomputing (popular in the 90’s) • Cluster Computing • P2P • The Grid was born at a workshop at Argonne National Laboratory in September 1997 DCSLab

  14. What is Grid? DCSLab

  15. What is Grid? • Using Grid computing, resources from several different organizations are pooled into virtual organizations (VO) to solve specific problems • Grid computing extends the virtual machine concept to “organizational-level virtual machines” VO (2) VO (1) A C B DCSLab

  16. Why scientists need Grid • Grid computing is all about achieving greater performance by pooling together resources from different organizations • Scientists need Grid because it provides the following… • Infinite storage space • Infinite computing power • Possibility to collaborate with distant colleagues, sharing resources, data, and results DCSLab

  17. How does Grid work? • The big picture Registration Submit Job Step 1 Join Virtual Organization Step 4 Create proxy credential Step 2 Get Certificate Step 5 Write job description file (JDL file) Step 3 Install Certificate Step 6 Submit JDL file to Resource Broker Step 7 Check status and get result DCSLab

  18. How does Grid work? • Sample of job description file • <?xml version="1.0" encoding="utf-8" ?> • <job xmlns="http://www.globus.org/namespaces/2004/10/gram/job/description"> • <executable>C:\\Bin\dotNetGridFTPClient.exe</executable> • <argument>-cert</argument> • <argument>CN=a, OU=snu.ac.kr, OU=simpleCA.snu.ac.kr, O=Grid</argument> • <argument>-batch</argument> • <argument>file:///C:\GridFTPRoot\GRenderProject\test.pre</argument> • <argument>gsiftp://147.46.xxx.xxx/GRenderProject/test.pre</argument> • <jobType>single</jobType> • <stdout>C:\GridFTPRoot\dotnetgridftp.exe.txt</stdout> • </job> DCSLab

  19. How does Grid work? • Resource Broker queries Information Service and Replica Catalog to decide which resources to use DCSLab

  20. How does Grid work? • Once the appropriate resources have been located, the job is executed on the remote site DCSLab

  21. How does Grid work? • When the job is finished, the computing element transfers output to the Resource Broker, which in turn passes it back to the user DCSLab

  22. Things to Consider… • How do we decide what resources are part of each virtual organization • How do we decide which resources will be allocated to a task • How do we get resources to communicate among themselves • How do we split up a task so that it can be performed in parallel • Grid computing enables us to access heterogeneous resources from different organizations by providing a set of protocols, technologies, and methodologies that provide an answer to these questions DCSLab

  23. Important Remarks • Grid systems involve heterogeneous resources from different organizations • Coordinating all the resources involves a lot of different services such as job schedulers, resource managers, information services, etc DCSLab

  24. OGSA, WSRF, and GT4 • Common services in a grid system • VO Management Service • Resource Discovery and Management Service • Job Management Service • Security & Data Management etc… • Solution to successful coordination of these services: “Standardization” • The Open Grid Services Architecture (OGSA) • Developed by Global Grid Forum to define a common, standard, and open architecture for grid-based applications DCSLab

  25. OGSA, WSRF, and GT4 • We need a middleware on which to base the OGSA  Web Services • Web Services is yet another distributed computing technology like CORBA and RMI that allows us to create client/server applications DCSLab

  26. OGSA, WSRF, and GT4 • Why Web Service? • Advantages of Web Services • Platform independent • Language independent • Loosely coupled DCSLab

  27. OGSA, WSRF, and GT4 • Web Services is usually stateless • But OGSA requires “stateful services” • Web Services Resource Framework (WSRF) • WSRF specifies how to make Web Services stateful Statefulness Ability to remember information, keep state, from one invocation to another DCSLab

  28. OGSA, WSRF, and GT4 • A stateful Web Service invocation • Keep a state in a separate entity called a “resource” DCSLab

  29. OGSA, WSRF, and GT4 • How do we address a resource? • Use WS-Resource-qualified endpoint reference Endpoint Reference Uri: http://www.example.com/myservice ResourceID: 0x09EB23FA DCSLab

  30. OGSA, WSRF, and GT4 DCSLab

  31. OGSA, WSRF, and GT4 • Globus Toolkit 4 (GT4) • A open source toolkit, developed by The Alliance, which we can use to create Grid systems • Realization of the OGSA requirements • De facto standard for the Grid community • Includes high-level services, programming libraries and tools that we can use to build Grid applications • Security • Data Management • Execution Management • Information Services • Common Runtime DCSLab

  32. OGSA, WSRF, and GT4 • GT4 services Data Replication CredentialMgmt Replica Location Grid Telecontrol Protocol Delegation Data Access & Integration Community Scheduling Framework WebMDS Python Runtime Reliable File Transfer CommunityAuthorization Workspace Management Trigger C Runtime Authentication Authorization GridFTP Grid Resource Allocation & Management Index Java Runtime Security Data Mgmt Execution Mgmt Info Services CommonRuntime DCSLab

  33. OGSA, WSRF, and GT4 DCSLab

  34. OGSA, WSRF, and GT4 • Layered diagram of OGSA, GT4, WSRF, and Web Services Applications OGSA Web Services WSRF DCSLab

  35. Implementing the Grid on Classical Virtual Machine systems • Change the level of abstractionof jobs from the user process level to the full machine • The unit of work with which the grid middleware deals becomes a system virtual machine rather than a process DCSLab

  36. Implementing the Grid on Classical Virtual Machine systems • User isolation • Ensures greater system integrity • Platform independence • Not necessary for there to be a physical system that matches the exact configuration specified by the user • Task management and accounting • Resource control at a higher level of granularity; simplifies job allocation process • Portability • Allows an application to run on any platform that has a VMM specified by the application DCSLab

  37. Implementing the Grid on Classical Virtual Machine systems • Three important aspects of a task in an system VM-based grid • Ability to capture the state of a VM • Ability to instantiate a VM • Ability to save user data DCSLab

  38. User request 2: query (data, image, compute server) 7: Login session window, 5: copy/access user data isolation 1: user request 3: setup VM image 6: return handler to user (URL) Y V2 X V1 4: start VM Implementing the Grid on Classical Virtual Machine systems • Scenario from Figueiredo, Dinda, and Fortes (2003) Information service User ‘Y’ User ‘X’ Front end ‘F’ Physical server pool P Image Server I Data Server D2 Data Server D1 DCSLab

  39. Conclusion • Skeptical Views on Grid • Just an elaborate marketing strategy • Media hype • Too much expectations and promises that can never be fulfilled • Trust/security issues that are inherently difficult to solve (VO involves human interaction) DCSLab

More Related