1 / 57

Grid Computing in the Industry

Grid Computing in the Industry. North Carolina Statewide Grid. Duke University, Nov 2, 2004. Wolfgang Gentzsch Managing Director MCNC Grid Computing and Networking Services

foy
Download Presentation

Grid Computing in the Industry

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Computing in the Industry North Carolina Statewide Grid Duke University, Nov 2, 2004 Wolfgang Gentzsch Managing Director MCNC Grid Computing and Networking Services This Lecture aims at complementing your background in distributed computing technologies and developments, providing an actual overview of Grid Computing, and its applications and benefits for Education, Industry and Economy.

  2. My current Background: MCNC & The Grid Infrastructure for North Carolina NC Statewide Grid Initiative MCNC Enterprise Grid • MCNC • Microelectronics Center of North Carolina • Founded in 1980 • Independent, private, non-profit organization • Operates NCREN since 1984 • Past: Supercomputer Center for universities • Now: Grid Service Provider, offering Video, • Network, Grid, and Datacenter Services • 50+ employees NC BioGrid NCREN North Carolina Research & Education Network

  3. The Industry’s View of a Grid : An IT Utility on a . . . Grid Middleware (the glue) Managing . . . Networked Distributed Resources

  4. Why Should We Care about Grids ? “ It’s the next big thing ! “ Grid technologies advance Science and Education in that we can do things which haven’t been possible before. Grid infrastructure attracts and enables new businesses and creates new jobs, especially in today's rural areas. Grids make us more competitive by better utilizing resources, bringing results ‘to market’ faster, and delivering with higher quality.

  5. Grid Benefits Department, Enterprise, and Global Grids • Access: transparent, remote, secure, wireless • Sharing:enable collaboration over the network • Failover: migrate/restart applications automatically • On Demand: get resources, when you need them • Productivity:more work done in shorter time • Virtualization: access compute services, not servers • Heterogeneity:platforms, OSs, devices, software • Resource Utilization: increase from 20% to 80+% • Virtual Organizations: build & dismantle on the fly

  6. However, There is Still a Long Way to Go ! • Grids are over-hyped: currently, they promise much more than they can really offer. • Grid technology is far from mature and complete. • Grid standards are (mostly) still missing. • Grids are very complex IT infrastructures. • Grids bring new challenges: sharing resources, loosing direct control, security, intellectual property, legal, social, political issues . . .

  7. Grid Architecture, Technology, Standards

  8. Architectural Requirements • Data Distribution • Location, volume, refresh • Security of data • Usability • Administrative Skill set and • Client environment • Psychological factors important • Operations Management • 100s CPUs / SA • Resources added in large blocks • Change control is critical • Availability • Downtime impact • Individual jobs • Maintenance windows • Scalability • Growth 1-3-5 years • Scaling strategy and • Response to peak loads • Technology refresh, evolution • Manageability • Skill set / workload of SA • Expected stability • Code management, software distribution • Security • User authentication • Internet access • Data Security requirements

  9. ... other interfaces Grid Service Service Data Element Service Data Element Service Data Element Implementation / Interfaces Hosting Env/runtime: C, J2EE Systems Infrastructure The Globus Toolkit and the Open Grid Services Architecture (OGSA) • Integration of Grid technologies and Web Services • OGSA defines a “Grid Service” • In terms of WSDL interfaces, defines mechanisms: • creating distributed systems • lifetime management • change management • credential management • notification Notification Authorization Service Creation Service Registry Manageability Concurrency

  10. Global Grid Forum (GGF) • Community-driven set of working groups that are developing standards and best practices for distributed computing ("Grids" and "Metacomputing") efforts • Formed in 2001 (merger of US, APAC, Euro efforts) • Open Grid Services Architecture (OGSA)

  11. Enterprise Grid Alliance (EGA) • Industry-driven consortium to implement standards in industry products and make them interoperable • Founding members: Oracle, Sun, Fujitsu, • ‘Enforce’ complementary position in pushing GGF into the academic corner

  12. Sys Mgmt The Changing Grid Landscape Developer View: Write here Run anywhere Service Composition Utility Computing HPTC Grid OnDemand Web Svcs “Compute Power from the wall socket” Standards Public SOA Outsourcing/Hosting Software as a Svc . . . HP Technologies IBM IBM Enabled Business Models Enterprise IT Mgmt Products Sun Market making / Productization Details see backup slides

  13. Cluster Grid Implementation, Example

  14. Internet Department Grids Browser to CGM (Remote Server Setup & Configuration) Compute Grid Manager Auto OS Deployment Grid Installation/Mgmt Centralized Server Mgmt Auto Download of Modules Solaris Servers Linux Servers Workstations (Linux or Solaris)

  15. Gbit-E switch Gbit-E switch Gbit-E switch Gbit-E switch Gbit-E switch V240 / V880 NFS V240 / V880 NFS V880 QFS/NFS Server V880 QFS/NFS Server V240 / V880 NFS V240 / V880 NFS FC Switch Enterprise Grid Reference Architecture Browser Access via GEP SunRay Access Workstation Access Optional Control Network (Gbit-E) Myrinet Myrinet Servers, Blades, & VIZ Myrinet Linux Racks Grid Manager Workstations Sun Fire Link Data Network (Gbit-E) NAS/NFS Simple NFS HA NFS Scalable QFS/NFS

  16. Project C Team B User 1 Department 5 Department 4 Contractor X User 2 Department 3 Project A Department 1 Department 2 Department resource access Campus wide resource demand Grid Enterprise Policy Management Clear definition and documentation of priorities and policies is key to Grid resource management

  17. a Proprietary Data b c d e SGE f g h SGE SGE Data mapped and available through Avaki Data Grid Clients view / Avaki HPC a b c d e f g h Globus & Avaki over multiple local Grids • Grid Engine cluster mgmnt within an admin domain & file system area • Globus/Avaki knits together resources, handles files, binary management, and high level resource selection Organization B Organization C Organization A Avaki Data Grid

  18. Applications for The Grid • Single-CPU Jobs:jobmix, many users, many serial applications, suitable for grid (e.g in universities and research centers) • Array Jobs: 100s/1000s of jobs, one user, one serial application, varying input parameters, suitable for grid (e.g. parameter studies in Optimization, CAE, Genomics, Finance) • Massively Parallel Jobs:one job, one user, one parallel application, no/low communication, scalable, fine-tune for grid (time-explicit algorithms, film rendering, pattern recognition) • Parallel Jobs:one job, one user, one parallel application, high interprocs communication, not suitable for distribution over the grid, but for parallel system in the grid (time-implicit algorithms, direct solvers, large linear algebra equation systems)

  19. Grid Computing Today in Industry

  20. Computing as a Utility =>> Enhancing The Grid with a Business Model • On Demand: Get a service at your finger tip • From the Wall Socket: Don't care about the infrastructure • Metering & Billing: Pay as you go, for what you use What's a Utility? Like electricity, water, gas, heat, telephony

  21. Time-to-market Pressures vs. Quality Products • Multiple, concurrent projects • Projects with different schedules and milestones • Overlapping demands for resources • Large and growing data sets • Larger, more complex product designs • Not enough throughput • Low utilization of compute resources • Insufficient in-house IT expertise • Increasingly complex operations management • Heightened focus on budget/TCO/ROI Customer Challenges

  22. More Compute Power • Improved Resource Access • Increased Productivity • Better Utilization of Existing Resources • Reduced Costs • Reduced Management Complexity Critical Customer Requirements

  23. Who Uses Department and Enterprise Grids? • Life Sciences • Startup and cost efficient • Custom research or limited use applications • Multi-day application runs (BLAST) • Exponential Combinations • Limited administrative staff • Complementary techniques • Electronic Design • Time to Market • Fastest platforms, largest Grids • License Management • Well established application suite • Large legacy investment • Platform Ownership issues • Financial Services • Market simulations • Time IS Money • Proprietary applications • Multiple Platforms • Multiple scenario execution • Need instant results & analysis tools • High Performance Computing • Parallel Reservoir Simulations • Geophysical Ray Tracing • Custom in-house codes • Large scale, multi-platform execution

  24. How Real is all this Stuff? direction of technology adaptation vendorinterest researchactivity department enterprise global

  25. Company Example: Sun Microsystems

  26. Sun Microsystems & Grid Computing • Slogan: The network is the computer • Strategy: N1, manage N computers as 1 • Acquisitions • Gridware => Sun Grid Engine, Distributed Resource Manager • Pirus => Dynamic Storage Provisioning • Terraspring => Automatic System Provisioning • Centerrun => Automatic Application Provisioning

  27. Managing GridsWhat we did for the big SMPs......we are now doing for the Network • Resource Virtualization • Domains, Zones • Interdomain resource mgmt • Routing • Soft cabling within the box Solaris N1

  28. Virtualization of Resources Service1 Vertical Horizontal StorageNetwork Service2 Service3 Compute Storage Compute & Data Grid

  29. Services sharing resources Services Services Services 2 3 1 VirtualNetwork VirtualStore VirtualCompute N1: managing services, not servers

  30. Grid Compute Service DB App Svr Web Server N1 Provisions Grid Computing DB App Svr App Svr Web Server Web Server Web Server App Svr Web Server DB

  31. The N1 Effect on EfficiencyRadical improvement in costs & uptime • Sys Utilization 80+ % N1 6-15% 500+ • Server/Admin 15-30 Costs 100TB • Terabytes/DBA 1TB 500+ • Ports/Admin 50-100 99.999+ % • Availability 99.9 % Uptime Hours • Time to Deploy Weeks

  32. Grid Computing Use Cases, Deployments, Best Practices

  33. Leeds, York + Sheffield Universities • Deliver stable, well-managed HPC resources supporting multi-disiplinary research • Deliver a Metropolitan Grid across the Universities Example: White Rose Grid in England

  34. WRG Hardware Maxima Snowdon Pascali Titania Maxima

  35. WRG Architecture Overview portal GT2.0 White Rose Grid GT2.0 GT2.0 GT2.0 GT2.0 GEEE GEEE GEEE GEEE Solaris Solaris Solaris Linux Titania Maxima Snowdon Pascali

  36. WRG Key Components • Globus Toolkit 2.4 Provides a secure means for inter-campus actions • Transferring jobs • Moving data • Gathering information about resources • Grid Engine Enterprise Edition Manages the campus grid compute resources • Delivers a single interface for a heterogeneous grid • Guarantees a share of campus resource for grid and local users

  37. WRG Key Components • Grid Portal Development Kit Provides a portal interface into Globus Toolkit • Transferring jobs • Moving data • Gathering information about resources • MyProxy MyProxy provides a server with client-side utilities to store and retrieve delegated X.509 credentials via the Grid Security Infrastructure (GSI).

  38. Grid Engine Enterprise Edition, Share Policies GT2.0 GT2.0 GEEE GEEE Solaris Solaris GT2.0 GT2.0 GEEE GEEE Solaris Linux White Rose Grid

  39. Example: Grid Service Provider MCNC & The Grid Infrastructure for North Carolina NC Statewide Grid Initiative MCNC Enterprise Grid • MCNC • Microelectronics Center of North Carolina • Founded in 1980 • Independent, private, non-profit organization • Operates NCREN since 1984 • Past: Supercomputer Center for universities • Now: Grid Service Provider, offering Video, • Network, Grid, and Datacenter Services • 50+ employees NC BioGrid NCREN North Carolina Research & Education Network

  40. GCNS Grid Computing and Networking Services Mission Advance education, innovation and economic development throughout North Carolina by delivering next generation information technology that enables the academic, research, government and business communities to discover, create, share and apply knowledge.

  41. The GridThe next IT Infrastructure of North Carolina • Proving ground for Grid • Successful prototype apps • Catalyst for collaboration • International recognition NC BioGrid 2002 2003 2004 2005

  42. The Grid The next IT Infrastructure of North Carolina • Cluster and SMP resources • Research platform for GTEC • Core component in NCGrid • Revenue generation MCNC Enterprise Grid NC BioGrid 2002 2003 2004 2005

  43. The Grid The next IT Infrastructure of North Carolina • State-wide partnership • Leverage lessons learned • Grid education & training resource • Enable first mover applications NC Grid Initiative MCNC Enterprise Grid NC BioGrid 2002 2003 2004 2005

  44. MCNC: 3-Year GCNS Grid Roadmap Grid Service Provider: • Network, Computing, Data, Video • Partner with SC Sites • Build GTEC Service Portfolio • Start Grid Consulting • Annual GSP Workshop Easy Access: Training, Web Courses Access Grid Node Grid Appliance QuadA Awareness Creation: Conferences, Workshops, PR MCNC Enterprise Grid NC BioGrid NC Statewide Grid CY04 CY03 CY05 CY06

  45. Internet Internet 2 NLR Internet North Carolina’s Foundation for Grid: NCREN 4-7 MCNC-owned Clusters distributed throughout the state Locations still under evaluation Cisco EPA Existing: Blend of owned and leased fiber and circuits moving toward resilient rings powered by Cisco routers Planned: Strong focus on owned and leased fiber, Lambda, and few circuits, in resilient rings powered by Cisco routers and Wave Division Multiplexers

  46. GCNS Enterprise Grid Avaki Data Grid Global Grid Resource DB (GIIS) Users Campus Grids Grid Gatekeeper / Interactive Nodes Data Grid Access Servers (8 total, i.e. 1 per 8 nodes) LSF Master Job Scheduler 32-CPU SGI Altix Linux SMP Server 128-CPU IBM Linux Cluster (64 nodes) 8-TB Storage

  47. Network, Grid and Data Center Services GTEC, NLR, ANR and other Innovation Initiatives DEPLOYMENT Self-serve Data Center Services Value-add Information Systems Services Enterprise Grid Services State-wide Grid Services Data Archival Services Information Security Services Hosting & Infrastructure Grid Computing Information Assurance DATA CENTER NCREN

  48. The NC State-Wide Grid Roadmap • 06/04: “Do-Grid-Yourself” Workshop • 07/04: Phase 1, Awareness Creation • 10/04 – 06/05: Deliver/Connect Grid Appliance Clusters to University Partners in NC • 10/04: Develop “Do-Grid-Yourself” Training Course • 10/04: Start QuadA Project: Access, Accounting, Authentication, Authorization • 12/04: Start Deliver Grid Training to Partner Univs • 01/05 – 06/05: Work with Grid Users to Port Apps • 03/05: Build Access Grid Node • 06/05: 1st NC Statewide Grid Workshop

  49. Grid Projects Driving Grid Adoption MCNC Grid SP Grid ColLab Kids Grids Gaming Grid School Grids Grid-Info Grid Download-The-Grid AAAA Project Grid Training Courses Startup Grid Grid Appliance Grid Portal GGF & EGA NC Statewide Grid MCNC Enterprise Grid NC BioGrid MCNC Supercomputing 2000 20004 2008

  50. Grid Vision

More Related