1 / 49

High Productivity Computing With Windows

High Productivity Computing With Windows. Windows HPC Server 2008. Agenda. Why Microsoft in HPC?. Current Issues HPC and IT data centers merging: isolated cluster management Developers can’t easily program for parallelism

edie
Download Presentation

High Productivity Computing With Windows

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. High Productivity Computing With Windows Windows HPC Server 2008

  2. Agenda

  3. Why Microsoft in HPC? • Current Issues • HPC and IT data centers merging: isolated cluster management • Developers can’t easily program for parallelism • Users don’t have broad access to the increase in processing cores and data • How can Microsoft help? • Well positioned to mainstream integration of application parallelism • Have already begun to enable parallelism broadly to the developer community • Can expand the value of HPC by integrating productivity and management tools • Microsoft Investments in HPC • Comprehensive software portfolio: Client, Server, Management, Development, and Collaboration • Dedicated teams focused on Cluster Computing • Unified Parallel development through the Parallel Computing Initiative • Partnerships with the Technical Computing Institutes

  4. Microsoft’s Productivity Vision for HPC Windows HPC allows you to accomplish more, in less time, with reduced effort by leveraging users existing skills and integrating with the tools they are already using. Administrator Application Developer End - User • Integrated Turnkey HPC Cluster Solution • Simplified Setup and Deployment • Built-In Diagnostics • Efficient Cluster Utilization • Integrates with IT Infrastructure and Policies • Integrated Tools for Parallel Programming • Highly Productive Parallel Programming Frameworks • Service-Oriented HPC Applications • Support for Key HPC Development Standards • Unix Application Migration • Seamless Integration with Workstation Applications • Integration with Existing Collaboration and Workflow Solutions • Secure Job Execution and Data Access

  5. Windows HPC Server 2008 • Complete, integrated platform for computational clustering • Built on top the proven Windows Server 2008 platform • Integrated development environment • Evaluation available from http://www.microsoft.com/hpc

  6. What’s new for HPC in Windows Server 2008?

  7. What’s new in the HPC Pack 2008? • New System Center UI • PowerShell for CLI Management • High Availability for Head Nodes • Windows Deployment Services • Diagnostics/Reporting • Support for Operations Manager • Support for SOA and WCF • Granular resource scheduling • Improved scalability for larger clusters • New Job scheduling policies • Interoperability via HPC Profile Systems Management Job Scheduling Networking & MPI Storage • Improved iSCSI SAN & parallel file system Support in Win2008 • Improved Server Message Block ( SMB v2) • New 3rd party parallel system file support for Windows • New Memory Cache Vendors • NetworkDirect (RDMA) for MPI • Improved Network Configuration Wizard • Shared Memory MS-MPI for multi-core • MS-MPI integrated with Windows Event Tracing

  8. Spring 2008, NCSA, #23 9472 cores, 68.5 TF, 77.7% Spring 2008, Umea, #40 5376 cores, 46 TF, 85.5% Spring 2008, Aachen, #100 2096 cores, 18.8 TF, 76.5% Spring 2006, NCSA, #130 896 cores, 4.1 TF Winter 2005, Microsoft 4 procs, 9.46 GFlops Spring 2007, Microsoft, #1062048 cores, 9 TF, 58.8% Fall 2007, Microsoft, #1162048 cores, 11.8 TF, 77.1% 30% efficiencyimprovement Windows HPC Server 2008 Windows Compute Cluster 2003

  9. Windows HPC Server 2008 Ready for Prime-time #23 Summer 2008 7.8% improvement in efficiency on the same hardware running Linux About 4 hours to deploy

  10. Systems Management

  11. Improved Efficiency for the Systems Admin • Simple to setup and manage in a familiar environment • Turnkey cluster solutions through OEMs • Simplify system and application deployment • Base images, patches, drivers, applications • Focus on ease of management • Comprehensive diagnostics , troubleshooting and monitoring • Familiar, flexible and “pivotal” management interface • Equivalent command line support for unattended management • Scale up • Scale deployment, administration, infrastructure • Head node failover • Cluster usage reporting • Compute node filtering • Better integration with enterprise management • Patch Management • System Center Operations Management • PowerShell • Windows 2008 high Availability Services

  12. Head Node High Availability • Eliminates single point of failure with support for high availability • Requires Windows Server 2008 Enterprise Failover Clustering Services • Next generation of cluster services • Major improvement in configuration validation and management • HPC Pack Includes • Setup integration with Failover Clustering Services • Head Node and Failover Node set up with SQL Failover Cluster • Job Scheduler services failover • Management console linked to Windows Server Failover Management console Private Network Windows Failover Clustered Head node Win2008 Enterprise Clustered SQL Server Failover Head node Win2008 Enterprise Clustered SQL Server Shared Disk

  13. System Center Operations Manager for HPC

  14. Networking & MPI

  15. User Mode Kernel Mode NetworkDirect A new RDMA networking interface built for speed and stability • Priorities • Comparable with hardware-optimized MPI stacks • Focus on MPI-Only Solution for version 2 • Verbs-based design for close fit with native, high-perf networking interfaces • Coordinated w/ Win Networking team’s long-term plans • Implementation • MS-MPIv2 capable of 4 networking paths: • Shared Memory between processors on a motherboard • TCP/IP Stack (“normal” Ethernet) • Winsock Direct (and SDP) for sockets-based RDMA • New RDMA networking interface • HPC team partners with networking IHVs to develop/distribute drivers for this new interface Socket-Based App MPI App MS-MPI Windows Sockets (Winsock + WSD) RDMA Networking TCP/Ethernet Networking Networking Hardware Networking Hardware Networking Hardware Networking Hardware Networking Hardware Networking Hardware Mini-port Driver WinSock Direct Provider NetworkDirect Provider TCP IP NDIS Kernel By-Pass Networking Hardware Networking Hardware Networking Hardware Networking Hardware Networking Hardware Networking Hardware Networking Hardware User Mode Access Layer Hardware Driver (ISV) App CCP Component OS Component IHV Component

  16. High Speed Networking Technologies Cisco Voltaire Qlogic Open Fabrics Myrinet, Infiniband, 10GigE NetEffect Bandwidth Myricom 1Gig Ethernet 100MB Ethernet Availability

  17. Job Scheduling

  18. What’s new in in Job Scheduling

  19. Broader Application Support 2003 (focusing on batch jobs) 2008 (focusing on Interactive jobs) Job Scheduler Resource allocation Process Launching Resource usage tracking Integrated MPI execution Integrated Security WCF Brokers WS Virtual Endpoint Reference Request load balancing Integrated Service activation Service life time management Integrated WCF Tracing + App.exe App.exe App.exe App.exe Service (DLL) Service (DLL) Service (DLL) Service (DLL)

  20. Service-Oriented Jobs Job status Service usage report Tracing logs Job status Service usage report Tracing logs Job Scheduler Admin Experience Router Job Service Job Node heat maps, Performance monitor, & Event logs WCF Broker Compute Nodes Redundant Head Nodes Control Path 1 n Service Instances Create a Session User Experience Node Manager Node Manager Node Manager … Client 2 Start Broker Service Instance Service Instance Service Instance Data path Send/Receive Messages 3 WCF Broker Nodes Occurring on the back end Track service resource usage Run service as the user Restart upon failure Support application tracing Balance the requests Grow & shrink service pool Provide WS Interoperability

  21. Job Templates

  22. Job Templates Job Submission Job Submission Job Submission Permission to Submit Permission to Submit Permission to Submit 1 Admin creates resource partitions by creating job submission policies Job Template Job Template Job Template Example Policies Cluster Resources 2 Users submit using different templates Cluster Resources Cluster Resources

  23. Five New Scheduling Policies • Resource Matching • Job submit /nodegroup:appX myapp.exe • Job Admission Policies via Templates • Job submit /template:groupY myapp.exe • Multi-Level Processor Allocation • job submit /numsockets:4-8 myapp.exe • Adaptive Allocation (Grow/Shrink) • job submit myapp.exe • Preemption • CluscfgPreemptionEnabled=true

  24. Scenario: Placement via Job Context • node grouping, job templates, filters MATLAB A A A A A C0 C1 C2 C3 Application Aware An ISV application (requires Nodes where the application is installed) MATLAB M M Multi-threaded application (requires machine with many Cores) MATLAB MATLAB MATLAB MATLAB MATLAB MATLAB MATLAB MATLAB MATLAB MATLAB MATLAB MATLAB Capacity Aware A big model (requires Large memory machines) P0 P1 M M |||||||| |||||||| Numa Aware M M M M 4-way Structural Analysis MPI Job |||||||| |||||||| M M P2 P3 C0 C1 C2 C3 IO IO Quad-core 32-core M

  25. Scenario : Memory Bus Saturation Policy: Multi-level Allocation Node 1 Node 2 J2 J2 J1 J2 J2 J1 S3 S2 S1 S0 S0 S1 S2 S3 P1 P1 P1 P1 P1 P1 P1 P1 P0 P0 P0 P0 P0 P0 P0 P0 J2 J2 J2 J2 P2 P2 P2 P2 P2 P2 P2 P2 P3 P3 P3 P3 P3 P3 P3 P3 J2 J2 J1 J2 J2 J3 J2 J2 J1: /numsockets:3 /exclusive: false J3: /numsockets:3 /exclusive: false J2: /numsockets:14 /exclusive: false

  26. Scenario: mixed workload Policy: Priority Resource Allocation • Faster turnaround for high priority jobs • Uses Grow/Shrink policy, Preemptive Scheduling Policies Job 3 gets Submitted Job 3 gets Completes Processors Job 1 Tasks Job 2 Tasks Job 3 High Priority Time

  27. Scenario: Long Tail Parametric Apps Policy: Grow/Shrink Resource Allocation • Addresses “Long Tail” problem for jobs with many tasks of varying runtime • Improve cluster utilization by giving nodes where they are needed and to the highest priority jobs Idle Processors Idle Processors Processors Idle Processors Job 1 Tasks Job 2 Tasks Job 3 Tasks Time

  28. Job Policies Summary

  29. Interoperability & Open Grid Forum LSF / PBS / SGE / Condor Linux, AIX, Solaris HPUX, Windows Windows Cluster Windows Center Window Center

  30. Job Scheduling Summary

  31. Parallel Programming

  32. Parallel Programming • Parallel Program Tools • Available Now • Development and Parallel debugging in Visual Studio • 3rd party Compilers, Debuggers, Runtimes etc.. available • Emerging Technologies – Parallel Extensions to .NET Framework • LINQ/PLINQ – natural OO language for SQL queries in .NET • Task Parallel libraries • currently CTP June ‘08

  33. Version Comparison

  34. HPC Storage Solutions Greater Sophistication Aggregate (Mb/s/core) • IBM – GPFS • Panasas – Active Scale • SUN - Lustre • HP - PolyServe • Ibrix - Fusion • Quantum - StorNext • SANbolic – Melio file system • Windows Server 2003 • Windows Server 2008 • … Number of cores in cluster

  35. Release Schedule • RC1– Publicly Available • RTM – Early Fall 2008 Beta 1 RTM CTP 1 CTP2 Beta 2 Nov 2007 March 2008 April 2008 Fall 2008 May 2008

  36. Industry Focused Partners

  37. Windows HPC Server 2008 Offerings Basic Turnkey High Availability Enterprise Management • Turnkey Solutions up to 1000 nodes • HPC Pack • Management • Deployment • Job Scheduling • 3rd Party packages for enhanced job scheduling, policies • No single point of failure • Windows Server 2008 Enterprise for head node high availability • SQL Server 2008 Enterprise for high availability • Scale to many 1000’s of nodes • System Center Operations Manager for enhanced monitoring • System Center Configuration Manager for enhanced deployment and patch management SystemsCenter Configurations Manager 3rd Party SystemsCenter Operations Manager SQL Server 2008 Express 3rd Party SQL Server 2008 Standard 3rd Party SQL Server 2008 Standard HPC Pack HPC Pack HPC Pack Windows HPC Server 2008 Windows HPC Server 2008 Windows Server 2008 Enterprise Windows HPC Server 2008 Windows Server 2008 Enterprise • Lightweight small footprint/No UI/CLI • HyperV • No .NET Framework Windows Server 2008 Core

  38. Mainstream HPC Windows HPC Roadmap • Mainstream High Performance Computing on Windows platform • Interoperability: Web Services for Job Scheduler, Parallel File Systems • Applications: Service Oriented, Batch, .NET • Turnkey: Enabling pre-configured OEM solutions • Scale: Large scale, non-uniform clusters, diagnostics framework Version 2 H2 2008 SP1 & Web 2007 • Service Pack 1 • Performance & Reliability Improvements • Support for Windows Server 2003 SP2 • Support for Windows Deployment Services • Vista Support for CCP Client tools • Web Releases • MOM Pack • PowerShell for CLI • Tools for Accelerating Excel V1 Summer 2006 • Mainstream High Performance Computing on Windows platform • Simple to set up and manage in familiar environment • Integrated with existing Windows infrastructure

  39. Resources • Microsoft HPC Web site – Evaluate Today! • http://www.microsoft.com/hpc • Windows HPC Community site • http://www.windowshpc.net • Windows HPC Techcenter • http://technet.microsoft.com/en-us/hpc/default.aspx • HPC on MSDN • http://code.msdn.microsoft.com/hpc • Windows Server Compare website • http://www.microsoft.com/windowsserver/compare/default.mspx

  40. © 2008 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

  41. About the MPI Forum • The MPI forum is an international forum that defines the MPI standard. Since 1992 is has defined MPI 1.0 1.2 and 2.0 standards. This standard body is important as over the years MPI has become the de-facto parallel programming interface/paradigm. • The MPI forum has come together this year, 2008 in an effort to renew and update the MPI standard to put MPI 2.1, 2.2 and 3.0 standards. The MPI forum is determined goal is to answer the world demand for significant features which will enable MPI to better handle the scope of tomorrow’s systems and the breath of applications using MPI. • The MPI forum body includes representatives from many software companies, Universities, MPI implementers and hardware vendors. The forum meets in person every two months to discuss proposals, issues at hand and vote on the outcome. Microsoft takes a leading role in this forum leading several workgroup and hosting some of the forum meetings. • The MPI forum goal is to release MPI 2.1 standard by the end of 2008, MPI 2.1 by mid 2009 and MPI 3.0 by the end of 2010. • The unofficial wiki page web site; you can find MPI workgroups information on these pages. • https://svn.mpi-forum.org/trac/mpi-forum-web • The official MPI web site • http://www.mpi-forum.org • The meetings info web pages • http://meetings.mpi-forum.org

  42. Windows HPC Success Stories: Financial Services Financial Services Firm Scales Fast-Growing Hedging Application on Low-Cost Windows Cluster Fund Manager Improves Performance, Eases Administration with Computing Cluster “It’s like having an incredibly powerful CPU at your finger tips. The ease of use creates a kind of transparency where the line between the desktop and the compute cluster no longer exists.” - Christopher Mellen, Head of Research, Grinham Managed Funds Investment Bank Improves Competitive Edge with High Performance Computing Investment Bank Plans to Boost IT Performance, Reduce Hardware Costs with New Server Technology “From now on, whatever we will be testing, developing, or evaluating will be based on Windows Server 2008—whether it’s business-driven, transition-driven, or research-driven.”- Sorin Manta, Manager, Windows Server Infrastructure, Technology and Operations, BMO Capital Markets IT Services Company Develops Risk and Trading Systems on Integrated Platform ISV Opts Against Open Source, Doubles Revenue with Office Business Application “By using Microsoft technologies as the foundation for our development, we’re producing an easy-to-use, understandable, low-cost offering that can be confidently adopted by financial services customers.”- Graham Twaddle, Founder and Chief Executive Officer of Corporate Modelling

  43. Windows HPC Success Stories: Manufacturing Switch from Linux to Windows Increases Value of HPC for Golf Equipment “Manufacturer jobs that took 40 hours are now down to 8, enabling engineers to test and refine their designs much faster.”- John Loo, Design Systems Senior Manager, Callaway Golf Leading HPC Vendor Eases Adoption for Customers, with New Windows-Based Offerings “Many companies see value from HPC, but they have little to no experience with Linux… We need to provide a way for them to advance their theories and research using what they already know.” - Beverly Bernard, Product Manager, SGI 64-bit Compute Cluster Performs Crash Analysis, Aides in Safety Compliance Shipbuilder Introduces HPC on Microsoft to Develop High Value-Added Ship Types Boeing Tests High Performance Computing Cluster, Improves Processing Time “Because we develop intelligent graphics software in a Windows environment, it makes sense to work with an HPC cluster that supports that framework.” - John F. Bremer Advanced Computing Technologist, Intelligent Graphics Group Boeing Manufacturer Provides Engineers with Easy-to-Use, High-Performance Computing Solution Engineers at French Production Company Improve Productivity with New Business Solution

  44. Windows HPC Success Stories: Manufacturing Australian Company Delivers Solutions Faster, Expands Capabilities with HPC Solution “I came in with zero knowledge of Windows HPC Server 2008 deployment, although I knew a lot about Linux. Within a couple of days, I was deploying Windows-based nodes.” - Dr. Simon Beard ,Systems Specialist, On Demand Group, ISA Technologies NASCAR Team Turns to High Performance Computing to Sharpen Competitive Edge “With simulation times reduced from 24 hours to about 30 minutes, we can run multiple simulations for each race and better tune the simulations for each car, track, and set of track conditions.” - Mark PaxtonResearch and Development Engineering Manager, NASCAR Team, Chip Ganassi Racing Pebble Bed Modular Reactor (PBMR) in South Africa adopts Windows HPC Technology Over Linux. SIMULIA Delivers Simulation Solutions Faster with Windows HPC Server 2008, and Sees the Advanced Programming Tool Set as a Critical Asset. Microsoft High Performance Computing Solution Helps Oil Company Increase the Productivity of Research Staff “With Windows Compute Cluster Server, setup time has decreased from several hours—or even days for large clusters—to just a few minutes, regardless of cluster size.” - IT Manager, Petrobras CENPES Research Center

  45. Windows HPC Success Stories: Manufacturing French Yacht Team Streamlines Design with Secure and Familiar Technology “The transition from Linux to Windows Compute Cluster Server 2003 was flawless. In fact, it was so easy we didn’t even notice a change in the office.” - Bernard Nivelt, Lead Designer, AREVA Challenge Easier Cluster Management Helps Northrop Grumman Improve Productivity “Windows Compute Cluster Server has caused a paradigm shift. Before I had to limit my problem size because I ran out of resources. Now I feel enabled to think bigger.” - Thi Pham, PhD, Systems Engineer, Space Technology Sector, Northrop Grumman Simulation Software on x64 Compute Clusters Boosts Performance, Reduces Costs Florida Boosts Productivity, Cuts Run Times with High-Performance Computing Cluster Software Company Delivers 64-Bit Fidelity and Speed for Computer-Aided Engineering

  46. Windows HPC Success Stories: Science, Education, Research Windows HPC Server 2008 Ranks at 23 Among the World’s TOP500 Largest Supercomputers with 68.5 TFlops and 77.7 Percent Efficiency. (June 2008) “The performance of Windows HPC Server 2008 has yielded efficiencies that are among the highest we’ve seen for this class of machine.”- Robert Pennington, Deputy Director, National Center for Supercomputing Applications The Umeå Cluster Achieved a LINPACK Score of 46.04 TFlops and 85.6 Percent Efficiency, Making Their System the Second Largest Windows Cluster Ever Deployed and the Fastest Academic Cluster in Sweden. (June 2008) Leading Supercomputing Center in Italy Eases Use, Improves Access with New Cluster “It will be a big benefit for us to offer researchers a high-performance computing resource with a familiar interface and a natural, user-friendly way to use the cluster from home.” - Dr. Marco Voli, CINECA Researchers’ Move from Linux to Windows Yields Performance Gains, New Capabilities “We were quite surprised when, without any optimization, the new Windows–based HPC system outperformed our highly optimized Linux cluster.” – Valerie Daggett, Professor, University of Washington Facility for Breakthrough Science Seeks to Expand User Base with Windows-Based HPC

  47. Windows HPC Success Stories: Science, Education, Research Early Detection of Cancer One Step Closer to Solution with Microsoft, Dell and Intel “The user interface and structure of the Microsoft Windows Compute Cluster Server make managing a large, high-performance computing cluster far less daunting than with other operating systems.” - Dr Robert Moritz, Manager of the Proteomics Facility at LICR and Director of the Australian Proteomics Computational Facility National IT Center Improves Customer Service with High-Performance Compute Cluster. “Using Windows for high-performance computing means we can offer our customers real added value.”  - Uwe Wössner, Head of the Visualization Department, High Performance Computing Center Stuttgart Inventor of Beowulf Cluster Exposes Young Minds to High-Performance Computing Portuguese University Accelerates Cancer Research with High-Performance Computing Microsoft Researchers Boost Task Productivity Fiftyfold with Cluster Server Software

  48. Windows HPC Success Stories: Science, Education, Research Supercomputing Solution Reduces IT Administration Needs at University of Cincinnati Genome Research Institute. Research-Driven University Breaks Down Barriers to High-Performance Computing Environmental Scientists Join Forces Against Climate Change with Integrated Platform “This is the first time we’ve delivered an integrated solution whereby researchers can sit in front of a Web browser and drive it to completely different scenarios using the data and models of different institutions.”- Simon Cox, Professor of Computational Methods at the University of Southampton and Technical Director for Genie Scientists Accelerate Research and Insight with Accessible, High-Performance Computing Environment “Even students who come from a Linux background and are using Microsoft developer tools for the first time are finding the change to be positive.”- Iain Buchan, Director of the Northwest Institute for BioHealth Informatics at the University of Manchester University of British Columbia (Vancouver) selects Windows HPC technology over Linux for Masters of Digital Media graduate program

  49. Windows HPC Success Stories: Digital Content Creation Digital Marketing Firm Installs Microsoft HPC Solution to Simplify IT Operations Digital Media School Deploys Render Farm Technology, Cuts Compute Runtime By Days “Before we had render farm, every student rendered on his or her own PC, so sharing images and viewing the current status was not easy. Now we can decide how many PCs will render a particular image. On 32 machines it takes just a couple of hours—this is a huge reduction in time.”- Ng Kian Bee, Deputy Director, Games & Digital Entertainment of NYP’s SIDM

More Related