Developing High Performing Parallel Application Services on Windows Azure

1. AZR303 Developing High Performing Parallel Application Services on Windows Azure Wen-ming Ye Sr. Technical Evangelist Microsoft Corporation

2. Session Goals and Agenda • Understand Compute Intensive Workload in Windows Azure • Learn Build, Deploy, and Run Services using the HPC Scheduler • Get ideas for teaching your old CI apps some new Cloud tricks!

3. What’s the Problem? • Some apps run too slowly on a single computer • Splitting them up into chunks, then running those chunks in parallel on multiple computers can speed them up • Doing this effectively requires creating a compute cluster, with: • Tools to create and manage the cluster • A batch scheduler for running apps on the cluster

4. How Does Azure Extend High Performance Computing to the Masses? Solve large computational problems HPC is best described as massively parallel Becoming a critical part of businesses

5. Example Workloads That Need a Cluster • Digital media rendering and encoding • Financial and insurance risk calculation • Engineering modeling and simulation • Computational life sciences • Earth sciences • Data analytics

6. video Earthquake and Weather Simulations High resolution Computing

7. demo A 20,000 x 20,000 pixel Tobacco Mosaic Virus render

8. Embarrassingly Parallel AppsAn illustration Cluster of Compute Nodes Compute Node Compute Node Compute Node Application Logic Application Logic Application Logic 1. Submit work Compute Node Compute Node Compute Node 2. (Optionally) Interact with application Application Logic Application Logic Application Logic 3. Get results Compute Node Compute Node Compute Node Application Logic Application Logic Application Logic

9. Tightly Coupled App (MPI)An illustration Cluster of Compute Nodes Compute Node Compute Node Compute Node Application Logic Application Logic Application Logic 1. Submit work Compute Node Compute Node Compute Node Application Logic Application Logic Application Logic 2. Get results Compute Node Compute Node Compute Node Application Logic Application Logic Application Logic

10. IO Intensive ApplicationsAn illustration Cluster of Compute Nodes Compute Node Compute Node Compute Node Application Application Application 1. Submit work Compute Node Compute Node Compute Node Application Application Application 2. Get results Compute Node Compute Node Compute Node Application Application Application

11. HPC Server Makes This Easy – offloading long running jobs Client Windows HPC Application Logic Logic Windows HPC Server Logic Job Scheduling Cluster Management Head Node (Physical or VM) Compute Nodes Windows 7 Windows HPC Server Windows Azure

12. Windows HPC Server 2008 R2 SP3

13. Cloud - Applications in AzureJune 2012 Release • Application in hosted entirely in Azure • No on-premise Head Node • Application is accessed from • Rich client on premise • Portal • Web Application • Multiple business models • Packaged application is sold to a customer • Application is available as a service in the cloud • Scheduler and Runtimes Supported in Azure • Workloads: Parametric Sweep, SOA, MPI • No Head Node (Scheduler Service, no Admin Services)

14. Building a Scheduler Enabled Deployment • Azure Subscription • Your application • Construct a base CS Package • Customize and Deploy

15. Windows Azure Scheduler In Action A Windows Azure Deployment Front End Compute Worker Front End Node Manager Application Front End (Web Page) Application Windows Azure Scheduler Compute Worker Compute Worker Compute Worker Consistent HPC Job Scheduler API Compute Worker

16. Visual Studio Sample Solution – Easy first development experience for WAHS parallel applications in Azure • Deployment Application • Auto-create and initialize hosted service, storage, HPC database in same location • Progress bar and status • PSH Scripts available too Example: Cloud Numerics…

17. demo Deploy a cluster using Visual Studio

18. Scenario: Full desktop client offloading long running tasks to the Azure HPC Scheduler

19. Simplicity of Microsoft HPC Compute Intensive Paradigms Run on: • Client • Cluster • Cloud Transparently

20. demo Openfoam compute offload Rich client application connecting to windows Azure scheduler for offloading a fluid dynamics model

22. Scenario: More code to run: Moving code form Linux/Unix platforms

23. OSS Eco-system To create Scalable TC Azure services

24. Scenario: Building a Service for sharing simulation engine and data using the Azure HPC Scheduler

25. demo Weather as a Service

26. Roadmap Spring 2012 All items considered general availability, unless noted. *Denotes Spring in the US, 1st half of 2012 • Hybrid Clusters Across Enterprise and Cloud • Support for Client->Job Scheduler .NET API tunneled over HTTP (i.e., same Job Scheduling API for on-premise and Windows Azure clusters) • Building HPC and Parallel Applications in the Cloud • Initializing HPC SQL Azure DB from HN in Azure Windows HPC Pack 2008 R2 Service Pack 4

27. Who would use it? • Do it yourself cluster • Power user • In house developer • Packaged software • Leverage additional compute resource in Windows Azure • Software as a Service • Vendor pays for the compute resources • Vendor bills end user directly

28. 9(-5) Tips for Best Practices • Hide cloud from your users by providing a transparent experience • Minimize data transfer between on-premises and the cloud • CPU intensive jobs are best suited for pay as you go model • Be creative and focus on what you can do on Azure today

29. The Matrix can not be told, tryit yourself! • Subscribe to my blog http://blogs.msdn.com/hpctrekker • ReadApplication Models and Data Considerations • Download SP4 from download.microsoft.com • Get a free Trial of Azure for 30 days • Download and try out the Azure Burst Samples • Implement your own solution and let us know!

30. Thank You!

31. Track Resources @WindowsAzure @ms_teched Hands-On Labs Meetwindowsazure.com DOWNLOAD Windows Azure Windowsazure.com/ teched

32. Resources Learning TechNet • Connect. Share. Discuss. • Microsoft Certification & Training Resources http://europe.msteched.com www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers • http://microsoft.com/technet http://microsoft.com/msdn

33. Evaluations Submit your evals online http://europe.msteched.com/sessions

34. © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. Required Slide Track PMs will supply the content for this slide, which will be inserted during the final scrub. Track Resources My Blog: http://blogs.msdn.com/hpctrekker The Azure Burst Samples and White paper Windows HPC ResourceKit Windows HPC Server TechCenter