An Assessment OF VIRTUALIZATION SYSTEMS FOR hOSTING HPC APPLICATIONS IN CLOUD COMPUTING

1. An Assessment OF VIRTUALIZATION SYSTEMS FOR hOSTING HPC APPLICATIONS IN CLOUD COMPUTING Ahmad Hassan Centre for Advanced Computing and Emerging Technologies,School of Systems Engineering,The University of Reading, UK 29-MAR-10

2. Agenda • Personal Portfolio • Hypothesis • Motivation • Research concerns • Background • Experimental test-bed • Benchmark results • Conclusion • References • Appendix Erasmus Mundus Conference

3. Personal Portfolio • Worked as a Researcher in Large Hadron Collider (LHC) experiment at European Organization for Nuclear Research (CERN), Switzerland • Erasmus Mundus MSc in Network and E-business Centred Computing • University of Reading (UoR) – United Kingdom • Aristotle University of Thessaloniki (AUTh) – Greece • University Carlos III Madrid (UC3M) – Spain Erasmus Mundus Conference

4. Hypothesis “virtualization and in particular cloud computing are viable technologies for executing HPC applications with minimal performance degradation. “ Erasmus Mundus Conference

5. Motivation • Leverage of running multiple virtual machine concurrently on single physical hardware • Cloud Computing for running scientific applications • Virtualization features • Logging and monitoring of virtualization systems for Service Level Agreements (SLA) in cloud Erasmus Mundus Conference

6. Research Concerns • Do scientific applications run more efficiently on VM based systems, rather than on bare hardware? • Does virtualization effect the compute intensive applications? • Does Logging and monitoring impact the performance of virtual machines? • Application packaging and deployment on cloud based systems Erasmus Mundus Conference

7. Background • Virtualization • Cloud Computing • Virtualization Techniques • Related Research in HPC Benchmarking Erasmus Mundus Conference

8. Virtualization • Software abstraction layer for the underlying physical hardware. • CPU, Disk, RAM and Network • Abstraction layer is known as Virtual Machine Monitor or hypervisor • Type-I Hypervisor • Hypervisor runs directly on the top of underlying physical hardware i.e. XEN • Type-II Hypervisor • Hypervisor lies at one layer above the host operating system i.e. VMware server Erasmus Mundus Conference

9. Cloud Computing (1/2) • Platform to deliver IT resources as a service • compute, storage, collaboration tools, networking and a range of applications • Software as a Service (SaaS) • Complete application is provided as a service, e.g. Google applications • Platform as a Service (PaaS) • Provides a runtime application platform, and Web development tools for running applications on clouds, e.g. Windows Azure Platform and Google Application engine Erasmus Mundus Conference

10. Cloud Computing (2/2) • Infrastructure as a Service (IaaS) • Way to deliver IT resources, i.e. Compute, Storage over the Internet, for example Amazon Elastic Compute Cloud (EC2) • According to a study conducted by Gartner [7]: “By 2011, early technology adopters will forgo capital expenditures and instead purchase 40% of their IT infrastructure as a service. ‘Cloud computing’ will take off, thus untying applications from a specific infrastructure.” Erasmus Mundus Conference

11. Virtualization benefits for Cloud computing • Server Consolidation • Consolidate individual workloads onto a single physical machine • Migration • Isolate VMs from physical hardware and thus allows the migration of VMs from one platform to other. • Resource Sharing • Running many virtual machines over single hypervisor • Encapsulation • A virtual machine can act as a software appliance • Isolation Erasmus Mundus Conference

12. Virtualization Techniques (1/2) • Full Virtualisation • Hypervisor acts as a mediator between the operating system and the underlying hardware for transmitting hardware instructions. The VM’s are unaware of all this underlying procedure. For example VMware workstation • Hardware Assisted virtualisation • Underlying hardware supports the virtualisation technologies and provides assistance to enable virtualization i.e. Intel-VT and AMD-V. For example KVM Erasmus Mundus Conference

13. Virtualization Techniques (2/2) • Para-virtualization • An OS-assisted virtualization, where guest operating system is modified to get better performance. For example XEN Erasmus Mundus Conference

14. Related research in HPC Benchmarking (1/2) • Lamia [1] revealed that Para-virtualization does not impose significant performance overhead in high performance computing • AMD claimed to achieve near-native performance through virtualization • Anand et.al [2] found that XEN virtualization impacts different application in different ways • Tikotekar [4] tested the effect of virtualization on hyper-spectral radiative transfer code, Hydrolight and observed 11 % performance overhead Erasmus Mundus Conference

15. Related research in HPC Benchmarking (2/2) • Macdonell et.al [5] found the performance of GROMACS, is under 6% and for I/O intensive applications i.e. BLAST and HMMer the overhead is 9.7% • Zhao et.al [6] tested VMs for distributed computation within a grid framework. The study concluded that the overheads of virtual machines are acceptable when considering the variety of extra features provided by virtualisation systems than bare metal Erasmus Mundus Conference

16. Limitations of Existing Research • Not all the virtualization systems were considered • Most of the studies used XEN for benchmarking • Same HPC application was not used for benchmarking across different virtualization systems • Needs wide range of application testing on variety of hardware resources for general conclusion Erasmus Mundus Conference

17. Experimental Test-bed (1/3) • The virtualization systems considered for benchmarking were: • XEN • Linux KVM • CITRIX Xenserver • VMware Server Erasmus Mundus Conference

18. Experimental Test-bed (2/3) • The following HPC applications were identified and deployed for benchmarking of virtualization systems • PARallel Kernels and BENCHmarks (PARKBENCH) • High Performance Linpack (HPL) benchmark • DL_POLY molecular simulation package • ScaLAPACK (Scalable LAPACK) • PARKBench, HPL and ScaLAPACK are available on netlib.org (Univ. of Tennessee) Erasmus Mundus Conference

19. Experimental Test-bed (3/3) • This research study used the following hardware resources: • AMD64 cluster • IBM JS20 Blade server • Thamesblue (IBM JS21 blades) Erasmus Mundus Conference

20. Test-bed deployment Erasmus Mundus Conference

21. PARKBench Results (1/2) • Logging and Monitoring OFF Erasmus Mundus Conference

22. PARKBench Results (2/2) • Logging and Monitoring ON Erasmus Mundus Conference

23. High Performance LAPACK (HPL)Results (1/2) • Logging and Monitoring OFF Erasmus Mundus Conference

24. High Performance LAPACK (HPL)Results (2/2) • Logging and Monitoring ON Erasmus Mundus Conference

25. DLPOLY Molecular Simulation Results (1/2) • Logging and Montioring OFF Erasmus Mundus Conference

26. DLPOLY Molecular Simulation Results (2/2) • Logging and Montioring ON Erasmus Mundus Conference

27. ScaLAPACK Results (1/2) • Logging and Montioring OFF Erasmus Mundus Conference

28. ScaLAPACK Results (2/2) • Logging and Montioring ON Erasmus Mundus Conference

29. Observations • Considering the bare metal performance as ‘1’, the following values show the multiplying factor for the performance of virtualization system comparing the bare metal performance Erasmus Mundus Conference

30. Conclusion (1/2) • Virtualization imposes little performance overhead • Logging and Monitoring causes some overhead but not to a larger extent • Can we say that the hypothesis, developed for this study, has been proved • In some cases, near native performance was observed • Wide range of virtualization features VS little performance degradation Erasmus Mundus Conference

31. Conclusion (2/2) • Needs further research in the field by testing more applications and identifying the parameters involve in the performance degradation of HPC application on virtualization systems • Cloud review community http://cloudreview.org Erasmus Mundus Conference

32. References • Para-virtualisation for HPC URL:http://homepages.inf.ed.ac.uk/group/lssconf/files2008/youseff.pdf • Anand Tikotekar, Geo_roy Valle, Thomas Naughton, Hong H. Ong, Christian Engel-mann, and Stephen L. Scott. An Analysis of HPC Benchmarks in Virtual Machine Environments. In 3rd Workshop on Virtualization in High-Performance Cluster and Grid Computing (VHPC) 2008 • Sequoia Scientific, Inc., Redmond,WA, USA. Hydrolight 4.2, 2000 • A. Tikotekar, G. Vall´ee, T. Naughton, H. Ong, C. Engelmann, S. L. Scott, and A. M. Filippi, “Effects of virtualization on a scientific application running a hyperspectral radiative transfer code on virtual machines,” in HPCVirt ’08: Proceedings of the 2nd workshop on System-level virtualization for high performance computing. USA ACM, 2008. Erasmus Mundus Conference

33. References • Macdonell, C. and P. Lu. Pragmatics of Virtual Machines for High-Performance Computing: A Quantitative Study of Basic Overheads. in 2007 High Performance Computing & Simulation Conference (HPCS'07). 2007. Prague, Czech Republic • Zhao et al., 2004] Zhao, M., Zhang, J., and Figueiredo, R. (2004). Distributed File System Support for Virtual Machines in Grid Computing. In HPDC ’04: Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC’04), pages 202–211, Washington, DC, USA. IEEE Computer Society • Gartner Press Release, “Gartner Highlights Key Predictions for IT Organisations and Users in 2008 and Beyond” 1/31/08 [Date accessed: 29-Oct-2009] Erasmus Mundus Conference

34. Thanks of your attention Questions please !! Ahmad Hassan Contact: ahmad.hassan@cern.ch Web: http://cern.ch/ahmadh/portfolio Erasmus Mundus Conference

35. Appendix Erasmus Mundus Conference

36. Benchmark Designs • The benchmarks were designed to run on virtualisation system and bare metal Erasmus Mundus Conference

37. State-of-the-art Technologies • XEN Para-virtualization system • Open source virtualization system started by University of Cambridge Computer laboratory • Type-I hypervisor • Provides an abstraction to the underlying physical hardware • Undertakes scheduling • Direct access to the physical hardware, to guest operating system through a split driver model • Needs modified guest operating systems. Erasmus Mundus Conference

38. XEN Architecture • XEN Split Driver Model Erasmus Mundus Conference

39. Linux KVM • Kernel module added to the existing Linux open source distribution • Enables using standard Linux as hypervisor • Hardware-assisted virtualization that uses processor virtualization extensions i.e. INTEL-VT, AMD-V. • Provides Para-virtualization for I/O devices through Para-virtualization drivers. Erasmus Mundus Conference

40. KVM architecture • KVM and QEMU relationship Erasmus Mundus Conference

41. VMware Server • Supports both full virtualization and Para-virtualization solutions. • Uses Virtual Machine Interface (VMI) Technology for Para-virtualization • Volume Shadow Copy Service (VSS) to take snapshots • Virtual Machine Communication Interface (VMCI) for communication between host and virtual machine • VMware infrastructure Web Management Interface for web based management of virtual machines • VMware is retiring Para-virtualisation support in favour of Hardware assisted virtualization in future products Erasmus Mundus Conference

42. Experimental Test-bed (1/3) • Hardware platforms • AMD64 hardware based cluster Erasmus Mundus Conference

43. Experimental Test-bed (2/3) • IBM JS20 Blade Specification Erasmus Mundus Conference

44. Experimental Test-bed (3/3) • Thamesblue Specification Erasmus Mundus Conference

45. Hypervisor Types (1/2) • Type-I Hypervisor • Hypervisor runs directly on the top of underlying physical hardware i.e. XEN Erasmus Mundus Conference

46. Hypervisor Types (2/2) • Type-II Hypervisor • Hypervisor lies at one layer above the host operating system i.e. VMware server Erasmus Mundus Conference

47. PARKBench Results • Logging OFF • Logging ON Erasmus Mundus Conference

48. High Performance LAPACK (HPL)Results Logging & Monitoring OFF Logging & Monitoring ON Erasmus Mundus Conference

49. DLPOLY Molecular Simulation Results Logging & Monitoring OFF Logging & Monitoring ON Erasmus Mundus Conference

50. ScaLAPACK Results Logging & Monitoring OFF Logging & Monitoring ON Erasmus Mundus Conference