A study of introduction of the virtualization technology into operator consoles

1. A study of introduction of the virtualization technology into operator consoles T.Ohata, M.Ishii / SPring-8 ICALEPCS 2005, October 10-14, 2005 Geneva, Switzerland

2. Contents • Virtualization technology overview • Categorize virtualization technologies • Performance evaluation • How many virtual machines run on a server • Introduction into the control system • System setup • Conclusion ICALEPCS 2005 in Geneva, Switzerland

3. What is virtualization technology? ICALEPCS 2005 in Geneva, Switzerland

4. CPU CPU MEMORY MEMORY Network card Network card DISK DISK Mainframe Host computer Overview of a virtualization technology • Originated from IBM System/360 • Enable to consolidate many computers into a small number of host computer • Each virtual machine (VMs) has independent resources (CPU, disks, MAC address, etc.) like a stand-alone computer VM VM VM VM VM VM VM VM ICALEPCS 2005 in Geneva, Switzerland

5. Why we need virtualization technology? ICALEPCS 2005 in Geneva, Switzerland

6. We can construct an efficient control system We have over 200 computers only in beamline control system Computer proliferation • Increasing maintenance tasks such as version up, patching etc. • We facedincreasing hardware failure maintain them by a few staff Problem of present control system Network distributed computing is standard method ICALEPCS 2005 in Geneva, Switzerland

7. Virtualization technology has revived We can reduce a number of computers. We can cut hardware costs and their maintenance costs drastically. consolidation General-purpose server ICALEPCS 2005 in Geneva, Switzerland

8. Category of virtualization technology- Three virtualization approaches - * Evaluated products ICALEPCS 2005 in Geneva, Switzerland

9. 1. Resource multiplex • Originated from mainframe • Major UNIX vendors released several products • A layer multiplexes hardware resources (called hypervisor or virtual machine monitor) Need small patch to kernel Less overhead Special OS to suit layer interface S/W S/W S/W OS OS OS Multiplex hardware resources CPU, memory, etc. Hardware ICALEPCS 2005 in Geneva, Switzerland

10. 2. Emulation • Many emulator for PC/AT, 68K and game machines • Suitable for development and debugging Usable unmodified OS Some overhead in transform instructions emulation overhead S/W S/W S/W OS OS OS Emulation layer Operating system Hardware Hardware ICALEPCS 2005 in Geneva, Switzerland

11. 3. Application shielding • Developed for web hosting of IPS (internet service provider) to obtain separate computing environment • Partition makes invisible computing space from others No overhead partitions S/W S/W S/W Operating system Hardware Hardware ICALEPCS 2005 in Geneva, Switzerland

12. Performance evaluation How many VMs can run on a server computer ICALEPCS 2005 in Geneva, Switzerland

13. Evaluated products * Next sheet ICALEPCS 2005 in Geneva, Switzerland

14. Fair Share Scheduler (FSS) Scheduling policy, CPU usage is equally distributed among tasks CPU pinning Pin a VM to specific CPU (effective in SMP environment) (Linux has “affinity” function, which can pin only a process) Live migration VMs migrate to other host dynamically VMs can be running during migration Special function VM VM VM VM VM VM VM VM Host 1 Host 2 Live migration ICALEPCS 2005 in Geneva, Switzerland

15. Measurement procedure Response time between virtual machine and VME by using MADOCA application Message queue (SYSV IPC) and ONC-RPC network communication protocol VM RPC (Remote Procedure Call) Message size is 350 bytes including RPC header and Ethernet frame header VME MADOCA: Message And Database Oriented Control Architecture ICALEPCS 2005 in Geneva, Switzerland

16. Measure response time Measurement bench MADOCA client VM VM VM VM • 1~10 VMs are running on single server computer • (Dual Xeon 3.0GHz) • MADOCA client is running on each VM Network MADOCA server MADOCA server MADOCA server MADOCA server 1~10 MADOCA servers on a network ICALEPCS 2005 in Geneva, Switzerland

17. HP B2000 (reference) average response time [sec] Number of VMs Number of VM dependency of average response time • HP B2000 is present operator console • VMware and UML becomes worse at many VMs • 5~6 VMs of Solaris and Xen are comparable to HP workstation ICALEPCS 2005 in Geneva, Switzerland

18. Statistics of response time @ 10VMs response time[msec] better better ICALEPCS 2005 in Geneva, Switzerland

19. Limit of hardware resources- CPU utilization - • CPU utilization of the Host of VMs • No more IDLE time at 5~6 VMs (%) Solaris container CPU utilization 5~6 VMs are optimum Number of VMs ICALEPCS 2005 in Geneva, Switzerland

20. Limit of hardware resources- Network interface card (NIC) utilization - • Traffic on the GbE network interface card • Utilization is a few percent of full bandwidth • Saturation comes from CPU overload (MB/s) Solaris container NIC utilization Number of VMs ICALEPCS 2005 in Geneva, Switzerland

21. Limit of hardware resources- Page fault frequency - • Page fault wastes CPU time • It makes performance deterioration • Saturation come from miss hit of TLB and swap out Solaris container Number of VMs ICALEPCS 2005 in Geneva, Switzerland

22. How many VMs are optimum? 5~6 VMs are optimum@(Dual Xeon 3.0GHz) • Large page size on large addressing space architecture is important. - Physical Address Extension (PAE) or 64-bit architecture • Many core CPU is attractive. - One CPU core is enough for 2~3 VMs If you want to run more VMs… ICALEPCS 2005 in Geneva, Switzerland

23. Introduction into the control system We installed virtualization technology into a beamline control. • We use Xen and Linux PC servers by replacing HP operator console. • Control application programs ported onto VM (Linux). replace • We installed a pair of Xen host and NFS server to keep image file of VM. ICALEPCS 2005 in Geneva, Switzerland

24. VM Control programs VM Control programs VM VM VM VM VM Image VM Image VM Image System setup and live migration It is possible to use continuously during maintenance. X-server (thin client) Primary Xen host Secondary Xen host Migration Enable shutdown A few 100msec Gigabit Ethernet NFS server ICALEPCS 2005 in Geneva, Switzerland

25. Future plan- High availability cluster - • We are studying high availability Single System Image (SSI) cluster configuration with Xen • Migration function of Xen is not effective when host computer suddenly dies. software software software software Single System Image cluster VM VM VM VM VM VM VM VM VM Xen hypervisor Xen hypervisor Xen hypervisor Structure of OpenSSI with Xen ICALEPCS 2005 in Geneva, Switzerland

26. Future plan (cont’)- reduandant storage - • We will introduce a redundant storage system such as SAN, iSCSI and NAS. • NFS server is a single failure point Primary Xen host FC Switch SAN storage SAN fibers Secondary Xen host FC Switch ICALEPCS 2005 in Geneva, Switzerland

27. Cost estimation About 50 HP-UX workstations will be replaced 8 PC-base servers + redundant storage (6 VMs runs on each PC server) 75% of total cost can be saved (only hardware) ICALEPCS 2005 in Geneva, Switzerland

28. Conclusion • We studied several virtualization technology to introduce as operator console. • We measured performances of some virtualization environments, and verified they are stable. • 5~6 VMs are optimum for one server computer. • We introduced Xen, which has live migration function, into beamline control system. • We have plan to apply Xen for more beamline. ICALEPCS 2005 in Geneva, Switzerland

29. Thank you for your attention. ICALEPCS 2005 in Geneva, Switzerland

30. Running on Xen secondly host Running on Xen primary host