游清權

1. An Approach to Implementing the NIC Virtualization by the Hybrids of Single System Image and Hardware-assisted Virtualization Technologies 游清權 Distributed System Lab

2. Outline • Introduction • Background • Related works • Problem description • Implementation • Performance comparison • Conclusion Distributed System Lab

3. 1.Introduction • Business requirement have been long challenged the servers’ performance. • Distributed computing, high performance benefits versus price, scalability and transparency, reduce the cost of implementation. • To simplify the management , the single system image [1] is usually implemented in the distributed system • There are abundant I/O resources in the DVMM System , especially NIC, becoming an important index to reflect the servers’ performance. Distributed System Lab

4. 2-1. Background-Virtualization • Divide single physical resource into multiple copies of virtual execution environment • Both Intel and AMD have explicitly supported virtualization in their CPU designs, which are called VT-x [4] and SVM [5] respectively • Virtualization technologies on OS level • Full virtualization, • VMware workstation, KVM • Para-virtualization , • Xen [11], Denali [12]. • hardware-assisted virtualization [7] , • VMware ESX 3.0 [13] and Virtual Box [14]. Distributed System Lab

5. 2-2.SSI(single system image) • Hides the heterogeneous and distributed nature of theavailable resources and presents them to users • Applicationsand OS as a single unified computing resource [1]. • Advantages : Single entry point, single user interface, and singlememory space … etc. • SSIcan be implemented in various levels • Difficulties : Application< middleware< OS Distributed System Lab

6. 2-3. DVMM • Hybrid of virtualization and SSI system features • Features: Scalability, friendly programmability, high transparency and high availability. • Virtualization technology: • Easy to construct an illusion of the hardware resources • Eliminates the discrepancies among them, and thus reduces the difficulty to implement the SSI in large extend. • Makes it possible to collect and unify global computing resources • GOS utilize the advantages of both Distributed System Lab

7. DVMM • VMMs are interconnected by the gigabit Ethernet, distributed shared memory and abstract as an integration to provide the SSI • Composed: • Initialization, Instruction virtualization, interrupts virtualization, memory virtualization and communication module [2] Distributed System Lab

8. 3.Related works - VMware Distributed System Lab

9. 3.Related works - VMware • Full virtualization-Not only support the open source OS, but alsothe close source ones • NIC emulation is connected to host in two ways • it can be bridged to the same physical network as a physical NIC • It can be connected to a virtual network created on the host [6]. Distributed System Lab

10. 3.Related works - VMware Distributed System Lab

11. 3.Related works - Xen • Xen can support both the Para-virtualizationand hardware-assisted virtualization • Para-virtualization : • Need to make a modification to the GOS • Gain high NIC performance • Hardware-assistedvirtualization: • Needn’t to modify the GOS • Needs the supports of the AMD’s SVM or the Intel’s VT-x • Needs the frequent the VM entry and VM exit operations [4] • NIC performance: • VMware workstation< Hardware-assisted < Para-virtualization Distributed System Lab

12. 3.Related works - Xen Distributed System Lab

13. 4.Problem description • Abundant NIC resources are deployed in each node. • Goal :Implementing NIC virtualization lies inefficiently and transparently utilizing the NIC resourcesdeployed in each node of the DVMM system. • 4 problems need to be properly solved • How to • Detect ,collect and unify the global NIC • Intercept and emulate the NIC’s I/O instructions (AMD’s SVM) • Define proper data structures • Invoke the memory virtualization module construct the data path Distributed System Lab

14. 5.Implementation • NIC virtualization: • Resident level between the OS and the VMM • VMM makes it possible for the OS to utilize the global NIC resources transparently • Hardware-assisted virtualization makes it easy to detect the global NIC resources and catch the I/O operations performed by NICs • Cooperation between network card virtualization and memory virtualization module reduces the work of analyzing the details of SSI • Thus makes it possible for us to concentrate on the key techniques related to NIC virtualization Distributed System Lab

15. 5.Implementation Distributed System Lab

16. 5-1.Collecting and unifying the NIC resources • To solve the first problem ,worthy of having a probe into the principle of both emulated BIOS and PCI device. • The EBIOS [3] is used to substitute the traditional BIOS in the DVMM environment • GOS running on this environment utilizes it to probe the virtual hardware resources of the system • NICs are mainly catalogued into PCI type and BIOS offers access to the device configuration address space by reading and writing related registers in the PCI controller. Distributed System Lab

17. 5-1.Collecting and unifying the NIC resources • DVMM system, when a physical machine is power on • NIC information of which needs to be obtained and transferred to the NIC virtualization module for collection • After the global NIC information is ready • Utilize the virtualization technology to construct virtual network card and establish a mapping between the physical and virtual NIC • At last, as the global network card information is collected and unified, it needs to be written into the EBIOS Having owned those preconditions, when the guest OS boots, it can get the global NIC resources. Distributed System Lab

18. 5-1.Collecting and unifying the NIC resources Distributed System Lab

19. 5-2.Distinguish three types of I/O instructions • Three types of NIC I/O instructions must be distinguished • Non-memory-reference instruction (NMR for short) • Memory-reference instruction (MR for short) • DMA • Specification of the AMD on virtualization • Execute those instructions , they will induce the VM exits. • Specific exit reasons are recorded in VMCB structure • Easily obtain which type of I/O instruction induces the VM exits. Distributed System Lab

20. 5-3.The design of the core data structure • Types of NIC request need to be distinguished: local and remote • According to the • NIC request • The physical NIC itself whether located in the same physical node, • which kinds of I/O instruction induce VM exits Distributed System Lab

21. 5-4. Construct the data path • MR and DMA are both kinds of memory access operations • Two types of instruction require defining the interface between memory virtualization module and NIC virtualization module • Two rings structure: Req ring, and Ack ring • DSM provides two functions to access those rings,which are called send_pin_req and handle_ack_ring respectively Distributed System Lab

22. 5-4. Construct the data path Distributed System Lab

23. 5-4. Construct the data path • Two services are mainly provided to the upper level , which are Pin and Unpin requests, the use of those functions are to lock the related pages • DSM also provides two functions to execute those requests, which are called dsm_pin_pages and dsm_unpin_pages • When an operation needs the participation of the memory • Initiates a Pin request in order to lock pages. • Inserts the request in Req ring. • DSM executes the necessary operations to lock pages. • Inserts a Ack in Ack ring. Distributed System Lab

24. 5-5.NIC I/O instructions execution analysis in DVMM • GOS executes I/O instructions, by AMD spec ->sensitive instructions • The execution of them will induce the VM exits • Exit reason is encoded and recorded in VMCB data structure • DVMM updates its corresponding ioreq structure to symbol the exit reason NMR , MR , DMA ? Distributed System Lab

25. NMR • NMR instructions including: INT, OUT • Executions are mainly accesses to I/O ports in DVMM system, NO need to invoke the memory virtualization module If (I/O operation == local)DVMM will invoke local I/O device model to manipulate the physical NIC else DVMM will send the ioreq structure to remote that the physical NIC • When physical NIC has done, virtual interrupt occurs and can be captured by DVMM and NIC virtualization module updates the ioreq structure. • If this NMR operation is remote, the DVMM needs to send the updated ioreq back to local node. Distributed System Lab

26. NMR Distributed System Lab

27. MR • MR instructions including: INS, OUTS and MMIO • I/O operation must explicitly invoke the memory virtualization module to accomplish the task. Distributed System Lab

28. MR • Multiple pages may be involved in a single MR instruction • Needed to calculate the total numbers of page needed • Need to invoke the Pin requests to acquire the necessary pages • Related pages are successfully acquired, the DVMM will put the acknowledgement in ACK Ring, and NIC virtualization module can obtain the result by polling the ring • NIC virtualization module will get the necessary pages needed to finish the operation Distributed System Lab

29. MR • If this operation is local, • DVMM can access the virtual memory allocated by the memory virtualization model and plus invokes the I/O device model • else • the NIC virtualization module needs to send the ioreq structure and the locked page numbers for the remote node. • Once the physical NIC has finished the job, a virtual interrupt occurs and will be captured by DVMM. • The NIC virtualization module needs to invoke Unpin request to release the pages locked by the MR, and updates the ioreq structure. • If the MR I/O operation is remote, the DVMM needs to send the updated ioreq back to local node Distributed System Lab

30. DMA • DMA instruction is a typical memory-reference one, and it executes are basic the same as the MR type instructions • The main difference lies in the device model • There is a virtual DMA controller in the device model and a dedicated virtual memory used as virtual DMA buffer • Due to the reason the NIC virtualization module mainly needs to convey the related page indexes to the virtual DMA controller, and DMA controller will do the subsequent job to finish the task. Distributed System Lab

31. 6.Performance comparison • Implement a prototype of the DVMM NIC virtualization module • Experiments mainly make a comparison with the Virtual Multiprocessor [9] projects Distributed System Lab

32. 6.Performance comparison • From the table 1 • Overall I/O performance of the DVMM is better than Virtual Multiprocessor • Programs above are small, and they are not involved in the participation of disk. • Result mainly reflect the performance of the NIC I/O to a large extent Distributed System Lab

33. 6.Performance comparison • DVMM is a type of prototype one • Hope by continuous improving the system, we can obtain the final expectations • Moderate performance, good transparency and low cost of implementation Distributed System Lab

34. 7.Conclusion • Solving the problems of how to manage and utilize the global NIC resources in the DVMM • Propose a feasible approach to effectively collecting and unifying the NIC resource deployed among the nodes of the system. • Analyzes the key technologies on how to cooperate with the memory virtualization module to construct the basic data path • The approach is based on hardware-assisted virtualization and SSI Distributed System Lab