Virtual Machine Queue Architecture Review

2. Virtual Machine Queue Architecture Review • Ali Dabagh • Architect • Windows Core Networking • alid@microsoft.com • Don Stanwyck • Sr. Program Manager • NDIS Virtualization • don.stanwyck@microsoft.com

3. Agenda • Introduction • Network device sharing in virtualization environment • Performance bottlenecks • Virtual Machine Queue (VMQ) Architecture • NIC embedded switch • Coexistence with offload technologies • Call to action • Q&A

4. Why VMQ? • Virtualization is now • Networking optimization supports virtualization • Virtualization impacts performance – VMQ helps overcome those effects • VMQ supports Live Migration • VMQ coexists with other optimizations • Broad industry support

5. Terminology • Virtual Service Provider (VSP) • A driver running in the parent partition • Provides a specific functionality to Virtual Machines • Network VSP (NVSP) implemented as an NDIS intermediate driver • Virtual Service Client (VSC) • A driver running in the child partition • Usually provides a device class functionality in child partitions • Communicates with its VSP counterpart in the parent partition • Network VSC (NVSC) implemented as an NDIS virtual miniport driver • Virtual Machine Bus (VMBus) • A communication bus for passing control and data messages between child and parent partition • VMBus Channel • A communication channel created between a VSC in child partition and VSP in parent partition

6. Synthetic Device Data Path Parent Partition VM1 VM2 Network Virtual Service Provider Ethernet Routing VLAN Filtering Data Copy TCP/IP TCP/IP Port 2 • Port 1 VM NIC 1 VM NIC 2 Miniport Driver NIC VM BUS

7. VMQ Concepts: Receive Queues • Receive Queues • Are allocated by Network Virtual Service Provider • Collect network packets that pass filter tests • Properties of a receive queue • Unique queue ID per miniport adapter • Interrupt processor affinity • Filters • Receive buffers • Default Queue • Always exists • Collects packets that are not collected by any other queue

8. VMQ Concepts: Receive Filters • Receive filters are set on receive queues • One or more “network header field tests” • Unique filter ID per miniport adapter • Specifying network headers in a filter • VMQ uses MAC header • Specifying fields within the headers • VMQ uses destination MAC address and VLAN tag • Setting multiple tests in one receive filter • A packet must pass all the network header field tests • Setting multiple filters on the same queue • Packet is accepted if it passes any of those filters • Specifying test operation and test value • VMQ uses “Equal” test operation

9. Receive Filters and Receive Queues

10. Packet Filtering Data Flow

11. Virtual Machine Queue Overview • Classifying received packets • Parsing the packets in hardware • Setting received packet queue ID • Grouping received packets • All packets in a receive indication call are for the same VM NIC • Applying VLAN filtering in hardware • Packets with invalid VLAN IDs are dropped by the NIC • Concurrent receive processing for multiple VMs • Every queue is serviced by a different processor • Avoiding copy from NIC receive buffers to VM address space • Using switch on the NIC to do route lookup on transmit

12. Synthetic Device Data Paths With VMQ Parent Partition VM1 VM2 Network Virtual Service Provider Ethernet TCP/IP TCP/IP Routing VLAN filtering Data Copy Port 2 • Port 1 VM NIC 1 VM NIC 2 Miniport Driver Q1 Q2 Default Queue VM BUS Switch/Routing unit NIC

13. Using VM Address Space for Receive Buffers Parent Partition VM1 VM2 Address Space NIC Queue 1 Queue 2 Address Space Address Space H D H D H D H D H D H D H D H D H D H D

14. NIC Embedded Switch • Receive queues are paired with transmit queues • Each queue pair is a switch port • No MAC address learning is required • The MAC address of each port is the one set on the receive queue • Switch inspects destination MAC address +VLAN ID of transmit packets • If they pass the filter set on a receive queue, DMA to that queue • Otherwise send the packet on the wire • Advantages in VM to VM communication • Avoids route lookup in software • Avoids a packet copy • Offload support in hardware • Advantages in VM to wire communication • Avoids route lookup in software

15. Coexistence with Other Offload Technologies • Transmit and receive checksum • Still needs to be calculated and verified • Should work for both VM-to-VM and VM-to-wire data flow • Large send offload • Should work for both VM-to-VM and VM-to-wire data flow • Jumbo frame support • Maximum packet size • Scatter/gather support for received packets • Chimney offload • If NIC supports both VMQ and Chimney, VMQ is used

16. Summary • VMQ improves network throughput • Distributes processing of network traffic for multiple VMs among multiple processors • VMQ reduces CPU utilization by: • Offloading packet classification to hardware • Avoiding network data copy • Avoiding route lookup on transmit path • VMQ is compatible with most task offloads • VMQ is secure • VMQ supports live migration

17. Call to Action • Virtualization is NOW • Network optimizations are crucial for virtualization support • These vendors are among those committed to supporting VMQ in their NICs: • Alacritech • Broadcom • Intel • Neterion • ServerEngines • Solarflare • Build/update all your drivers to be NDIS 6.20 compliant

18. Resources • Windows 7 WDK will be available as of WinHEC • Windows Logo Program Web site: http://www.microsoft.com/whdc/winlogo/default.mspx • NDIS 6 Feedback alias:ndis6fb@microsoft.com

19. Related Sessions