Hosting Virtual Networks on Commodity Hardware

1. Hosting Virtual Networks on Commodity Hardware VINI Summer Camp

2. Decouple Service from Infrastructure • Service: “slices” of physical infrastructure • Applications and networks that benefit from • Flexible, custom topologies • Application-specific routing • Infrastructure: needed to build networks

3. Fixed Physical Infrastructure

4. Shared By Many Parties

5. Network Virtualization: 3 Aspects • Host: Divide the resources of one physical node into the appearance of multiple distinct hosts • Network stack: Give each process its own interfaces, routing table, etc. • Links: Connect two nodes by composing underlying links

6. Why Virtual Networks • Sharing amortizes costs • Enterprise network or small ISP does not have to buy separate routers, switches, etc. • Large ISP can easily expand to new data center without buying separate equipment • Programmability and customizability • Testing in realistic environments

7. Why Commodity Hardware • Lower barrier to entry • Servers are inexpensive • Routing (e.g., Quagga), and forwarding (e.g., Click) software is open source (free) • No need for specialized hardware • Open-source routing software: Quagga, etc. • Network processors can be hard to program • Easy adaptation of physical infrastructure • Expansion is easy: buy more servers

8. Commercial Motivation:Logical Routers • Consolidation • PoP and Core • Simpler physical topology • Fewer physical interconnection • Application-Specific Routing • PoP and Core • Simpler physical topology • Fewer physical interconnection • Wholesale Router Market • Proof-of-Concept Deployment

9. Other Beneficiaries • Interactive applications: require application-specific routing protocols • Gaming • VoIP • Critical services: benefit from custom data plane • Applications that need more debugging info • Applications with stronger security requirements

10. Requirements • Speed: Packet forwarding rate that approach that of native, in-kernel • Flexibility: Support for custom routing protocols and topology • Isolation: Separation of resource utilization and namespaces

11. Host Virtualization • Full virtualization: VMWare Server, KVM • Advantage: No changes to Guest OS, good isolation • Disadvantage: Slow • Paravirtualization: Xen, Viridian • OS-Level Virtualization: OpenVZ, VServers, Jail • Advantage: Fast • Disadvantage: Requires special kernel, less isolation

12. Network Stack Virtualization • Allows each container to have its own • Interfaces • View of IP address space • Routing and ARP tables • VServer does not provide this function • Solution 1: Patch VServer with NetNS • Solution 2: OpenVZ • VServer is already used for PlanetLab

13. Link Virtualization • Containers need Ethernet connectivity • Routers expect direct Ethernet connections to neighbors • Linux GRE tunnels support only IP-in-IP • Solution: Ethernet GRE (EGRE) tunnel

14. Synthesis • Tunnel interface outside of container • Permits traffic shaping outside of container • Easier to create point-to-multipoint topology • Need to connect tunnel interface to virtual interface

15. Connecting Interfaces: Bridge • Linux bridge module: connects virtual interface with the tunnel interface • speed suffers due to bridge table lookup • allows point-to-multipoint topologies

16. Optimization: ShortBridge • Kernel module used to join virtual interface inside the container with the tunnel interface • Achieves high packet forwarding rate

17. Evaluation • Forwarding performance • Packets-Per-Second • Source->Node-Under-Test->Sink • Isolation • Jitter/loss measurements with bursty cross traffic • Scalability • Forwarding performance as the number of containers grow • All tests were conducted on Emulab • 3GHz CPU, 1MB L2 Cache, 800MHz FSB, 2GB 400MHz DDR2 RAM

18. Forwarding Performance - Click • Minimal Click configuration • Raw UDP receive->send • Higher jitter • ~80’000PPS

19. Forwarding Performance - Bridged Allows more flexibility through bridging ~250’000PPS

20. Forwarding Performance – Bridged w/o Tunneling Xen: often crashes, ~70’000PPS OpenVZ: ~300’000PPS NetNS: ~300’000PPS

21. Forwarding Performance – Spliced Avoids bridging overhead Point-to-Point topologies only ~500’000PPS

22. Forwarding Performance - Direct No resource control ~580’000PPS

23. Overall Forwarding Performance

24. Forwarding for Different Packet Sizes

25. Isolation • Setup: • 5 nodes. 2 pairs of source+sink • 2 NetNS containers in spliced mode • pktgen used to generate cross flow • iperf measures jitter on another flow • Step function • CPU utilization < 99%: no loss, 0.5ms jitter • CPU utilization ~> 100%: loss, 0.5ms jitter for delivered packets

26. Scalability Test Setup

27. Scalability Results

28. Tradeoffs • Bridge vs. Shortbridge • Bridge enables point-to-multipoint • Shortbridge is faster • Data-plane flexibility vs. Performance • Non-IP forwarding requires user-space processing (Click)

29. Future Work • Resource allocation and scheduling • CPU • Interrupts/packet processing • Long-running deployment on VINI testbed • Develop applications for the platform

30. Questions • Other motivations/applications? • Other aspects to test? • Design alternatives?