Advanced Topics in Networking: SDN/NFV Overview

1. Advanced Topics in Networking: SDN/NFV Overview Dr. Tal Anker | Distinguished Architect, Mellanox BGU Course, December 2017

2. Agenda • SDN • Glimpse at NFV • Extra’s (if time allows): • Virtual interfaces overview • Virtual switching overview • OpenStack Networking Support • Tunneling • Network Nodes

3. SDN

4. LIMITATIONS OF EXISTING NETWORKS (Legacy) • Difficult to perform real world experiments on large scale production networks. • Research stagnation-huge costly equipment to be procured and networks to be setup by each team for research • Networks have remained the same for many years • Rate of innovation in networks is slower as protocols are defined in isolation-lack of high level abstraction. • Closed systems • Hard to collaborate meaningfully due to lack of standard open interfaces. • Vendors starting to open-up but not meaningfully. • Innovation is limited to vendor/vendor partners • Huge barriers for new ideas in networking.

5. Feature Million of linesof source code Billions of gates Limitations of Current Networks (II) Many complex functions baked into infrastructure • OSPF, BGP, multicast, differentiated services,Traffic Engineering, NAT, firewalls, … Feature Operating System Specialized Packet Forwarding Hardware Cannot dynamically change according to network conditions

6. SOFTWARE DEFINED NETWORKING • Data Plane: processing and delivery of packets • Based on state in routers and endpoints • E.g., IP, TCP, Ethernet, etc. • Control Plane: establishing the state in routers • Determines how and where packets are forwarded • Routing, traffic engineering, firewall state, … • Separate control plane and data plane entities • Have programmable data planes—maintain, control and program data plane from a central entity i.e. control plane software called controller. • An architecture to control not just a networking device but an entire network.

7. Software-DefinedNetworking (SDN) Protocols Protocols Control Programs Control Programs Global Network View Network Operating System Control via open forwarding interface Packet forwarding

8. ARCHITECTURE OF SDN In the SDN architecture, the control and data planes are decoupled, network intelligence and state centralized, and the underlying network infrastructure is abstracted from the applications.

9. SDNLAYERS • Infrastructure layer: it is the foundation layer consists of both physical and virtual network devices such as switches and routers. All the network devices will implement a control protocol agent, e.g. OpenFlow, to implement traffic forwarding rules. • Control layer: This layer consists of a centralized control plane that is decoupled from the physical infrastructure to provide centralized global view to entire network. The layer will use OpenFlow protocol to communicate with below layer i.e. infrastructure layer. (E.G. ODL, ONOS, etc.) • Application layer: it consists of network services, application and orchestration tools that are used to interact with control layer. It provide an open interface to communicate with other layers in the architecture.

10. OPENFLOW PROTOCOL • OPENFLOW is an open API that provides a standard interface for programming the data plane switches. It is a protocol for remotely controlling the forwarding table of a switch or router and is one element of SDN. • It is implemented on Ethernet switches to allow the forwarding plane i.e. data plane to be managed by a controller present on control plain in SDN architecture. OpenFlow based controllers will discover and maintain an inventory of all the links in the network and then will create and store all possible paths in entire network. • OpenFlow protocol can instruct switches and routers to direct the traffic by providing software-based access to flow tables that can be used to quickly change the network layout and traffic flows as per users requirements.

11. OpenFlow OpenFlow Controller OpenFlow Protocol (SSL/TCP) Control Path OpenFlow Data Path (Hardware)

12. OPENFLOW SWITCH AND CONTROLLER • An OpenFlow Switch contain one or more flow tables ­that implement packet lookups and forwarding, and an OpenFlow channel to link to an external controller .The switch interconnects with the controller and the controller directs the switch using the OpenFlow protocol. • The controller can delete, add or update flow entries in flow tables existing in the switch, both reactively i.e. in response to packets or proactively, using the OpenFlow protocol. • Controller make this decision based on policies set by administrator or depending on the conditions of the network and the decision it makes is forwarded to flow table entries of all the switches in the network.

13. SDN - Status • The main deployment of SDN in general and OpenFlow in particular, is Cloud • The OVS (Open Virtual Switch) which is the prevalent virtual switch in the cloud deployments, is controlled via OpenFlow • The main controller is ODL (Open Daylight) but that starts to change (ONOS) • The main usage for the OF is to: • Build and maintain the flow switching rules • E.g. OVS gets its rules from ODL controller and from CLI • Create VTEP (VxLAN tunnel termination points) • E.g. Nuage SDN controller • WAN optimizations • OF is not enough… • E.g. One need to manage port status, LAG configuration and many other aspects of a network element • OVSDB, SNMP, Netconf – all of these are complementary protocols • Network Functionk Virtualization (NFV) – heavily relies on SDN, OF etc.

14. NFV (BRIEF)

15. The NFV Concept v v Traditional Network Model: APPLIANCE APPROACH Virtualised Network Model: VIRTUAL APPLIANCE APPROACH A means to make the network more flexible and simple by minimising dependency on HW constraints VIRTUAL APPLIANCES DPI GGSN/SGSN CG-NAT BRAS DPI PE Router Firewall BRAS GGSN/SGSN ORCHESTRATION, AUTOMATION & REMOTE INSTALL PE Router STANDARD HIGH VOLUME SERVERS Session Border Controller CG-NAT Firewall • Network Functions are based on specific HW&SW • One physical node per role • Network Functions are SW-based over well-known HW • Multiple roles over same HW Source: Adapted from D. Lopez Telefonica I+D, NFV

16. Target Independent Software Vendors Classical Network Appliance Approach WAN Acceleration Message Router Session Border Controller CDN Orchestrated, automatic & remote install. CarrierGrade NAT Firewall DPI Tester/QoE monitor Standard High Volume Servers Standard High Volume Storage Radio AccessNetwork Nodes SGSN/GGSN PE Router BRAS Standard High Volume Ethernet Switches • Fragmented non-commodity hardware. • Physical install per appliance per site. • Hardware development large barrier to entry for new vendors, constraining innovation & competition. Network Virtualization Approach

17. Network Functions Virtualization • Network Functions Virtualization is about implementing network functions in software - that today run on proprietary hardware - leveraging (high volume) standard servers and IT virtualization • Supports multi-versioning and multi-tenancy of network functions, which allows use of a single physical platform for different applications, users and tenants • Enables new ways to implement resilience, service assurance, test and diagnostics and security surveillance • Provides opportunities for pure software players • Facilitates innovation towards new network functions and services that are only practical in a pure software network environment • Applicable to any data plane packet processing and control plane functions, in fixed or mobile networks • NFV will only scale if management and configuration of functions can be automated • NFV aims to ultimately transform the way network operators architect and operate their networks, but change can be incremental

18. Benefits & Promises of NFV (i) Reduced equipment costs (CAPEX) through consolidating equipment and economies of scale of IT industry. Increased speed of time to market by minimising the typical network operator cycle of innovation. Availability of network appliance multi-version and multi-tenancy, allows a single platform for different applications, users and tenants. Enables a variety of eco-systems and encourages openness. Encouraging innovation to bring new services and generate new revenue streams.

19. Benefits & Promises of NFV (ii) • Flexibility to easily, rapidly, dynamically provision and instantiate new services in various locations • Improved operational efficiency • By taking advantage of the higher uniformity of the physical network platform and its homogeneity to other support platforms. • Software-oriented innovation to rapidly prototype and test new services and generate new revenue streams • More service differentiation & customization • Reduced (OPEX) operational costs: reduced power, reduced space, improved network monitoring • IT-oriented skillset and talent Source: Adapted from D. Lopez Telefonica I+D, NFV

20. NFV vs SDN • NFV: re-definition of network equipment architecture • NFV was born to meet Service Provider (SP) needs: • Lower CAPEX by reducing/eliminating proprietary hardware • Consolidate multiple network functions onto industry standard platforms • SDN: re-definition of network architecture • SDN comes from the IT world: • Separate the data and control layers, while centralizing the control • Deliver the ability to program network behavior using well-defined interfaces

21. NFV Concepts • Network Function (NF): Functional building block with a well defined interfaces and well defined functional behavior • Virtualized Network Function (VNF): Software implementation of NF that can be deployed in a virtualized infrastructure • VNF Set: Connectivity between VNFs is not specified, e.g., residential gateways • VNF Forwarding Graph: Service chain when network connectivity order is important, e.g., firewall, NAT, load balancer • NFV Infrastructure (NFVI): Hardware and software required to deploy, mange and execute VNFs including computation, networking, and storage.

22. High-level Architecture • NFV framework consists of: • NFV infrastructure • Network functions • NFV Orchestration software

23. Functionality breakdown • NFV infrastructure • Virtual compute, storage and network. • Network function layer • Virtual Network Function (VNF) along with its individual management system. • NFV management software • NFV infrastructure manager that manages the virtual infrastructure, • VNF manager • Orchestration software that manages the VNF lifecycle.

24. VNF Forwarding Graph - Logical View Note – a VNF forwarding Graph “Link” is a tunnel– e.g. Vxlan tunnel, NSH etc. (Slide 43)

25. VNF Forwarding Graph - Physical View

26. Extras

27. Cloud Computing and Networking • Cloud - scalable computing resources (CPU, Memory, Storage), services and applications on demand over the network with pay as you grow model • Cloud networking is Virtualized. Network has to scale, isolate, perform, gauge and provide services vCPU vMem vStor vCPU vCPU vCPU vCPU vCPU vCPU vCPU vMem vStor vCPU vCPU vCPU vCPU vCPU vCPU vNet In the next slides we will concentrate on the networking components used in cloud computing (and if time allows –we will briefly discuss the ways to accelerate cloud networking performance)

28. Virtual Interfaces – Device Emulation • Host emulates a complete HW device • E.g., Intel e1000 NIC • Guest runs unmodified driver • Pros • No need to install special drivers in guests • Transparentmigration • Unlimited virtual interfaces • Cons • Slow • Emulation exists only for very simple devices • High overhead Host Qemu process VM User Kernel e1000 driver e1000 emulator User Kernel SW Switch phys netdev macvtap netdev NIC

29. Virtual Interfaces – Para-Virtualization • Host exposes a virtual “SW-friendly” device • E.g., virtio-net • VM runs special device driver • Host emulates device back-end • Pros • Decent performance • Transparent migration • Unlimited virtual interfaces • Cons • Simple devices only Host Qemu process VM User Kernel virtio-net virt-io emulator User Kernel SW Switch phys netdev macvtap netdev NIC

30. Virtual Interfaces – Accelerated Para-Virtualization • Same para-virtual control interface • Fast path offloaded to host kernel • vhost_net - part of the Virtio driver moved from the userspace into the kernel. • Reduces copy operations, lowers latency and CPU usage. Host Qemu process VM User Kernel virtio-net virt-io control User Kernel SW switch vhost-net phys netdev macvtap netdev NIC

31. Virtual Interfaces – Physical Device Pass-Through • Hosts grants guest direct access to a physical device • Security and isolation still maintained • PCI configuration space is virtualized • IOMMU governs DMA access • VM runs standard device driver • Pros • Near-native performance • VMs can use any device that is passed to them • Cons • No transparent migration • Very limited scalability (physical devices are not shared) Host Qemu process VM User Kernel HW driver User Kernel HW driver NIC 1 NIC 2 Switch

32. Virtual Interfaces – Virtual Device Pass-Through (SR-IOV) • Single Root I/O Virtualization (SR-IOV) • Hosts grants guest direct access to a virtual device • Security and isolation still maintained • PCI configuration space is virtualized • IOMMU governs DMA access • VM runs device driver for virtual function • Pros • Near-native performance • High scalability (128-256 VFs) • Cons • No transparent migration Host Qemu process VM User Kernel VF driver User Kernel PF driver NIC Virtual Function Phyiscal Function Embedded switch Switch

33. Virtual Switching - Overview • Required for VM to VM communication (same host) • External switch (802.1Qbg) • Classical switch does not switch traffic back to the source port (thus required a change, “hairpin”) • Unnecessary BW consumption on the NIC-Switch link • Virtual switch • Linux bridge • OVS (and other industry variants) • NIC based hardware embedded switch (eswitch) • Accelerated OVS over DPDK • All of the above holds also for Containers

34. Linux Bridge • Compliance to 802.1D • Emulates a legacy Layer 2 switch in software • FDB, unknown destination flooding (FDB miss), spanning tree, etc. • Linux Bridge vs a regular Host • Bridge places NIC HW in promiscuous mode • Further offloads available (e.g. MAC, VLAN demux) • The Bridge switches traffic between • Local VMs • VM to the adjacent switch through the Host NIC Host VM VM eth0 eth0 User Kernel Linux Bridge NIC driver NIC

35. OVS • Open-vSwitch • L2,3,4 and more • Openflow based • FastpathvsSlowpath • Fastpath – flows cache • Consulted on every received packet (multi threaded) • Cache miss  packet handover to user space • Slowpath - ovs-vswitchd – user space daemon • Openflow pipeline executed on the packet  install a new entry in the cache for subsequent packets • The original packet is handed back to the fastpath module Host VM VM eth0 eth0 ovsdb • ovs-vswitchd User Kernel Slow path openvswitch.kmod NIC driver NIC

36. OVS – Cont’d • Overall state is kept in local DB in user space • L2 FDB, L3 fwd state, QoS policy, etc. • Encap/Decap for tunneling • Overwrite capabilities (change packet fields) • Performance boost • HW offload – e.g. tunneling, packet processor, etc. • User space with DPDK support OVS VXLAN Tunneling Support VM VM VM VM OS OS OS OS vTap vTap vTap vTap BR0 BR1 VXLAN Overlay Open vSwitch (OVS) VNI100 VNI300 UDP IP VxLAN Overlay (tenant) networks Underlay Network (Layer 2 or Layer 3)

37. NIC Hardware Embedded Switch – OVS Control Path • Enables switching between virtual SRIOV functions • Utilize OVS control plane • Fast path in HW embedded switch • L0 cache like

38. User Mode OVS and DPDK • Move OVS data path from Kernel to user space • User space process uses the DPDK libraries to efficiently process packets • VMs connect from Virtio through VHOST (in the VM kernel) to the Virtio of the DPDK process Host Qemu Qemu VM VM Virtio Driver Virtio Driver Vhost* Vhost* Usermode OVS DPDK Virtio PMD Virtio PMD DPDK PMD User Space Kernel Physical NIC

39. OpenStack Networking before Neutron • Nova offers "networking as a service" in OpenStack (nova-network) • Note: It was the only offer before Quantum (old Neutron project name) • Nova-network is still present today, and can be used instead of Neutron • Points to keep in mind: • Limited Network Topologies supported • Only Flat, • Flat DHCP • and VLAN DHCP

40. OpenStack Neutron - Advantages • Neutron improves nova-network in multiple areas • Larger number of Network Topologies and services supported • L3: Self-Tenant provisioning • Security (ingress + egress rules support) • LBaSS • VPNaSS (coming)

41. Openstack support - Nova & Neutron Flows • Nova compute send port create with direct/normal VNIC_TYPE (and PCI address if Direct) to neutron server • Mechanism driver (e.g. OVS mechanism driver) in neutron server binding VNIC_TYPE (Direct/Normal) and return appropriate VIF type • Neutron server send VIF type to nova compute. • Neutron server send RPC call to neutron agent to apply appropriate rule (according to network type) on the VNIC Neutron Server (1) Nova Compute (2) ML2 Plugin (3) Mechanism driver (4) Neutron Agent

42. Overlay Networks (Tunnels etc.) • Customers may want to use the full 4K vlans. • Even if one segregates the vlan space between customers, there are not enough VLANs! • Each TOR can be connected to many servers • Each server may run many VMs • Each VM may use many VLANs • VMs of a specific tenant may reside in different racks, or even in different data centers • Tunnels are used to group multiple VLANs into one (logical) segment, identified by a tenant ID (in VXLAN it is VNI (VxLAN Network Identifier), and in NVGRE its TNI (Tenant Network Identifier). Both use 24bits for the ID • Brief comparison • VxLAN uses UDP port pairs to correspond to Src/Dst mac pairs • Transparent to load balancers etc • VxLan was recently updated to support multiple passenger packet types (VxLAN-gpe) and not only L2 Ethernet • IETF draft-ietf-nvo3-vxlan-gpe-01.txt • NVGRE uses GRE encapsulation • Requires load balancers to hash on GRE header. Not common.

43. Overlay Networks (VXLAN/NVGRE/GENEVE) & Its Acceleration Virtual Overlay Networks Simplifies Management and VM Migration NICs Overlay Accelerators Enable Bare Metal Performance NVGRE/VXLAN Overlay Networks Virtual Domain 1 Virtual Domain 2 Virtual Domain 3 Virtual View Physical View Server Server VM5 VM6 VM7 VM8 VM1 VM2 VM3 VM4 Network Underlay Infrastructure Overlay Network Virtualization: Isolation, Simplicity, Scalability

44. VxLAN and NVGRE Tunnels (L2 over L3) • VTEP – Virtual Tunnel End Point – usually resides within the virtual switch • Before sending the packet, the flow spec is examined and the result may be to switch, or to do Encap (or decap) and then to switch the modified packet • Encap – Add proprietary tunnel header and prepend with external IP header destined to the receiving VTEP (the other “side” of the tunnel) • BUM (Broadcast, Unknown & Multicast) • If a VM is not local on the server, the destination VTEP needs to be known – or learned (i.e. mapping between destination L2 and the destination VTEP). • L2 domain -> Multicast

45. Turbocharge Overlay Networks with Modern NICs • Advantages of Overlay Networks • Simplification • Automation • Scalable • Problem: Performance Impact!! • Overlay tunnels add network processing • Limits bandwidth • Consumes CPU • Solution: • Overlay Network Accelerators in NIC • Penalty free overlays at bare-metal speed • HW encap/decap (A.K.A Virtual Switch Offloads)

46. Network Nodes • Two VMs tenants (different L3 domains) needs to communicate • Each VM communicates with its default gateway as in a legacy network • However, the L3 node has per tenant IP interface and is reachable via standard cloud L2 “VPN” – Tunnels • E.g., a VxLAN tunnel between VM1 to the default gateway, which terminates the tunnel (VTEP), and performs routing decision • In the case where VM2 is of a different tenant in the cloud, the router is connected to VM2 using a tunnel (thus VTEP routing) • An L3 network node can send traffic to “Bare metel” (or legacy) machines (no tunneling) • A network node in OPSK can be a SW server (linux) or even a 3rd party device (HW based Switch Router with VTEP capabilities) • Similar to the above – FW or LB (which OPSK provide as a service)

47. User-space Networking - Overview • HW managed by kernel • Applications create resource via system calls, e.g. • Queues (descriptor rings) • Registering memory buffers • Data-path bypasses kernel • Shared queues between application and HW • HW accesses registered buffers directly • Direct signaling mechanism (“doorbells”) • Direct completion detection • In memory polling • Can also register for event (interrupt) • Multiple HW resources • No need for locking if resources are accessed by a single thread • Efficient • Asynchronous progress • Zero copy Application / Middleware Registered Memory Thread Thread Comp Q Send Q Recv Q Access library HW user-space driver User Kernel HW kernel driver NIC

48. User-space Networking – Packet Interfaces • Verbs Raw Ethernet Queues as an example • Basic data path operations • Post packet buffers to be sent out • Post buffers to receive packets • Poll for completions • Checksum offloads for TCP/UDP • Insert checksum on TX • Validate checksum on RX • VLAN insertion/stripping • Receive-side scaling (RSS) • Distribute incoming packets into multiple queues • Distribution is semi-random (hash based) • Flow steering • Deterministic steering of specific flows to specific RQs • Deliver very high packet rate to the application • E.g., 25Mpps for 64b packets Application / Middleware Registered Memory Thread Thread Comp Q Send Q Recv Q Access library HW user-space driver User Kernel HW kernel driver NIC

49. User-Space Middleware – Data Plane Development Kit (DPDK) • A set of data plane libraries and network interface controller drivers for fast packet processing • Functionality • Memory/Queue/Buffer managers (Memory pools, pre-allocations of buffers, lockless queues) • Poll-Mode-Driver - PMD (w/out asynchronous interrupt base signaling – for speedup) • Packet flow classification • L3 (LPM – V4/V6) • 5 tuple (via Hash) • L2 (FDB via Hash) • Encryption (via Extensions) • Intel advanced Encryption Standard, IPsec • Minimal overhead • Leverages user-space packet interfaces

50. Additional Slides