OpenNF : Enabling Innovation in Network Function Control

1. OpenNF: Enabling Innovation in Network Function Control Aditya Akella With: Aaron Gember, RaajayVishwanathan, ChaithanPrakash, Sourav Das, Robert Grandl, and Junaid Khalid University of Wisconsin—Madison

2. Network functions, or middleboxes Introduce custom packet processing functions into the network Load balancer SSL Gateway Firewall CachingProxy … IntrusionPrevention Common in enterprise, cellular, ISP networks Stateful: detailed book-keeping for network flows WAN optimizer Traffic scrubber [Sherry et al., SIGCOMM 2012]

3. State-of-the-art Network functions virtualization (NFV) • Lower cost • Easy prov., upgrades Software-defined networking (SDN) • Decouple from physical • Better performance, chaining Xen/KVM

4. NFV + SDN: distributed processing Dynamic reallocation to coordinate processing across instances Load balancing SDN Controller MBox MBox Extract maximal performance at a given $$

5. NFV + SDN: distributed processing Dynamic reallocation to coordinate processing across instances Key abstractions Load balancing SDN Controller 1. Elastic SDN Controller 2. Always updated 3. Dynamic enhancement MBox MBox Hand-off processing fora traffic subset MBox MBox MBox MBox Extract maximal performance at a given $$ MBox

6. What’s missing? The ability to simultaneously… Meet SLAs • E.g., ensure deployment throughput > 1Gpbs Ensure accuracy/efficacy • E.g., IDS raises alerts for all HTTP flows containing known malware packages Keep costs low, efficiency high • E.g., shut down idle resources when not needed … needs more control than NFV + SDN

7. Example Scaling in/outsustain thr’put at low $ Not moving flows  bottleneck persists SLAs! Naively move flows  no associated state Accuracy! Scale down: wait for flow drain out Efficiency! Transfer live state while updating n/w forwarding HomeUsers Firewall CachingProxy IntrusionPrevention Web Server

8. OpenNF Quick and safe dynamic reallocation of processing across NF instances Quick: Any reallocation decision invoked any time finishes predictably soon Safe: Key semantics for live state transfers No state updates missed, order preserved, etc. Rich distributed processing based applications toflexibly meet cost, performance, security objectives

9. Outline Overview and challenges Design • Requirements • Key ideas • Applications Evaluation

10. OpenNF Overview and challenges Elastic scaling Hot standby Dynamic enhancement Reallocationoperations NF APIs and control plane for joint control over internal NF state and network forwarding state OpenNF Controller Coordination w/ network SDN Controller State import/export MBox MBox

11. Challenges Overview and challenges 1: Many NFs, minimal changes • Avoid forcing NFs to use special state structures/ allocation/access strategies  Simple NF-facing API; relegates actions to NFs 2: Reigning in race conditions • Packets may arrive while state is being moved; Updates lost or re-ordered; state inconsistency Lock-step NF state/forwarding update 3: Bounding overhead • State transfers impose CPU, memory, n/w overhead Applications control granularity, guarantees

12. NF state taxonomy C1: Minimal NF Changes State created or updated by an NF applies to either a single flow or a collection of flows Classify state based on scope Flow provides a natural way for reasoning about which state to move, copy, or share Per-flow state Multi-flow state TcpAnalyzer Connection HttpAnalyzer ConnCount Connection TcpAnalyzer All-flows state Statistics HttpAnalyzer

13. API to export/import state C1: Minimal NF Changes Three simple functions: get, put, delete(f) • Version for each scope (per-, multi-, all-flows) • Filter fdefined over packet header fields NFs responsible for • Identifying and providing all state matching a filter • Combining provided state with existing state No need to expose internal state organization No changes to conform to a specific allocation strategy

14. Operations move flow-specific NF state at various granularities copy and combine, or share, NF state pertaining to multiple flows Semantics for move (loss-free, order-preserving), copy/share (various notions of consistency) “Reallocate port 80 to NF2”

15. Move Control Application SDN Controller move (port=80,Inst1,Inst2,LF&OP) forward(port=80,Inst2) OpenNF Controller getPerflow(port=80) putPerflow(ID1,Chunk1) [ID1,Chunk1] delPerflow(port=80) putPerflow(ID2,Chunk2) [ID2,Chunk2] Inst1 Inst2

16. Load-balanced network monitoring C2: Race conditions Movinglive state: some updates (packets) may be lost, or arrive out of order vulnerable.bro reconstruct MD5’s for HTTP responses • Not robust to losses weird.broSYN and data packets seen in unexpected order • Not robust to reordering HTTP req HTTP req vulnerable.bro weird.bro move

17. Lost updates during move C2: Race conditions • Packets may arrive during a move operation • Fix: suspend traffic flow and buffer packets • May last 100s of ms • Packets in-transit when buffering starts are dropped Loss-free: All state updates due to packet processing should be reflected in the transferred state, and all packets the switch receives should be processed B2 move(blue,Inst1 ,Inst2 ) Inst2 is missing updates B2 B1 R1 Inst1 Inst2 Key idea: Event abstraction to prevent, observe and sequence state updates

18. Loss-free move using events C2: Race conditions Stop processing; buffer at controller • enableEvents(blue,drop) on Inst1; • get/delete on Inst1 • Buffer events at controller • put on Inst2 • Flush packets in events to Inst2 • Update forwarding B2 B3 B1 B1,B2 B1,B2,B3 B1 drop R1 Inst1 Inst2 Controller B2

19. Re-ordering of updates C2: Race conditions Switch Inst2 Controller Inst1 5. Flush buffer 6. Issue fwd update B2 B2 B3 B2 B3 B4 B4 B3 B3 B3 Order-preserving: All packets should be processed in the order they were forwarded to the NF instances by the switch Two-stage update to track last packet at NF1

20. Order-preserving move C2: Race conditions Track last packet; sequence updates Flush packets in events to Inst2 w/ “do not buffer” enableEvents(blue,buffer) on Inst2 Forwarding update: send to Inst1& controller Wait for packet from switch (remember last) Forwarding update: send to Inst2 Wait for event from Inst2 for last Inst1 packet Release buffer of packets on Inst2 B3 B4 B3 B1,B2,B3, B4 buf B1 B1,B2 B1,B2,B3 drop R1 B2 B3

21. Bounding overhead C3: Applications Apps decide, based on NF type, objective: granularity of reallocation operations move, copy or share filter, scope guarantees desired move: no-guarantee, loss-free, loss-free + order-preserving copy: no or eventual consistency share: strong or strict consistency

22. Load-balanced network monitoring C3: Applications movePrefix(prefix,oldInst,newInst): copy(oldInst,newInst,{nw_src:prefix},multi) move(oldInst,newInst,{nw_src:prefix},per,LF+OP) while (true): sleep(60) copy(oldInst,newInst,{nw_src:prefix},multi) copy(newInst,oldInst,{nw_src:prefix},multi) HTTP req HTTP req scan.bro vulnerable.bro weird.bro scan.bro scan.bro vuln.bro weird.bro

23. Implementation Impl & Eval OpenNFController (≈4.7K lines of Java) • Written atop Floodlight Shared NF library (≈2.6K lines of C) Modified NFs (4-10% increase in code) • Bro (intrusion detection) • PRADS (service/asset detection) • iptables (firewall and NAT) • Squid (caching proxy) Testbed: HP ProCurve connected to 4 servers

24. Microbenchmarks: NFs Impl & Eval Cost grows with state complexity Serialization/deserializationcosts dominate

25. Microbenchmarks: operations Impl & Eval D = f(load,state,speed) • State: 500 flows in PRADS; Traffic: 5000 pkts/sec • Move per-flow state for all flows Packetsdropped!686 462 881 packetsin events 1120 packets buffered at Inst2 838 packetsin events + Copy (MF state) – 111ms Share (strong) – 13ms per pkt Guarantees come at a cost!

26. Macrobenchmarks: end-to-end benefits Impl & Eval Load balanced monitoring with Bro IDS • Load: replay cloud trace at 10K pkts/sec • At 180 sec: move HTTP flows (489) to new Bro • At 360 sec: move HTTP flows back to old Bro OpenNFscaleup: 260ms to move (optimized, loss-free) • Log entries equivalent to using a single instance VM replication: 3889 incorrect log entries • Cannot support scale-down Forwarding control only: scale down delayed by more than 1500 seconds

27. Wrap up! • OpenNF enables rich control of the packet processing happening across instances of an NF • Quick, key safety guarantees, • Low overhead, minimal NF modifications http://opennf.cs.wisc.edu

28. Backup

29. Copy and share C2: Race conditions Used when multiple instances need to access a particular piece of state Copy – eventual consistency • Issue once, periodically, based on events, etc. Share – strong • All packets reaching NF instances trigger an event • Packets in events are released one at a time • State is copied between packets

30. Example app: Selectively invoking advanced remote processing C3: Applications HTTP req HTTP req Internet scan.bro vulnerable.bro weird.bro detect-MHR.bro scan.bro vulnerable.bro weird.bro checks md5sum of HTTP reply enhanceProcessing(flowid,locInst): move(locInst,cloudInst,flowid,per,LF) No need for: (1) order-preservation (2) copying multi-flow state ! Enterprise n/w

31. Existing approaches • Control over routing (PLayer, SIMPLE, Stratos) • Virtual machine replication • Unneeded state => incorrect actions • Cannot combine => limited rebalancing • Split/Merge and Pico/Replication • Address specific problems => limited suitability • Require NFs to create/access state in specificways => significant NF changes

32. Controller performance Improve scalability with P2P state transfers

33. Macrobenchmarks: Benefits of Granular Control Impl & Eval Two clients make HTTP requests • 40 unique URLs Initially, both go to Squid1 20s later  reassign client 1 to Squid2 Granularitiesof copy