An Experimenter’s Guide to OpenFlow

1. An Experimenter’s Guide to OpenFlow GENI Engineering Workshop June 2010 Rob Sherwood (with help from many others)

2. Talk Overview • What is OpenFlow • How OpenFlow Works • OpenFlow for GENI Experimenters • Deployments • Next Session: OpenFlow “Office Hours” • Overview of available software, hardware • Getting started with NOX

3. What is OpenFlow?

4. Short Story: OpenFlow is an API • Control how packets are forwarded • Implementable on COTS hardware • Make deployed networks programmable • not just configurable • Makes innovation easier • Goal (experimenter’s perspective): • No more special purpose test-beds • Validate your experiments on deployed hardware with real traffic at full line speed

5. How Does OpenFlow Work?

6. Ethernet Switch

7. Control Path Control Path (Software) Data Path (Hardware)

8. OpenFlow Controller OpenFlow Protocol (SSL/TCP) Control Path OpenFlow Data Path (Hardware)

9. MAC src MAC dst IP Src IP Dst TCP sport TCP dport * * * 5.6.7.8 * * port 1 Action OpenFlow Flow Table Abstraction Controller PC OpenFlow Firmware Software Layer Flow Table Hardware Layer port 2 port 1 port 3 port 4 5.6.7.8 1.2.3.4

10. OpenFlow BasicsFlow Table Entries Rule Action Stats Packet + byte counters • Forward packet to port(s) • Encapsulate and forward to controller • Drop packet • Send to normal processing pipeline • Modify Fields Eth type Switch Port IP Src IP Dst IP Prot TCP sport TCP dport VLAN ID MAC src MAC dst + mask what fields to match

11. Examples Switch Port Switch Port Switch Port MAC src MAC src MAC src MAC dst MAC dst MAC dst Eth type Eth type Eth type VLAN ID VLAN ID VLAN ID IP Src IP Src IP Src IP Dst IP Dst IP Dst IP Prot IP Prot IP Prot TCP sport TCP sport TCP sport TCP dport TCP dport TCP dport Forward Action Action Switching 00:1f:.. * * * * * * * * * port6 Flow Switching port3 00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6 Firewall * * * * * * * * * 22 drop

12. Examples Switch Port Switch Port MAC src MAC src MAC dst MAC dst Eth type Eth type VLAN ID VLAN ID IP Src IP Src IP Dst IP Dst IP Prot IP Prot TCP sport TCP sport TCP dport TCP dport Action Action Routing * * * * * * 5.6.7.8 * * * port6 VLAN Switching port6, port7, port9 vlan1 00:1f.. * * * * * * * *

13. OpenFlow UsageDedicated OpenFlow Network Statistics Statistics Statistics Action Action Action Rule Rule Rule Aaron’s code OpenFlow Protocol Controller PC OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlowSwitch.org

14. OpenFlow Road Map • OF v1.0 (current) • bandwidth slicing • match on Vlan PCP, IP ToS • OF v1.1: Extensions for WAN, late 2010 • multiple tables: leverage additional tables • tags, tunnels, interface bonding • OF v2+ : 2011? • generalized matching and actions: an “instruction set” for networking

15. What OpenFlow Can’t Do (1) • Non-flow-based (per-packet) networking • ex: sample 1% of packets • yes, this is a fundamental limitation • BUT OpenFlow can provide the plumbing to connect these systems • Use all tables on switch chips • yes, a major limitation (cross-product issue) • BUT an upcoming OF version will expose these

16. What OpenFlow Can’t Do (2) • New forwarding primitives • BUT provides a nice way to integrate them • New packet formats/field definitions • BUT plans to generalize in OpenFlow (2.0) • Setup new flows quickly • ~10ms delay in our deployment • BUT can push down flows proactively to avoid delays • Only a fundamental issue when delays are large or new flow-rate is high

17. OpenFlow for Experimenters • Experiment Setup • Design considerations • OpenFlow GENI architecture • Limitations

18. Why Use OpenFlow in GENI? • Fine-grained flow-level forwarding control • e.g., between PL, ProtoGENI nodes • Not restricted to IP routes or Spanning tree • Control real user traffic with Opt-In • Deploy network services to actual people • Realistic validations • by definition: runs on real production network • performance, fan out, topologies

19. Experiment Setup Overview Step 1: Write/Configure/Deploy OpenFlow controller • Each controller implements per-experiment custom forwarding logic • Write your own or download pre-existing Step 2: Create Slice and register experiment • Configure per-experiment topology, queuing • restricted to subset of real topology • Specify desired user traffic: e.g., tcp.port=80 Step 3: Control the traffic of Users that opt-in to Your experiment • Users opt-in via the Opt-In Manager website • Reserving a compute node makes the experimenter a user on the network

20. Experiment Design Decisions • Forwarding logic (of course) • Centralized vs. distributed control • Fine vs. coarse grained rules • Reactive vs. Proactive rule creation • Likely more: open research area

21. Centralized vs Distributed Control Centralized Control OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlow Switch Distributed Control Controller Controller Controller Controller

22. Flow Routing vs. AggregationBoth models are possible with OpenFlow Flow-Based Every flow is individually set up by controller Exact-match flow entries Flow table contains one entry per flow Good for fine grain control, e.g. campus networks • Aggregated • One flow entry covers large groups of flows • Wildcard flow entries • Flow table contains one entry per category of flows • Good for large number of flows, e.g. backbone

23. Reactive vs. Proactive Both models are possible with OpenFlow Reactive First packet of flow triggers controller to insert flow entries Efficient use of flow table Every flow incurs small additional flow setup time If control connection lost, switch has limited utility • Proactive • Controller pre-populates flow table in switch • Zero additional flow setup time • Loss of control connection does not disrupt traffic • Essentially requires aggregated (wildcard) rules

24. Examples of OpenFlow in Action Summary of demos in next session VM migration across subnets energy-efficient data center network WAN aggregation network slicing default-off network scalable Ethernet scalable data center network load balancing formal model solver verification distributing FPGA processing

26. Opt-In Manager • User-facing website + List of experiments • User’s login and opt-in to experiments • Use local existing auth, e.g., ldap • Can opt-in to multiple experiments • subsets of traffic: Rob & port 80 == Rob’s port 80 • Use priorities to manage conflicts • Only after opt-in does experimenter control any traffic

27. Deployments

28. OpenFlow Deployment at Stanford • Switches (23) • APs (50) • WiMax (1) 34

29. Live Stanford Deployment Statistics http://yuba.stanford.edu/ofhallway/wide-right.htmlhttp://yuba.stanford.edu/ofhallway/wide-left.html

30. GENI OpenFlow deployment (2010) 8 Universities and 2 National Research Backbones

31. Three EU Projects similar to GENI:Ophelia, SPARC, CHANGE Pan-European experimental facility • L2 Packet • Emulation • Wireless • Content delivery • L2 L3Packet • Optics • Content delivery • L2 Packet • Wireless • Routing • L2 Packet • Optics • Content delivery • L2 Packet • Shadow networks 37

32. Other OpenFlow deployments • Japan - 3-4 Universities interconnected by JGN2plus • Interest in Korea, China, Canada, …

33. An Experiment of OpenFlow-enabled Network (Feb. 2009 - Sapporo Snow Festival Video Transmission) KOREA OpenFlow Network Seoul OpenFlow Switch (Linux PC) Suwon NOX OpenFlow Controller VLAN on KOREN Data Transmission Daejeon TJB Controller TJB Broadcasting Company Deagu Gwangju Busan Sapporo Studio Japan OpenFlow Network Sapporo Japan A video clip of Sapporo snow festival is transmitted to TJB (Daejeon, KOREA) via ABC server (Osaka, JAPAN). Server Asahi Broadcasting Cooperation (ABC) at Osaka, Japan

34. Highlights of Deployments Stanford deployment McKeown group for a year: production and experiments To scale later this year to entire building (~500 users) Nation-wide trials and deployments 7 other universities and BBN deploying now GEC9 in Nov, 2010 will showcase nation-wide OF Internet 2 and NLR to deploy before GEC9 Global trials Over 60 organizations experimenting 2010 likely to be a big year for OpenFlow

35. Slide Credits • Guido Appenzeller • Nick McKeown • Guru Parulkar • Brandon Heller • Lots of others • (this slide was also stolen)

36. Conclusion • OpenFlow is an API for controlling packet forwarding • OpenFlow+GENI allows more realistic evaluation of network experiments • Glossed over many technical details • What does the API look like? • Stay for the next session

37. An Experimenter’s Guide to OpenFlow: Office Hours GENI Engineering Workshop June 2010 Rob Sherwood (with help from many others)

38. Office Hours Overview • Controllers • Tools • Slicing OpenFlow • OpenFlow switches • Demo survey • Ask questions!

39. Controllers

40. Controller is King • Principle job of experimenter: customize a controller for your OpenFlow experiment • Many ways to do this: • Download, configure existing controller • e.g., if you just need shortest path • Read raw OpenFlow spec: write your own • handle ~20 OpenFlow messages • Recommended: extend existing controller • Write a module for NOX – www.noxrepo.org

41. Starting with NOX • Grab and build • `git clone git://noxrepo.org/nox` • `git checkout -b openflow-1.0 origin/openflow-1.0` • `sh boot.sh; ./configure; make` • Build nox first: non-trivial dependencies • API is documented inline • `cd doc/doxygen; make html` • Still very UTSL

42. Writing a NOX Module • Modules live in ./src/nox/{core,net,web}apps/* • Modules are event based • Register listeners using APIs • C++ and Python bindings • Dynamic dependencies • e.g., many modules (transitively) use discovery.py • Currently have to update build manually • Automated with ./src/scripts/nox-new-c-app.py • Most up to date docs are at noxrepo.org

43. Useful NOX Events • Datapath_{join,leave} • New switch and switch leaving • Packet_in/Flow_in • New Datagram, stream; respectively • Cue to insert a new rule/flow_mod • Flow_removed • Expired rule (includes stats) • Shutdown • Tear down module; clean up state

44. Tools • OpenFlow Wireshark plugin • MiniNet • oftrace • many more…

45. OpenFlow WireShark Plugin Ships with OpenFlow reference controller

46. MiniNet • Machine-local virtual network • great dev/testing tool • Uses linux virtual network features • Cheaper than VMs • Arbitrary topologies, nodes • Scriptable • Plans to move FV testing to MiniNet • http://www.openflow.org/foswiki/bin/view/OpenFlow/Mininet

47. OFtrace • API for analyzing OF Control traffic • Calculate: • OF Message distribution • Flow Setup time • % of dropped LLDP messages • … extensible • http://www.openflow.org/wk/index.php/Liboftrace

48. Slicing OpenFlow • Vlan vs. FlowVisor slicing • Use cases

49. Switch Based VirtualizationExists for NEC, HP switches but not flexible enough for GENI Flow Table Flow Table Controller Controller Research VLAN 2 Research VLAN 1 Production VLANs Normal L2/L3 Processing

50. OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlow FlowVisor & Policy Control FLOWVISOR BASED VIRTUALIZATION Heidi’s Controller Craig’s Controller Aaron’s Controller OpenFlow Protocol OpenFlow Protocol