A SOFT Way for OpenFlow Interoperability Testing

A SOFT Way for OpenFlow Interoperability Testing Maciej Kuźniar, Peter Perešini, Marco Canini†, Daniele Venzano, Dejan Kostić‡ EPFL †TU Berlin/T-Labs ‡IMDEA Networks

Software-Defined Networking (SDN) Third-partycontrolprogram Controller

Interoperability at Deployment Time OpenFlowprogram One OpenFlow API specification… Are OF switches interoperable? Release OpenFlowmessages Interop is critical for the success of SDN

Inconsistency - Example FlowMod message Modify VLAN ID to 4097 Forward packet Network in 3 different states Which state is assumed by the controller? Where are packets forwarded? Switch I Responds with an error message Switch II Trims VLAN value to 12 bits Installs the rule Switch III Silently drops packets

Interop: How Hard Can It Be? OpenFlowmessages OpenFlowinterface Inputs Likely source of OpenFlow interopissues OF Switch OpenFlow Agent OS Flow TableHardware AbstractionLayer Packets ASIC switch chip This work: Finding differences between OpenFlow Agent implementations Hardware correctness is formally verified “Forwarding” interface

OpenFlow Software Agent Switch software is not provably correct  • Specifications • Rapid flux (3 revisions in ~ 1 year) • Ambiguities • Specifications  Implementation • Implementation freedom • Vendors may not follow the specs Testing, testing and testing…

Interoperability Event • Gather various vendors • Hook up switches and controllers • Create and run test cases • See what breaks and fix it [ONF Interop WG, March ’12] • Very high manual effort • Test cases are not exhaustive • It is not a one time thing

Automating Interop Testing Insight:systematically crosscheck OF implementations

The 10,000 foot view Test inputs OF Agent 2 OF Agent 1 Problem I: What inputs should we use? Input-drivenexecution Observablebehaviors Inconsistency!

Symbolic Execution 65535 ... 25 ... CTRL 24 1 port Symbolic messageport = ∗ port < 25? port < 25 yes no Problem II: Path explosion port == CTRL? port ≥ 25 ∧port = CTRL no port ≥ 25 ∧port ≠ CTRL yes Send ERROR Forward Forward Send to CTRL Send to CTRL Send ERROR

Challenges Manage test inputs and coverage efficiently Capture behaviors Avoid simultaneous access to all code

SOFT(Systematic OpenFlow Testing) OF Agent 1 OF Agent 2 Phase 1 • Automated solution to interop testing • Systematic code coverage • No simultaneous access to all agents ? Phase 2

Structuring Inputs C3 C1 1.0 FLOW MOD LEN1 * * * * * * * 1.0 STAT REQ LEN2 * * * * C2 * Further reductions • Some messages are independent • Many inputs are entirely concrete • Small number of messages • Concrete values at cost of coverage

Benefit of Concretizing Fully Symbolic 28h

Capturing Behaviors Externally observable outputs • OpenFlow reply messages • Data plane packets • Normalize harmless nondeterminism (e.g., Buffer IDs) Internal state changes affect successive inputs • Use concrete probe packets

Example Agent 1 Agent 2

Finding Inconsistencies Agent 1 Agent 2

Finding Inconsistencies 65535 ... 25 ... CTRL 24 1 port Agent 1: Agent 2: Is there an input thatcauses two distinct behaviors? No false positives

Limitations • Short sequences of inputs • Unable to find problems with a complex state • Is an inconsistency harmless? • Can it affect the controller? • How to test all initial configurations? • Agent’s behavior depends on initial config

Prototype & Evaluation • SOFT prototype built on top of Cloud9/Klee • Compared • OpenFlow 1.0 Reference Switch (55k LoC) • Open VSwitch1.0.0 (80k LoC) • Input Sequences containing 1 - 4 messages

Does SOFT Work? Mostly related to message validation • Result of underspecification • No expected behavior in the specification • Inconsistent interpretation of the specification Found 7 classes of inconsistencies:

Summary • SOFT automates interoperability • testing of OpenFlow Agents • Also useful for: • regression testing • specification improvements

A SOFT Way for OpenFlow Interoperability Testing