A WAN-in-LAB for Protocol Development

1. A WAN-in-LAB for Protocol Development Netlab, Caltech Lachlan Andrew, George Lee, Steven Low(PI), John Doyle, Harvey Newman

2. Outline • What and why is WAN-in-Lab? • What can I do with WiL? • Why would I use WiL? • How do I use WiL? • Future plans

3. What is WAN-in-Lab? • “Wide Area” Network in a laboratory • Real fibre delays • Carrier-class routers, switches, …

4. ? DummyNet EmuLab ModelNet WAIL UltraLight PlanetLab Abilene NLR LHCNet CENIC etc NS2 SSFNet QualNet JavaSim Mathis formula Optimization Control theory Nonlinear model Stocahstic model Why -- Spectrum of tools cost abstraction live netwk WANinLab emulation simulation maths All scales are important– WAN-in-Lab fills a gap

5. What can I do with WAN-in-Lab?

6. Other groups’ interests • Protocol development • FAST, delay-based • MaxNet, explicit signalling • ADPM, single-bit explicit signalling • Impact of small buffers (U. Pittsburgh) • Test automatic configuration of routers (MonALISA, Ultralight) • Test distributed file-system (MojaveFS)

7. TCP Benchmarking • Our current main direction • Evaluating others’ protocols, not ours • Web interface • Submit kernel patch • Standard tests automatically performed • Results mailed back • Explicit or implicit signalling protocols

9. Physical topology

10. Capabilities: Delay • 24 spools of 100km fibre, many loopbacks • Set delay by MEMS switching loops in/out • 130ms physical delay • more with IP loopback • 2 Dummynets: long delay for cross-traffic 125 ms, 1.8ms steps

11. External connections • Linked to Ultralight, 10Gbps Physics WAN • Smooth migration testing -> deployment • Delay • longer • jitter • Cross traffic • Monitordata routedthrough WiL

12. Why use WAN-in-Lab?

13. Why use WiL? • Complement other levels of abstraction, not replace them • Different ways to use it: reasons for each • Standard platform for TCP benchmarking • Easier to compare with others’ results • No need to write your own test suite

14. 83 packets Artifacts of software delays • Packets sent on 1ms “ticks” • 1Gbps = 83,333 pk/s 1ms

15. How can I use WAN-in-Lab?

16. Management structure

18. Time sharing • Coarse switching between projects • Servers rebooted, routers reconfigured • Switchover takes ~5 minutes • Book in advance • For longer bookings, book further in advance • Also “ad hoc” bookings for individual hosts • Can log in while others have booked

19. Future plans

20. Future plans • Benchmarking infrastructure • Standardise tests • Use it ourselves • Develop “indices” of TCP performance • Better control over capacities and buffers • Better cross-traffic generation • Currently Harpoon • Investigate differences from DummyNet • Integrate DAG cards

21. Conclusion • WAN-in-Lab fills the gap between emulation and live network experiments • Seeks to be as realistic as possible • Long links, simple topology • Focus will be on TCP benchmarking • We welcome people to use it <http://wil.cs.caltech.edu>

22. Spare Slides

23. Case Study: MaxNet

24. Aim: Wind Tunnel of Networking • WAN in Lab • Capacity: 2.5 – 10 Gbps • Delay: 0 – 120 ms round trip • Breakable • Won’t take down live network • Flexible, active debugging • Passive monitoring, AQM • Configurable & evolvable • Topology, rate, delays, route • Modular design stays up to date • Integral part of R&A networks • Transition from theory, implementation, demonstration, deployment • Transition from lab to marketplace • Global resource • Part of global infrastructure UltraLight led by Harvey Newman

25. Equipment • 4 Cisco 7609 routers with OC48 line cards • 6 Cisco ONS 15454 switches • A few dozen high speed servers • 1G switch to routers/servers • Calient switch for OC48 • 2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules • 63ms aggregate RTT delay, in two hops • 120ms using IP loopbacks

26. Accounts • Mail wil at cs.caltech.edu • Sudo access to “network” commands • Ifconfig/…/ • Custom commands to set topologies • Login to routers if required • Separate accounts for “benchmark only”

27. Configuration -- Delays • Want maximum delay from limited fibre • Signals traverse fibre 16 times • 4 WDM wavelengths • 4 OC48 (2.5G) MUXed onto OC192 (10G) • Lots of transponders • WDM amplifier joins 100km spools  200km

28. OC48 slot Configuration – delays -------WDM Wavelength-------- 16x200km Amp Bidirectional 100km Bidirectional 100km

29. Configuration – delays • Delay varied by adjusting the number of OC48 hops traversed • Calient optical switch selects required hops • Hop lengths 200km up to 1600km • Maximise granularity given limited switch ports Switch

30. Projects • TCP benchmarking • FAST • Delay-based congestion control • MaxNet • Explicit signalling congestion control • MojaveFS • New distributed file system • University of Pittsburgh • TCP with small buffers • University of Melbourne • Single-bit congestion marking

31. WAN-in-Lab testbed • Dummynet and simulation introduce artifacts • Also need to test on real equipment • WAN with real delays, located in a single room • Connected to an external WAN (Ultralight) • Open for the community to use for benchmarking OC-48 OC-48

32. WAN-in-Lab capabilities

33. OC48 slot Configuration -- delays -------WDM Wavelength-------- Amp Bidirectional 100km Bidirectional 100km

34. Using WAN-in-Lab • Contact me – lachlan at caltech . Edu • Coarse timesharing • Some users set up experiments while others run experiments • Software setup still being developed • Your chance to influence our directions to tailor it to your needs

35. Sample MaxNet results • Achieves realistic delay at 1Gbit/s