Queueing theory for TCP

1. Queueing theory for TCP Damon Wischik, UCL Gaurav Raina, Cambridge

2. TCP DJW 2007 2/24 if (seqno > _last_acked) { if (!_in_fast_recovery) { _last_acked = seqno; _dupacks = 0; inflate_window(); send_packets(now); _last_sent_time = now; return; } if (seqno < _recover) { uint32_t new_data = seqno - _last_acked; _last_acked = seqno; if (new_data < _cwnd) _cwnd -= new_data; else _cwnd=0; _cwnd += _mss; retransmit_packet(now); send_packets(now); return; } uint32_t flightsize = _highest_sent - seqno; _cwnd = min(_ssthresh, flightsize + _mss); _last_acked = seqno; _dupacks = 0; _in_fast_recovery = false; send_packets(now); return; } if (_in_fast_recovery) { _cwnd += _mss; send_packets(now); return; } _dupacks++; if (_dupacks!=3) { send_packets(now); return; } _ssthresh = max(_cwnd/2, (uint32_t)(2 * _mss)); retransmit_packet(now); _cwnd = _ssthresh + 3 * _mss; _in_fast_recovery = true; _recover = _highest_sent; } transmission rate [0–100 kB/sec] time [0–8 sec]

3. Models for TCP DJW 2007 3/24 Single TCP sawtooth One can explicitly calculate the mean transmission rate, stationary distribution etc.

4. Models for TCP * * * * * * * DJW 2007 4/24 Single TCP sawtooth Each flow increases its transmission rate linearly... until the total transmission rate hits capacity, when some flows experience dropped packets... and those flows back off. Billiards modelBaccelli+Hong 2002

5. Models for TCP * * * * * * * DJW 2007 5/24 Single TCP sawtooth Approximate the aggregate TCP traffic by means of a differential equation... The differential equation has increase & decrease terms, which reflect the TCP sawtooth. Fluid modelKelly+Maulloo+Tan 1998Misra+Gong+Towsley 2000Baccelli+McDonald+Reynier 2002 Billiards modelBaccelli+Hong 2002

6. Models for TCP * * * * * * * DJW 2007 6/24 QUESTIONS When are these models good? What is the formula for packet drop probability, to plug into the models? How does it depend on buffer size etc.? Single TCP sawtooth Fluid modelKelly+Maulloo+Tan 1998Misra+Gong+Towsley 2000Baccelli+McDonald+Reynier 2002 Billiards modelBaccelli+Hong 2002

7. Simulation results DJW 2007 7/24 small buffer int. buffer • large buffer dropprobability[0–30%] queuesize[0–100%] trafficintensity[0–100%] time [0–5s]

8. Simulation results DJW 2007 8/24 When we vary the buffer size, the system behaviour changes dramatically. WHY? HOW? small buffer int. buffer • large buffer ANSWER It depends on the timescale and spacescale of fluctuations in the arrival process... dropprobability[0–30%] queuesize[0–100%] trafficintensity[0–100%] time [0–5s]

9. Formulate a maths question DJW 2007 9/24 • What is the limiting queue-length process, as N, in the three regimes • large buffersB(N)=BN • intermediate buffersB(N)=B√N • small buffersB(N)=B • Why are these the interesting limiting regimes? What are the alternatives? [Bain, 2003] Round trip time RTT N TCP flows Service rate NC Buffer size B(N)

10. Intermediate buffers B(N)=B√N DJW 2007 10/24 • Consider a standalone queue • fed by N Poisson flows, each of rate x pkts/sec where x varies slowly, served at constant rate NC, with buffer size B√N • i.e. an MNx//DNC //1/B√N queue • xt=0.95 then 1.05 pkts/sec, C=1 pkt/sec, B=3 pkt • Observe • queue size fluctuates very rapidly; busy periods have duration O(1/N)so packet drop probability pt is a function of instantaneous arrival rate xt • queue size flips quickly from near-empty to near-full, in time O(1/√N) • packet drop probability is pt=(xt-C)+/xt queuesize trafficintensity time N=50 N=100 N=500 N=1000

11. Intermediate buffers B(N)=B√N DJW 2007 11/24 • Consider a standalone queue • fed by N Poisson flows, each of rate x pkts/sec where x varies slowly, served at constant rate NC, with buffer size B√N • i.e. an MNx//DNC //1/B√N queue • xt=0.95 then 1.05 pkts/sec, C=1 pkt/sec, B=3 pkt • Observe • queue size fluctuates very rapidly; busy periods have duration O(1/N)so packet drop probability pt is a function of instantaneous arrival rate xt • queue size flips quickly from near-empty to near-full, in time O(1/√N) • packet drop probability is pt=(xt-C)+/xt This formula was proposed for TCP by Srikant 2004 queuesize trafficintensity time N=50 N=100 N=500 N=1000

12. In the open-loop queue we assumed Poisson inputs with slowly varying arrival rate queue size flips quickly from near-empty to near-full queue size fluctuates very rapidly, with busy periods of duration O(1/N), so drop probability pt is a function of instantaneous arrival rate xt In the closed-loop TCP system the round trip time means that we can treat inputs over time [t,t+RTT] as an exogenous arrival process to the queue the average arrival rate should vary slowly, according to a differential equation the aggregate of many point processes converges to a Poisson process, and this carries through to queue size [Cao+Ramanan 2002] Closing the loop DJW 2007 12/24 • There is thus a separation of timescales • queue size fluctuates over short timescales, and we obtain pt from the open-loop M/D/1/B(N) model • TCP adapts its rate over long timescales, according to the fluid model differential equation

13. Intermediate buffers B(N)=B√N DJW 2007 13/24 • 2Gb/s link, C=50 pkt/sec/flow • 5000 TCP flows • RTT 150–200ms • buffer 619pkt • simulator htsim by Mark Handley, UCL dropprobability[0–30%] queuesize[0–619pkt] trafficintensity[0–100%] time [0–5s] queuesize[594–619pkt] time [50ms]

14. Intermediate buffers B(N)=B√N DJW 2007 14/24 • 2Gb/s link, C=50 pkt/sec/flow • 5000 TCP flows • RTT 150–200ms • buffer 619pkt • simulator htsim by Mark Handley, UCL dropprobability[0–30%] When the traffic intensity changes between <capacity and >capacity, the buffer can fill or drain in time O(1/√N) queuesize[0–619pkt] trafficintensity[0–100%] time [0–5s] Queue size fluctuations are O(1), invisible at the O(√N) scale queuesize[594–619pkt] time [50ms]

15. Instability and synchronization DJW 2007 15/24 • When the queue is not full, there is no packet loss, so transmission rates increase additively • When the queue becomes full, there are short periods of concentrated packet loss • This makes the flows become synchronized queuesize[0–619pkt] TCPwindowsizes[0-25pkt] time [0–5s] This is the billiards model for TCP

16. Large buffers B(N)=BN DJW 2007 16/24 dropprobability[0–30%] The total arrival rate is Nxt and the service rate is NC, so it takes time O(1) for the buffer to fill or drain queuesize[0–BN pkt] trafficintensity[0–100%] time [0–5s] queue size[(B-25)–B pkt] and we can calculate the packet drop probability when the queue is full time [50ms] Queue size fluctuations are O(1), invisible on the O(N) scale...

17. Small buffers B(N)=B DJW 2007 17/24 The queue size changes too quickly to be controlled, but the queue size distribution is controlled by the average arrival rate xt dropprobability[0–30%] queuesize[0–BN pkt] trafficintensity[0–100%] time [0–5s] queue size[(B-25)–B pkt] Queue size fluctuations are of size O(1), over timescale O(1/N), enough to fill or drain the buffer... time [50ms]

18. Conclusion * * * * * * * DJW 2007 18/24 Fine for a single flow Single TCP sawtooth Good for a small number of flows... This is also an extreme case of the fluid model, when flows are synchronized due to intermediate-sized buffers Arises as a law-of-large-numbers limit, when there are many TCP flows Fluid model Billiards model

19. Conclusion * * * * * * * DJW 2007 19/24 Single TCP sawtooth Proved by McDonald+Reynier Fluid model Billiards model

20. Instability & buffer size DJW 2007 20/24 • On the basis of these models, and of a control-theoretic analysis of the limit cycles that ensue, we claimed • a buffer of 30–50 packets should be sufficient • intermediate buffers will make the system become synchronized, and utilization will suffer • large buffers get high utilization, but cause synchronization and delay ½=1 B=100pkt B=50pkt B=40pkt B=30pkt B=20pkt B=10pkt traffic intensity distribution[85–110%] queue size distribution[0–100%] Plot by Darrell Newman, Cambridge

21. Instability & buffer size DJW 2007 21/24 • On the basis of these models, and of a control-theoretic analysis of the limit cycles that ensue, we claimed • a buffer of 30–50 packets should be sufficient • intermediate buffers will make the system become synchronized, and utilization will suffer • large buffers get high utilization, but cause synchronization and delay ½=1 B=100pkt B=50pkt • But when other people ran simulations they found... • Small buffers lead to terribly low utilization • Intermediate buffers work much better B=40pkt B=30pkt B=20pkt B=10pkt traffic intensity distribution[85–110%] queue size distribution[0–100%] Plot by Darrell Newman, Cambridge

22. Burstiness DJW 2007 22/24 • The problem is that the Poisson limit breaks down • A key step in the C+R proof is showing that each flow contributes at most one packet in a busy period • But TCP, on a fast access link, will produce bursty traffic:sources can send an entire window-full of packets back-to-back • For very fast access speeds, i.e. very bursty TCP traffic, a batch-M/D/1/B model is appropriate

23. Burstiness & access speeds DJW 2007 23/24 • Simulations of a system with a single bottleneck link with capacity NC, where each flow is throttled by a slow access link 40C access speed = 5C TCP traffic is bursty over short timescales, so the queue's stochastic fluctuations are large... TCP has continual feedback, and can nearly maintain a steady state TCP traffic is Poisson-like over short timescales, so the queue's stochastic fluctuations are small... TCP has bang-bang feedback and it is unable to maintain a steady state

24. Burstiness & access speeds DJW 2007 24/24 • Simulations of a system with a single bottleneck link with capacity NC, where each flow is throttled by a slow access link 40C access speed = 5C You have to be careful about short-timescale traffic statistics when you model TCP Queueing theory still has a role!