Bandwidth on the Internet

1. Bandwidth on the Internet Mat Ford ISOC Standards & Technology ISOC Advisory Council Meeting, Hiroshima, Japan Sunday November 8th 2009

2. Takeaways, upfront • Gross bandwidth growth more than catered for by new capacity • No Internet ‘brownouts’ anytime soon • Broadband creating new realities for access nets • More bandwidth provisioning isn’t the answer • Additional capacity is quickly consumed (by design) • Interests of content providers, ISPs and users not always well-aligned • We need more data • IETF has several streams of relevant work • ISOC has a panel on this topic during IETF76 Bandwidth on the Internet, ISOC AC Meeting

3. Bandwidth, defined • A term that describes the amount of information that can be passed through a communications channel in a given amount of time; that is, the capacity of the channel. The bandwidth is usually expressed in 'bits per second’… • Or bytes per second, leading to lots of confusion  • Used for expressing • Capacity of a channel or interface (e.g. 100Mbps ethernet i/f) • Rate of information transfer (e.g. downloading at 3.2KB/s) Bandwidth on the Internet, ISOC AC Meeting

4. Internet is sharing • Early objections to feasibility of packet-switching centred on the need for very large packet buffers to handle uncontrolled loads from end hosts • Leonard Kleinrock credited with demonstrating the theoretical feasibility of packet-switched communications • Internet protocols have their origin in the desire to share computers in diverse locations • Extension of the OS-paradigm of shared resources Bandwidth on the Internet, ISOC AC Meeting

5. Rules of the road • Resource sharing • Central to notions of Internet’s value and success • Enables interconnection of diverse applications • Over heterogeneous networking media with diverse speeds • Sharing creates potential for demand to outstrip supply • Necessitates a sharing mechanism • This is congestion control Bandwidth on the Internet, ISOC AC Meeting

6. How much bandwidth? • Cisco Forecasts 44 Exabytes per Month of IP Traffic in 2012 Bandwidth on the Internet, ISOC AC Meeting

7. Some rough consensus? Kenjiro Cho et al. see ~40% growth per annum since 2005, for IX peaks Bandwidth on the Internet, ISOC AC Meeting

8. What about capacity? • Telegeography: Capacity growing by 50% per annum Bandwidth on the Internet, ISOC AC Meeting

9. Bandwidth on the Internet, ISOC AC Meeting

10. Bandwidth on the Internet, ISOC AC Meeting

11. Bandwidth on the Internet, ISOC AC Meeting

12. Bandwidth on the Internet, ISOC AC Meeting

13. Bandwidth on the Internet, ISOC AC Meeting

14. Bandwidth on the Internet, ISOC AC Meeting

15. Bandwidth on the Internet, ISOC AC Meeting

16. Congestion Bandwidth on the Internet, ISOC AC Meeting

17. Definitions of congestion • Arrival rate > service rate (λ > μ) • Congestion = arrival rate/service rate (p=λ/μ) • In the presence of queues, congestion only present when packets are dropped? • This is TCP’s definition of congestion • For network operators, congestion is usually measured over longer timescales, e.g. 15mins • For economists, congestion occurs when increased use would impose a cost on existing users • In this case congestion can occur before queues start to build or packets are dropped Bandwidth on the Internet, ISOC AC Meeting

18. Evolution of congestion control • Congestion control originally handled by ICMP • Widespread incidents of congestion collapse in mid-1980s • Early TCP implementations had very bad retransmission behavior that pushed the entire network into a state where most packets were lost and the resultant throughput was negligible. • Addition of slow start, congestion avoidance, fast retransmit and fast recovery algorithms to TCP Image source: Internet Protocol Journal, Vol 9, No. 2 Bandwidth on the Internet, ISOC AC Meeting

19. Standards evolution • RFC 793 ‘Transmission Control Protocol’, STD 7 (Sept. 1981) • Doesn’t specify any congestion control mechanism • Congestion control handled by ICMP Source Quench • RFC 1122 ‘Requirements for Internet Hosts – Communication Layers’ (October 1989) • Mandates implementation of a congestion control algorithm • RFC 2581 ‘TCP Congestion Control’ (April 1999) • Defines the accepted mechanism • Required for the avoidance of congestion collapse • Flow-rate fairness Ref: RFC 4614 ‘A Roadmap for TCP Specification Documents’ Bandwidth on the Internet, ISOC AC Meeting

20. TCP • Implemented on end hosts • Regulates sending rate for a single flow • Doesn’t observe sending rate over time, or whether there are multiple flows from a single end host • Doesn’t regulate aggregate demand • Doesn’t know where congestion occurred, when it occurs • TCP will keep getting faster unless • All data is sent • Congestion experienced • End system maxed out Bandwidth on the Internet, ISOC AC Meeting

21. Animation credit: Guido Appenzeller TCP in action Bandwidth on the Internet, ISOC AC Meeting

22. TCP isn’t enough • In his original paper, Van Jacobson noted some limitations • While algorithms at the transport endpoints can insure the network capacity isn’t exceeded, they cannot insure fair sharing of that capacity. Only in gateways, at the convergence of flows, is there enough information to control sharing and fair allocation. Thus, we view the gateway ‘congestion detection’ algorithm as the next big step.’ • Whether or not multiple flows share common bottleneck links isn’t clear from the edge – only net ops have this view Bandwidth on the Internet, ISOC AC Meeting

23. Cable vs. DSL architectures Bandwidth on the Internet, ISOC AC Meeting

24. unresponsive flow3 Slide credit: Bob Briscoe TCP-fairness • voluntarily polite algorithms in endpoints • pushes until congested • equalises rates of data flows a game of chicken – taking all and holding your ground pays or start more ‘TCP-fair’ flows than anyone else (Web: x2, p2p: x5-100) or for much more data than others (video streaming or p2p file-sharing x200) • net effect of both (p2p: x1,000-20,000 higher traffic intensity) capacity bandwidth2 bandwidth1 time (VoIP, VoD) Bandwidth on the Internet, ISOC AC Meeting

25. In defence of flow-rate fairness • Floyd and Allman (2008) • ‘We do not, however, claim that flow-rate fairness is necessarily an optimal fairness goal or resource allocation mechanism for simple best-effort traffic. Simple best-effort traffic and flow-rate fairness are in general not about optimality, but instead are about a low-overhead service (best-effort traffic) along with a rough, simple fairness model (flow-rate fairness).’ Bandwidth on the Internet, ISOC AC Meeting

26. What’s different this time? • Congestion collapse isn’t new • Happened in the 80s • Predicted again in the 90s (Metcalfe’s gigalapse) • More predictions swirling around now • Growing diversity of uses/users • Increasing traffic volumes • Increasing economic value • Emergence of public policy debates Bandwidth on the Internet, ISOC AC Meeting

27. Impact of all-you-can-eat broadband source: Ellacoya 2007(now Arbor Networks) Source: Cho et al., Proceedings of ACM CoNEXT2008 Bandwidth on the Internet, ISOC AC Meeting

28. A changed landscape • Services that are now dominant consumers of bandwidth were simply not viable at pre-broadband access data rates • Natural suppression of user demand • Congestion is at the edge • Huge growth of mass-market broadband since 2000 • ‘a seismic shift in the nature of the congestion problem’ • Bandwidth hungry apps and multiple users per connection • Raised expectations • Access nets now the dominant constraint on achievable throughput • Changing diurnal patterns of consumption • Growing symmetry of traffic patterns Bandwidth on the Internet, ISOC AC Meeting

29. Slide credit: Bob Briscoe ISPs forced to subvert TCP bit-rate • equal bottleneck flow rates(TCP)? • access rate shared between active users, but weighted by fee (weighed fair queuing, WFQ)? • volume capstiered by fee? • heaviest applications of heaviest usersthrottled at peak times by deep packet inspection (DPI)? time bit-rate time bit-rate time bit-rate Bandwidth on the Internet, ISOC AC Meeting 29 time

30. Graphic credit: Bob Briscoe Congestion isn’t a bug, it’s a feature • Want transmissions to occur as fast as possible • Lack of congestion implies network could be more efficient • Techniques are available to enable the use of congestion signals to control flow rates without lost packets/retransmissions   bit-rate bit-rate time time Bandwidth on the Internet, ISOC AC Meeting

31. Slide credit: Bob Briscoe Better alternatives bit-rate bit-rate 1. TCP • light usage can go much faster • hardly affects completion time of heavy usage • doesn’t have to shift into night • BitTorrent & Microsoft have protocols to do this but... punished by #2, #3 & #4 NOTE: weighted sharing doesn't imply differentiated network service • just weighted aggressiveness of end-system's rate response to congestion weightedTCPsharing time time bit-rate 2. (weighted) fairqueuing time bit-rate 3. volume caps time bit-rate 4. deeppacketinspection(DPI) Bandwidth on the Internet, ISOC AC Meeting time

32. Slide credit: Bob Briscoe Limiting the horizon • Becoming impossible to deploy a new use of the Internet • Must negotiate arbitrary blocks and throttles en route • Two confusable motives • Fairer cost sharing • Competitive advantage to own services • How to deconfuse? how to encourage fairer cost sharing? • Make cost of usage transparent • Fixing Internet technology should avoid need for legislation Bandwidth on the Internet, ISOC AC Meeting 32

33. BBC iPlayer example • Original version used P2P to minimise BBC wholesale bandwidth costs • Latest update, P2P technology disabled • Disliked by end-users • Disliked by ISPs • BBC claim wholesale bandwidth costs have fallen • iPlayer Desktop warns users to beware ISP bandwidth caps and charges Bandwidth on the Internet, ISOC AC Meeting

34. Wrapping up • Recent studies conclude • P2P declining • Being replaced with video and other web2.0 content • Network capacity growing faster than traffic volumes • Interests of content providers, ISPs and users not always well-aligned • Fair management of congestion is a whole network issue Bandwidth on the Internet, ISOC AC Meeting

35. We need more data • http://mitas.csail.mit.edu/ • MINTS • CAIDA • M-Lab • Transparency wrt congestion management policies of net ops is a good start • Mismanagement of congestion is stifling application innovation • Want to avoid imposition of regulation that enshrines TCP flow-rate fairness in law – there is ample scope for innovation in congestion management mechanisms at both sub-second and sub-month timescales. Bandwidth on the Internet, ISOC AC Meeting

36. Network operators play a key role • Determine supply of network resources • Limit demand (access link capacity) • Is there a role for network operators at shorter time scales? Bandwidth on the Internet, ISOC AC Meeting

37. An inflection point? • Can the Internet community move to a new notion of what constitutes ‘fair’ resource allocation on the Internet? • TCP has ‘incumbent’s advantage’ • If it ain’t broke, don’t fix it! • Has the environment changed so significantly that it is broke? • Who should have responsibility for prioritising traffic during overload conditions? • How can the answer to that question be made to work in practice? Bandwidth on the Internet, ISOC AC Meeting

38. Relevant IETF work • Conex BoF • Exposing expected congestion along the forwarding path of the Internet • Ledbat WG • Congestion control algorithm for scavenger service • Alto WG • Protocols for better-than-random peer selection • MultipathTCP WG • Simultaneously use multiple paths in a single TCP session • Homegate BoF • Getting uniform set of requirements to aid deployment Bandwidth on the Internet, ISOC AC Meeting

39. ISOC Panel • Internet Bandwidth Growth: Dealing with Reality • 11.45am – 12.45pm, Tuesday 10th November • Light lunch provided • Orchid West Room, ANA Crowne Plaza Hotel • Panellists • Leslie Daigle (Moderator) • Kenjiro Cho, IIJ • Richard Woundy, Comcast • Danny McPherson, Arbor Networks • Lars Eggert, Nokia Research Center, IETF Transport AD • Audiocast & report will be available Bandwidth on the Internet, ISOC AC Meeting

40. References • The Evolution of Internet Congestion • Steven Bauer, David Clark, William Lehr (MIT) • Internet Cost Transparency: Mending Value Chain Incentives • Bob Briscoe (BT) • ATLAS Internet Observatory 2009 Annual Report • C. Labovitz et al. (Arbor Networks, Inc.) • Observing Slow Crustal Movement in Residential User Traffic • Cho et al. (IIJ) Bandwidth on the Internet, ISOC AC Meeting