1 / 38

Could the Internet hang up on Us? A longitudinal global study of Internet packet and data arrival predictability

Ulrich Speidel , ' Etuate Cocker, Firas Ghazzi , Nevil Brownlee Department of Computer Science, The University of Auckland. Could the Internet hang up on Us? A longitudinal global study of Internet packet and data arrival predictability. VoIP and other real-time protocols

valin
Download Presentation

Could the Internet hang up on Us? A longitudinal global study of Internet packet and data arrival predictability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ulrich Speidel, 'Etuate Cocker, FirasGhazzi, Nevil BrownleeDepartment of Computer Science, The University of Auckland Could the Internet hang up on Us? A longitudinal global study of Internet packet and data arrival predictability

  2. VoIP and other real-time protocols Causes and implications of variability in packet arrival Observing long-term trends in packet arrival variability with a beacon network Why the Pacific Islands are a great test bed – and why other parts of the world are important, too Preliminary observations Measures of arrival quality: jitter and entropy Agenda

  3. NikoRebenich, Stephen Neville, Aaron Gulliver, University of Victoria, B.C., Canada KashifNisar, Zeeshan Aziz, Suhaidi Hassan, Universiti Utara Malaysia Ming-Chui Dong, Vincent Wong, University of Macau RaimundEimann, Zurich, Switzerland (in a private capacity) Bernhard Platter, Bernhard Ager, ETHZ Zürich Hendrik Ferreira et al., University of Johannesburg, South Africa AJ Han Vinck, University of Duisburg-Essen, Germany Hirosuke Yamamoto, University of Tokyo Hidetoshi Yokoo, Gunma University Hiroyoshi Morita, University of Electro-Communication ITS, University of Auckland Muriel Médard, Jason Cloud, Massachusetts Institute of Technology Randy Bush (in a private capacity) Contributors TO DATE • Ministry of Lands, Survey, and Natural Resources, Tonga • Connect ISP, Fiji • Ministry of Infrastructure and Planning, Cook Islands • Oyster ISP (Telecom Cook Islands), Rarotonga, Cook Islands • Ministry of Fisheries, Government of Kiribati • IT Department, Government of Tuvalu • E.M. Jones Ltd., Tonga • Ministry of Revenue, Tonga • Solomon Telekom, Solomon Islands

  4. Ministry of Lands, Survey, and Natural Resources, Tonga Connect ISP, Fiji Ministry of Infrastructure and Planning, Cook Islands Oyster ISP (Telecom Cook Islands), Rarotonga, Cook Islands Ministry of Fisheries, Government of Kiribati IT Department, Government of Tuvalu E.M. Jones Ltd., Tonga Ministry of Revenue, Tonga Solomon Telekom, Solomon Islands Industry CONTRIBUTORS TO DATE

  5. Voice over IP (VoIP) telephony / videoconferencing / remote call centre industry Telemedicine (e.g. remote surgery / diagnostics / consultation) Remote control in industrial applications Internet of Things Financial applications (rapid trading) Why are real-time protocols important?

  6. VoICE OVER IP AND OTHER REAL-TIME PROTOCOLS • analogvoice signal is digitally encoded • encoded byte stream is chopped into a packet train • packets are transmitted via Internet to receiver • receiver buffers packets to establishconstant-rate dataflow to decoder • decoder recovers analog signal and replays it Transmitter Packets (~50 per second) blah Internet A/D Encoder Buffer blah Decoder D/A Receiver

  7. Delay between signal generation and replay is undesirable – must be kept as small as possible • Replay delay is composed of three components: • Physical latency of cables or satellite links along packet paths (fixed for a given path) • Time spent in router queues (variable) • Size of buffer at receiver (governed by variation in arrival times) VoICE OVER IP AND OTHER REAL-TIME PROTOCOLS

  8. Need to buffer more Real-time services such as VoIP, video conferencing, and remote manipulation may become impractical Downloads and streaming video/audio would still work Significant economic impact Effects are most likely to be felt between remote endpoints first IRregular packet arrival - ConSEQUENCES

  9. Length and number of router queues • Grow with increase in traffic and network mesh • Overflowing router queues cause packet drops • Load balancing at routers • Could this cause problems? Causes of variability in packet arrival

  10. Load balancing routers send packets to the same destination across different links If packets from the same stream are load balanced, it causes them to take different paths and experience different latencies In some cases, packets may overtake each other (out-of-order arrivals) LoaDBALANCINg ROUTERS

  11. PACIFIC RIM Submarine cables Which way do packetstravel between Asia and NZ?What are the possiblelatencies? http://www.submarinecablemap.com/

  12. Go Figure! Traceroute to our beacon at the University of Macau: >tracert 161.64.45.10 Tracing route to 161.64.45.10 over a maximum of 30 hops 1 1 ms 3 ms 3 ms tmk-net-207-gw.net.auckland.ac.nz [130.216.207.254] 2 2 ms 1 ms 1 ms cx-alpha-sxj-700.net.auckland.ac.nz [172.18.0.6] 3 3 ms <1 ms <1 ms dxj-260-1-to-cx-alpha.net.auckland.ac.nz [172.18.0.26] 4 2 ms 2 ms 2 ms br1-to-cx-alpha.net.auckland.ac.nz [130.216.246.41] 5 1 ms 1 ms 1 ms 203.167.201.41 6 4 ms 5 ms 5 ms unknown.telstraglobal.net [134.159.174.41] 7 29 ms 27 ms 31 ms i-0-0-1-1.sydo-core02.bx.telstraglobal.net [202.84.143.134] 8 29 ms 27 ms 27 ms i-0-3-2-0.sydo-core01.bi.telstraglobal.net [202.84.220.190] 9 212 ms 215 ms 215 ms i-0-5-2-0.hkth-core01.bx.telstraglobal.net [202.84.143.22] 10 141 ms 142 ms 141 ms i-3-0-0.hkth01.bi.telstraglobal.net [202.84.153.233] 11 155 ms 155 ms 155 ms tata-peer.hkth01.pr.telstraglobal.net [134.159.128.6] 12 144 ms 144 ms 144 ms 116.0.67.70 13 * 150 ms * z0l161.static.ctm.net [202.175.0.161] 14 160 ms 161 ms 160 ms cr1-191.macau.ctm.net [202.175.26.241] 15 161 ms 160 ms 160 ms ctm-int-umac.macau.ctm.net [202.175.0.58] 16 * * * Request timed out. … Trace complete. Traceroute to our beacon at Universiti Utara Malaysia: >tracert 103.5.183.14 Tracing route to 103.5.183.14 over a maximum of 30 hops 1 4 ms 1 ms 1 ms tmk-net-207-gw.net.auckland.ac.nz [130.216.207.254] 2 2 ms 1 ms 2 ms cx-alpha-sxj-700.net.auckland.ac.nz [172.18.0.6] 3 <1 ms <1 ms <1 ms dxj-260-1-to-cx-alpha.net.auckland.ac.nz [172.18.0.26] 4 1 ms 1 ms <1 ms br1-to-cx-alpha.net.auckland.ac.nz [130.216.246.41] 5 1 ms 2 ms 1 ms 203.167.201.41 6 154 ms 166 ms 155 ms te7-3.ccr01.sjc05.atlas.cogentco.com [38.122.92.105] 7 155 ms 155 ms 155 ms te0-2-0-4.ccr21.sjc01.atlas.cogentco.com [154.54.84.53] 8 133 ms 137 ms 135 ms be2162.mpd21.lax01.atlas.cogentco.com [154.54.27.173] 9 133 ms 133 ms 135 ms be2021.ccr21.lax04.atlas.cogentco.com [154.54.87.250] 10 133 ms 132 ms 144 ms 38.104.210.82 11 239 ms 239 ms 240 ms ae-2.cr-gw-2-sin-pip.sg.globaltransit.net [124.158.224.17] 12 238 ms 238 ms 238 ms 124.158.224.91 13 239 ms 240 ms 248 ms 124.158.226.150 14 248 ms 247 ms 248 ms ae-1.cr-02-glsfb.ni.time.net.my [223.28.2.121] 15 244 ms 245 ms 245 ms xe-0-0-0-0.er-01-glsfb.ni.time.net.my [223.28.2.2] 16 245 ms 244 ms 247 ms xe-0-1.es-01-glsfb.ni.time.net.my [223.28.16.130] 17 345 ms 345 ms 345 ms 203.121.112.18 18 345 ms 344 ms 345 ms 211.25.227.212 … Trace complete. via Sydney and Hong Kong via San Jose (California) and Los Angeles …and back across the Pacific!

  13. Technological progress or the forces of chaos it unleashes? Which effect will win? more traffic more powerfulrouters more congestion more malware new links extra bandwidth more users more inefficient routing more efficient codecs less predictable packetarrivals

  14. Basic idea: Pick a sample application, e.g., VoIP • Reality check: Applications don't stay stable, so need one whose behaviour we can control • Repeatedly use the application over time between remote end points where deterioration is likely to be seen early • See whether you observe any long-term changes • Do this for multiple pairs of endpoints so you can ascertain whether a change is specific to a particular endpoint or pair of endpoints Can we predict a trend?

  15. Basic idea: "Run a global network of beacon computers that act pair-wise as endpoints" To date (December 2013), 28 beacons have been installed in Canada, Cook Islands, Fiji, Germany, Japan, Kiribati, Macau, Malaysia, New Zealand, Solomon Islands, South Africa, Switzerland, Tonga, Tuvalu, and the United States. …and the network is growing! Additional beacons in Hawaii and Canada are under construction HOW? The beacon network

  16. The beacon network

  17. The beacon software is a simple C program that runs one experiment, writes it logs and then shuts down Scheduled via cron jobs & configured via command line parameters Writes plain text logs Requires sufficient privileges to run raw sockets (to determine TTL of received packets) Beacon software

  18. PC or Mac platform with Ubuntu/BSD/OSX Doesn't need to be overly fast – we use 600 MHz ALIX boards among others Must have real-time clock, but GPS / atomic clock synchronisation is not required Virtual machines are not a good idea – tend to spend too much time away from task at hand Beacon hardware

  19. Beacon exchanges 10,000 UDP packets of 110 bytes with a partner beacon Packets transmitted every 20 ms Packets are timestamped and serial-numbered At receiving end, packets are logged with arrival time stamp, serial number, arrival sequence number, and TTL observed This experiment typically runs 3 times a day between selected beacon pairs Beacon UDP experiments

  20. Two beacons exchange a predefined set of data via TCP Two modes available: VoIP (constant payload rate) and Download (maximum payload rate) Receiving end logs data arrival at application layer plus TTL and size of packets involved in communication (from raw socket) Beacon TCP experiments

  21. Want long-term global trend, not just local effects Want a "developed world" baseline but also see what it is like in remote places on the fringe – many of our beacons run in the Pacific for that reason (need I mention Africa?) Long paths generally are of interest – both in terms of latency and number of hops A lot of international traffic passes through "hub regions" (North America, Europe, SE Asia). What effect do these regions have on traffic that passes through them? Last but not least: We're looking for input (and your own experiments)! Why have beacons all over the world?

  22. Proper timing without proper timebase Precise timing on multitasking computers Getting two beacons to communicate (firewalls, ports, policies/politics) Scheduling Data volume backhaul (each beacon contributes ~3MB / experiment / day) Data processing & presentation (currently UDP only) Coordination with many partners, each with a unique network setup Making it longitudinal Beacon challenges

  23. Initial observations: Packet Loss

  24. Initial Observations: LOSS vs out-of-order

  25. Subjective approaches, e.g., Mean Opinion Score (MOS) – reliably replicable only with very large sample Objective approaches, e.g., jitter measurements. But: jitter can be random (=problem) or predictable (=less of a problem) How can we tell the difference? Packet train quality

  26. Consider the following simplified scenario: Assume uncongested routers but differences in latency, blue router load balances Have two paths/latencies -> high jitter, but not really queue-induced Jitter ignores systematic patterns in arrivals A good entropy estimator (i.e., not Shannon) can detect the difference NON-RANDOM jitter

  27. Entropy: "measure of disorder in a system". Usually given in the form of an entropy rate rather than absolute entropy Entropy rate: number of bits of actual information required to represent an object divided by number of bits actually used to represent it. Dimensionless ("bits/bit") Examples: Shannon entropy (well known). Better: Lempel-Ziv compression ratios, T-entropy, … Predictable arrivals or arrival patterns: low entropy "Really unpredictable arrivals" = high entropy Complement: entropy (rate)

  28. Map inter-arrival times of successive UDP packets to symbol bins, e.g.: • t < 17 ms: "A" • 17 ms < t < 19 ms: "B" • 19 ms < t < 21 ms: "C"… • Form string from these symbols: "CCCBDCACF…" • Determine entropy rate for string (e.g., as Lempel-Ziv compression ratio or T-entropy) • "Perfect" string will be "CCCCCCC…" – highly compressible, low entropy • Chaotic arrivals generate many new pattern combinations: harder to compress, higher entropy Entropy HOW-TO

  29. Entropy vs. jitter

  30. Entropy vs. jitter

  31. Where Is entropy Introduced?

  32. Picking a trend?

  33. Changing TTLs – PROOF OF PATH INSTABILITY

  34. Not ideal for real-time applications due to handshake delays Still very widely used for real-time apps as UDP is often blocked at firewalls (because it's historically been used infrequently) E.g., Skype fails over to TCP if UDP can't get through WHAT ABOUT TCP?

  35. TCP iN STREAMING Application Bytes needed for continuous rate immediate replay Cumulative amount of bytes received minimum buffer size Replay buffer underruns Bytes needed for continuous rate buffered replay (no underruns) Data [bytes] Actual replay with "boredom wheel" Time minimum buffer period Goal: Avoid buffer underruns with minimum buffer period / size

  36. TCP minimum buffer time trends?

  37. Real-time traffic and best-effort protocols are uneasy companions We propose to use a large beacon network to measure arrival shape of packet trains Our beacons already see interesting effects Effects are strongly path-specific and sometimes not easily explained Jitter and entropy are both useful for arrival shape evaluation A lot of work remains to be done! Ask us if you're interested in hosting a beacon https://iibex.auckland.ac.nz (https://130.216.197.6) Conclusions

  38. ? ? ? ? ? Questions ? ? ? ? https://iibex.auckland.ac.nz

More Related