Debugging Path MTU Discovery Failures - Techniques and Results

1. Debugging Path MTU Discovery Failures - Techniques and Results Internet2 Member Meeting September 21, 2005 Matthew Luckie, Kenjiro Cho, Bill Owens

2. Background - Why Do We Care? Large Packets == GOOD • TCP performance • interrupts Jumbo Packets (9000 bytes) == BETTER Fragmentation == REALLY BAD

3. Enter PMTUD If we want to avoid fragmentation, we must know the size of the largest packet the path will permit. This is especially critical when we go above 1500 bytes. Simple concept - send using your local MTU, set the Don’t Fragment bit, and watch for errors. Optimizations - have the routers tell you what their next-hop MTU is, to save searching.

4. PMTUD and Jumbo Most of the time this works. . . but only because most of the world is limited to 1500 bytes. ICMP packets are not first-class citizens: • firewalled • disabled • rate-limited The general consensus is that PMTUD fails in too many paths. So how do we deploy jumbo hosts?

5. PMTUD Failure Modes ICMP Black Hole no feedback from the path MTU Mismatches mixed configuration, or a low-MTU device ‘hidden’ between two L3 devices No Suggested MTU not fatal, but slows down the process Private IP Addresses error messages blocked by ingress filters

6. 1480 1500 Src GW FW Dst SYN TCP 3-Way Handshake SYN-ACK ACK HTTP GET HTTP Data Frag Reqd. HTTP Data Frag Reqd.

7. The Reverse-Path Problem A. Medina, M. Allman, S. Floyd. (2005) Measuring the Evolution of Transport Protocols in the Internet • 17% of 81776 targets failed at PMTUD • 35% of 500 ‘popular’ websites failed • Did not find –any– which tried smaller packets • Assumed middle-boxes filtering ICMP • 41% of targets did PMTUD • 30% did not attempt PMTUD

8. Testbed Topology NYSERNet - GigE 9000 byte connected server in NYC Abilene - GigE 9000 byte connected server in Chicago Targets - NLANR Active Measurement Project (AMP) hosts, 1500 byte connected

9. Testbed Software - scamper originally developed to probe IPv6 paths and find tunnels high-speed parallel traceroute with advanced MTU determination capabilities IPv4 and IPv6 capable http://www.wand.net.nz/scamper/

10. PMTUD Failure Diagnostics Initial Path Determination • begin with a traceroute using small packets • infer the forward path • determine which hops will send ICMP feedback, so we can distinguish between all packets being silently discarded, and just large ones

11. PMTUD Failure Diagnostics Next-Hop MTU Search • internal table of likely MTU values • start with local MTU • skip down to 1500 on failure • if not 1500, skip down to 1454 to check for the lower bound, set when feedback is obtained • if lower bound is a table value and upper bound is below the next largest, try +1 byte to confirm • if the next table value is below the upper bound, try it • if between table values, fall back to a binary search

12. PMTUD Failure Diagnostics Inferring MTU without feedback • perform next-hop MTU search to determine the PMTU • TTL search to find the hop where the MTU decreases silently • can locate the problem, but not the reason - ICMP disabled, filtered, blocked by RFC 1918?

13. 1500 1500 1500 1480 Src R1 * R3 Dst TTL 255, Size 1500 1. TTL 255, Size 1500 2. 3. TTL 255, Size 1454 ICMP Port Unreach 4. 5. TTL 255, Size 1480 ICMP Port Unreach 6. TTL 255, Size 1492 7. TTL 255, Size 1492 8. TTL 255, Size 1481 9. TTL 255, Size 1481 10.

14. 1500 1500 1480 1500 Src R1 * R3 Dst 11. Probe: TTL 1, Size 1500 Reply: ICMP Time Exceeded TTL 3, Size 1500 12. TTL 3, Size 1500 13.

15. PMTUD Failure Diagnostics Inferring MTU with invalid feedback • packet too big message with no next-hop MTU, or larger than the probe • use the bad message to speed up finding the actual MTU, working down through the table

16. Results Test runs on April 28, 2005 • 147 targets • 30% PMTUD failures, in four groups: no ICMP to packet too big message incorrect packet too big target machine with MTU mismatch

17. Results The number on the left is the number of AMP targets on a path with this failure mode. The number in brackets is the number of unique failure points.

18. Results No ICMP Messages • 7 failures (6 at 1500, 1 at 1536) • two due to ingress filters: • one originated ICMP with 127.0.0.1 • one originated with RFC 1918 address • another due to an “Internet-Free Zone” • another caused by a routing issue - routers in the forward path had no route back to our test source

19. Results No Packet Too Big Messages • 22 hops sent TTL Expired, but no PTB • 16 at 1500 • 4 at 4472, 2 at 4540, 1 at 4470, 1 at 2002 • 22 distinct problem locations • obtained technical diagnosis for seven, two were fixed before an answer could be obtained • two main causes: • no IP unreachables, but does not suppress TTL Expired • MTU mismatches

20. Results Incorrect Packet Too Big Messages • two hops at one location sent a message with incorrect next-hop MTU • we sent 9000 • PTB said to send 4586 • actually the path could only handle 4472

21. Results Target MTU Mismatches • seven AMP machines had 1500 byte interfaces, but were on a router interface that forwarded packets larger than 1500 • these machines shouldn’t receive packets larger than 1500 • . . . but tcpdump verified that two did: one at 1506, another at 2016

22. Results Specific Cases • Circuit Cross Connect - hidden 4470 limit • invalid next-hop MTU - 4458 limit, 4470 message • VoIP phone with integrated switch • different v4 and v6 MTUs

23. Acknowledgements • Matt Zekauskas collected nms1-chin (Chicago) dataset • WIDE funded development of scamper for a year • NLANR/NMA is funded by NSF ANI-0129677

24. References RFC 1191 - Path MTU Discovery RFC 1435 - IESG Advice from Experience with Path MTU Discovery RFC 1981 - Path MTU Discovery for IP version 6 RFC 2923 - TCP Problems with Path MTU Discovery Fragmentation Considered Harmful http://www.hpl.hp.com/techreports/Compaq-DEC/WRL-87-3.html Fragmentation Considered Very Harmful http://www.psc.edu/~jheffner/drafts/draft-mathis-frag-harmful-XX.html PMTUD Working Group http://www.psc.edu/~mathis/MTU/pmtud/