Internet Routing (COS 598A) Today: Interdomain Routing Convergence

1. Internet Routing (COS 598A)Today: Interdomain Routing Convergence Jennifer Rexford http://www.cs.princeton.edu/~jrex/teaching/spring2005 Tuesdays/Thursdays 11:00am-12:20pm

2. Outline • BGP convergence • Causes of routing changes • Detecting session failures • BGP path exploration • Route-flap damping • Damping persistent flapping • Interaction with path exploration • Stability of popular destinations • Are things really all that bad? • Reducing convergence delay • Avoiding complete path exploration • Why this is harder than it looks

3. Causes of BGP Routing Changes • Topology changes • Equipment going up or down • Deployment of new routers or sessions • BGP session failures • Due to equipment failures, maintenance, etc. • Or, due to congestion on the physical path • Changes in routing policy • Reconfiguration of preferences • Reconfiguration of route filters • Persistent protocol oscillation • More on this next week!

4. BGP Session Operation Establish session on TCP port 179 AS1 BGP session Exchange all active routes AS2 While connection is ALIVE exchange route UPDATE messages Exchange incremental updates

5. BGP Session Failure • BGP runs over TCP • BGP only sends updates when changes occur • TCP doesn’t detect lost connectivity on its own • Detecting a failure • Keep-alive: 60 seconds • Hold timer: 180 seconds • Reacting to a failure • Discard all routes learned from the neighbor • Send new updates for any routes that change AS1 AS2

6. Routing Change: Before and After 0 0 (2,0) (2,0) (1,0) (1,2,0) 1 1 2 2 (3,2,0) (3,1,0) 3 3

7. Routing Change: Path Exploration • AS 1 • Delete the route (1,0) • Switch to next route (1,2,0) • Send route (1,2,0) to AS 3 • AS 3 • Sees (1,2,0) replace (1,0) • Compares to route (2,0) • Switches to using AS 2 0 (2,0) (1,2,0) 1 2 (3,2,0) 3

8. 0 Routing Change: Path Exploration (2,0) (2,1,0) (2,3,0) (2,1,3,0) • Initial situation • Destination 0 is alive • All ASes use direct path • When destination dies • All ASes lose direct path • All switch to longer paths • Eventually withdrawn • E.g., AS 2 • (2,0)  (2,1,0) • (2,1,0)  (2,3,0) • (2,3,0)  (2,1,3,0) • (2,1,3,0)  null (1,0) (1,2,0) (1,3,0) 1 2 3 (3,0) (3,1,0) (3,2,0)

9. Convergence Overhead and Delay • Path exploration is expensive • Large number of possible paths • Might have to explore (nearly) all of them • Minimum Route Advertisement Interval • Minimum time between advertisement of routes for a given destination to a given neighbor • Rate limit on BGP update messages • … and allows combining multiple messages in one • Typical value of 30 seconds • Convergence delay • (30 seconds) * (# of paths)

10. Four Kinds of BGP Routing Changes • Destination becomes reachable • Switch from no path to a new path • Better path becomes available • Switch from old path to new, better path • Best path becomes unavailable • Switch from old path to new, worse path • Destination becomes unreachable • Switch from old path to no path at all lower delay higher delay

11. Questions About Convergence Delay • Reduce the MRAI timer? • High message overhead on the router? • Delays from overloading the CPU? • What is the right value? • Dependence on topology? • Worst-case: n! • Fully-connected graph (i.e., a clique) • No filtering of advertisements • Shortest-path routing • Destination dies completely • Typical case?????

12. Route Flap Damping

13. Persistent Routing Changes • Causes • Link with intermittent connectivity • Congestion causing repeated session resets • Persistent oscillation due to policy conflicts • Effects • Lots of BGP update messages • Disruptions to data traffic • High overhead on routers • Solution • Suppress paths that go up/down repeatedly • … to avoid updates and prefer stable paths

14. Route Flap Damping • BGP-speaking router • One or more BGP neighbors • Keep an “RIB-in” per neighbor • Select single best route per destination prefix • Route-flap damping • Penalty counter per (peer, prefix) pair • Increment penalty when peer changes route • Decrease penalty over time when route is stable • Design and deployed in the mid 1990s • Widely viewed as helping improve stability

15. Example Why Damping is Good • Consider AS 3 • Path #1: (3,1,0) • Path #2: (3,2,0) • If link (1,0) fails • AS 3 switches routes • If link (1,0) restores • AS 3 switches routes • If this happens a lot • Better for AS 3 to stick with (3,2,0) 0 (2,0) (1,0) 1 2 3

16. Damping Penalty Function suppression threshold penalty reuse threshold time

17. Configurable Damping Parameters • Penalty for a routing change • May vary with the type of update message • Advertisement vs. withdraw? Attributes change? • Decaying in absence of a change • Exponent in the exponential decay • Suppression threshold • Trigger for damping the route • Determines how many updates are tolerated • Reuse threshold • Trigger for considering the route again • Determines how long the route is not usable

18. Best Common Practices for Damping • Different parameters for different prefixes • More aggressive with small address blocks • Disable damping on certain prefixes (e.g., corresponding to the DNS root servers) • Avoid suppressing stable routes • Tolerate at least four routing changes • Suppress unstable routes for quite a while • Values ranging from 10 minutes to 1 hour • Values for 30 minutes are not uncommon

19. Interaction with Path Exploration • BGP routing convergence • Explore one or more alternate paths • Number of alternate paths may be quite high • Time between steps is small (e.g., 30 seconds) • Triggering route-flap damping • Increasing penalty with each step • Only small amount of decay between steps • Convergence may trigger route flap damping • Convergence may involve more than 4 changes • Routing change may trigger lost connectivity!!! • Confirmed by recent active measurement studies

20. Effects of Damping are Confusing • AS 0 is a stable network • Link (1,3) fails a lot • AS 3 switches routes back and forth a lot • Sends new BGP updates to its customers • Suppose AS 3 does not apply route-flap damping • AS 3’s customers • Eventually dampen route • Causes lost reachability to destination in AS 0 • How can AS 0 diagnose this problem, and fix it? 0 1 2 3

21. Open Questions • Want to suppress unstable routes • Otherwise, lots of update messages • … and lots of transient disruptions • Yet, want to tolerate path exploration • Otherwise, you suppress stable routes • … and black-hole otherwise reachable destinations • How to reconcile? • Better flap-damping parameters? • More information in update messages? • Something more gentle than suppression?

22. BGP Stability of Popular Destinationshttp://www.cs.princeton.edu/~jrex/papers/imw02.pdf

23. BGP Routing and Traffic Popularity • A possible saving grace… • Most BGP updates due to few prefixes • … and, most traffic due to few prefixes • ... but, hopefully not the same prefixes • Popularity vs. BGP stability • Do popular prefixes have stable routes? • Yes, for ~ 10 days at a stretch! • Does most traffic travel on stable routes? • A resounding yes! • Direct correlation of popularity and stability? • Well, no, not exactly…

24. BGP Updates • BGP updates for March 2002 • AT&T route reflector • RouteViews and RIPE-NCC • Data preprocessing • Filter duplicate BGP updates • Filter resets of monitor sessions • Removes 7-30% of updates • Grouping updates into “events” • Updates for the same prefix • Close together in time (45 sec) • Reduces sensitivity to timing Confirmed: few prefixes responsible for most events

25. Internet in out AT&T Amazon www.amazon.com 207.171.182.16 207.171.176.0/20 Two Views of Prefix Popularity • AT&T traffic data • Netflow data on peering links • Aggregated to the prefix level • Outbound from AT&T customers • Inbound to AT&T customers • NetRatings Web sites • NetRatings top-25 list • Convert to site names • DNS to get IP addresses • Clustered into 33 prefixes

26. 50% of events 1.4% of traffic (4.5% of prefixes) Traffic Volume vs. BGP Events (CDF) 50% of traffic 0.1% of events (0.3% of prefixes)

27. 1% had > 5 events per day No “popular” prefix had > 3 events per day Update Events/Day (CCDF, log-log plot) Most “popular” prefixes had < 0.2 events/day and just 1 update/event

28. An Interpretation of the Results • Popular  stable • Well-managed • Few failures and fast recovery • Single-update events to alternate routes • Unstable  unpopular • Persistent flaps: hard to reach • Frequent flaps: poorly-managed sites • Unpopular does not imply unstable • Most prefixes are quite stable • Well-managed, simple configurations • Managed by upstream provider

29. Avoiding Path Exploration

30. 0 Reducing Path Exploration By Tagging • When AS 1 sees (1,0) fail • Switches to (1,2,0) • Why not say “because the link (1,0) has failed”? • Allow ASes to discard all paths that use edge (1,0) • Should reduce exploration • E.g., AS 3 should not consider (3,2,1,0) • E.g., AS 2 should not consider (2,3,1,0) • Seems appealing, but… (1,0) (1,2,0) (1,3,0) (2,0) (2,1,0) (2,3,0) 1 2 3 (3,0) (3,1,0) (3,2,0)

31. Problem #1: Timing of Information • How long should the ASes believe the info? • What if the link (1,0) comes back up? • What if the info about the failure is still propagating? • Do the ASes need to remember the old paths? • E.g., should AS 2 remember (2,3,1,0) in case it learns later that (1,0) has come back up? • BGP is an incremental protocol, so forgetting information may be risky unless you will get it back again • But, these issues are probably surmountable • … with some attention to the details

32. Problem #2: AS With Multiple Routers/Links • BGP introduces abstraction • Treats each AS as a single node • Doesn’t distinguish between links • Example: one link fails • Should AS 1 tell others? • Need to identify which link? • Does it introduce more updates? • Internal BGP details matter • Some AS 1 routers don’t know about both paths through AS 0… 1 0 d

33. Internal BGP Convergence Briefly, the border router has no route at all!

34. Questions • Can we reduce path exploration • Hints in the BGP update messages • To avoid exploring a set of related paths • Handling the challenges • Timing details • Multiple routers and links per AS • … without excessive overhead • Can we change the problem • Server per AS that stores all candidate routes • Exchanging information about the root cause

35. Next Time: Protocol Divergence • Two papers • “The Stable Paths Problem and Interdomain Routing” • “Stable Interdomain Routing Without Global Coordination” • Review only of the first paper • Summary • Why accept • Why reject • Future work • Optional NANOG video on “BGP Wedgies”