Internet Routing (COS 598A) Today: Hot-Potato Routing

1. Internet Routing (COS 598A)Today: Hot-Potato Routing Jennifer Rexford http://www.cs.princeton.edu/~jrex/teaching/spring2005 Tuesdays/Thursdays 11:00am-12:20pm

2. Outline • Hot-potato routing • Selecting closest egress from a set • Hot-potato routing changes • Measuring hot-potato routing • BGP and IGP monitoring • Inferring causality • Characterizing hot potatoes • Frequency and number of destinations • Convergence delays and forwarding loops • Avoiding hot potatoes • Operational practices • New egress-selection techniques

3. Multiple links Middle of path Multiple Links Between Domains 4 3 5 2 6 7 1 Web server Client

4. dest multiple egress points 10 9 Hot-potato routing = route to closest egress point when there is more than one route to destination • All traffic from customer to peers • All traffic to customer prefixes • with multiple connections Hot-Potato Routing New York San Francisco ISP network Dallas

5. “Equally good” BGP Decision Process • Highest local preference • Lowest AS path length • Lowest origin type • Lowest MED (with same next hop AS) • Lowest IGP cost to next hop • Lowest router ID of BGP speaker

6. 2 dest 3 1 Motivations for Hot-Potato Routing • Simple computation for the routers • IGP path costs are already computed • Easy to make a direct comparison • Ensures consistent forwarding paths • Next router in path picks same egress point • Reduces resource consumption • Get traffic out as early as possible • (But, what does IGP distance really mean???)

7. dest - failure - planned maintenance - traffic engineering 10 9 11 11 Hot-Potato Routing Change New York San Francisco ISP network Routes to thousands of destinations switch egress points!!! Dallas • Consequences: • Transient forwarding instability • Traffic shift • Interdomain routing changes

8. Why Care about Hot Potatoes? • Understanding of Internet routing • Frequency of hot-potato routing changes • Influence on end-to-end performance • Operational practices • Knowing when hot-potato changes happen • Avoiding unnecessary hot-potato changes • Analyzing externally-caused BGP updates • Distributed root cause analysis • Each AS can tell what BGP updates it caused • Someone should know why each change happens

9. Measuring Hot Potatoes

10. Measuring Hot Potatoes is Hard • Cannot collect data from all routers • OSPF: flooding gives complete view of topology • BGP: multi-hop sessions to several vantage points • A single event may cause multiple messages • Group related routing messages in time • Router implementation affects message timing • Analyze timing in the measurement data • Controlled experiments with router in lab • Many BGP updates caused by external events • Classify BGP routing changes by possible causes

11. Measurement Infrastructure • Measure both protocols • BGP and OSPF monitors • Correlate the two streams • Match BGP updates with OSPF events • Analyze the interaction Z OSPF messages BGP updates ISP backbone X M Y

12. refresh link failure weight change chg cost chg cost del Stream of BGP updates Algorithm for Matching Stream of OSPF messages Transform stream of OSPF messages into routing changes Match BGP updates with OSPF events that happen close in time time Classify BGP updates by possible OSPF causes

13. CHG Y, 7 DEL X 1 ADD X, 5 X 5 Y 7 Y 7 X 5 Y 7 X 5 Y 4 Computing Cost Vectors • Transform OSPF messages into path cost changes from a router’s perspective OSPF routing changes: Z 1 1 2 1 X M 2 10 10 2 2 LSA weight change, 10 LSA weight change, 10 LSA delete 1 Y

14. Classifying BGP Updates • Cannot have been caused by cost change • Destination just became (un)available in BGP • New BGP route through same egress point • New route better/worse than old (e.g., shorter) • Can have been caused by cost change • New route is equally good as old route (perhaps X got closer, or Y got further away) M Z X dst Y

15. 10 sec 70 sec 180 sec The Role of Time • OSPF link-state advertisements • Multiple LSAs from a single physical event • Group into single cost vector change • BGP update messages • Multiple BGP updates during convergence • Group into single BGP routing change • Matching IGP to BGP • Avoid matching unrelated IGP and BGP changes • Match related changes that are close in time Characterize the measurement data to determine the right windows

16. Characterizing Hot Potatoes

17. Frequency of Hot-Potato Changes router A router B

18. Variation Across Routers dest dest NY NY SF SF 9 10 1 1000 B A Small changes will make router A switch exit points to dst More robust to intradomain routing changes Important factors: - Location: relative distance to egresses - Day: which events happen

19. Impact of an OSPF Change router A router B

20. Transfer delay BGP Reaction Time BGP scan First BGP update All BGP updates

21. 81 seconds delay Transferring Multiple Prefixes Cumulative Number of Hot-Potato Changes time BGP update – time LSA (seconds)

22. 1 – BGP decision process runs in R2 1 – BGP decision process runs in R2 2 – R2 starts using E1 to reach dst 2 – R2 starts using E1 to reach dst 3 – R1’s BGP decision can take up to 60 seconds to run 10 R2 R1 111 10 100 E2 E1 dst Packets to dst may be caught in a loop for 60 seconds! Data Plane Convergence 10 R2 R1 111 10 100 E2 E1 dst Disastrous for interactive applications (VoIP, gaming, web)

23. BGP Updates Over Prefixes prefixes with only one exit point Cumulative %BGP updates OSPF-triggered BGP updates affects ~50% of prefixes uniformly % prefixes

24. Avoiding Hot Potatoes

25. Z Z 10 1 X X 10 1000 Y Y Small changes will make Z switch exit points to dst More robust to intra-domain routing changes Reducing the Impact of Hot Potatoes • Vendors: better router implementation • Avoid timer-driven reaction to IGP changes • Move toward an event-drive BGP implementation • Operators: avoid equal-distant exits dst dst

26. dst Reducing the Impact (Continued) • Operators: new maintenance practices • Careful cost-in/cost-out of links • (But, is this problem over-constrained???) Z 5 100 5 X 10 10 4 10 Y

27. Is Hot-Potato Routing the Wrong Design? • Too restrictive • Egress-selection mechanism dictates a policy • Too disruptive • Small changes inside can lead to big disruptions • Too convoluted • Intradomain metrics shouldn’t be so tightly coupled with BGP egress selection

28. 9 B B A A 4 D 3 8 10 3 4 G 8 E 5 F C Strawman Solution: Fixed Ranking • Goal: no disruptions from internal changes • Each router has a fixed ranking of egresses • Select highest-ranked egress for each destination • Use tunnels from ingress to egress • Disadvantage • Sometimes changing egresses would be useful • Harm from disruptions depends on application dst

29. For each ingress, destination, egress: hot-potato routing m(i,dst,e) = d(i,e), d is intradomain distance fixed ranking Egress Selection Mechanisms automatic adaptation m(i,dst,e) = static rank(i,e) robustness to internal changes

30. TIE: Tunable Interdomain Egress Selection • Flexible policies • Tuning  and  allows covering a wide-range of egress selection policies • Simple computation • One multiplication and one addition • Information already available in routers • Easy to optimize • Expressive for a management system to optimize m(i,dst,e) = (i,dst,e) * d(i,e) + (i,dst,e)

31. Using TIE • Decouples egress selection from IGP paths • Egress selection is done by tuning  and  • Requirements • Small change in router decision logic • Use of tunnels • Configuring TIE • Network designers define high-level policy • Network management system translate policy into parameters

32. Example Policy: Minimizing Sensitivity • Problem definition • Minimize sensitivity to equipment failures • No delay more than twice design time delay • Simple change to routers • If distance is more than twice original distance • Change to closest egress • Else • Keep using old egress point • Cannot change routers for all possible goals

33. Output of simulation phase Minimizing Sensitivity with TIE dst B A 20 10 11 9 C At design time: m(C,dst,A) < m(C,dst,B) 9.(C,dst,A) + (C,dst,A) < 10.(C,dst,B) + (C,dst,B) 11.(C,dst,A) + (C,dst,A) < 10.(C,dst,B) + (C,dst,B) 20.(C,dst,A) + (C,dst,A) > 10.(C,dst,B) + (C,dst,B) Optimization phase: solve integer programming

34. Evaluation of TIE on Real Networks • Topology and egress sets • Abilene network (U.S. research network) • Set link weight with geographic distance • Configuration of TIE • Considering single-link failures • Threshold of delay ratio: 2 •  [1,4] and 93% of (i,dst,e)=1 •   {0,1,3251} and 90% (i,dst,e)=0 • Evaluation • Simulate single-node failures • Measure routing sensitivity and delay

35. Effectiveness of TIE • Delay • Within the 2x target whenever possible (i.e., whenever hot-potato could do it) • Lower delay than the fixed-ranking scheme • Sensitivity • Only a little more sensitive than a fixed ranking scheme • Much less sensitive than hot-potato routing

36. Conclusion • Hot-potato routing • Simple, intuitive, distributed mechanism • But, large reaction to small changes • Studying hot-potato routing • Measurement of hot-potato routing changes • Characterization of hot potatoes in the wild • Guidelines for vendors and operators • Improving the routing architecture • Identify egress selection as its own problem • Decouple from the intradomain link weights

37. Next Time: Root-Cause Analysis • Two papers • “Locating Internet Routing Instabilities” • “A Measurement Framework for Pin-Pointing Routing Changes” • NANOG video • “Root Cause Analysis of Internet Routing Dynamics” • Review just of the first paper • Summary, why accept, why reject, future work • Think about your course project • One-page written proposal by Thursday March 24 • Final written report due Tuesday May 10