Stable Internet Routing Without Global Coordination

1. Stable Internet Routing Without Global Coordination Jennifer Rexford Princeton University http://www.cs.princeton.edu/~jrex Joint work with Lixin Gao, Michael Schapira, and Yi Wang

2. What is an Internet? • A “network of networks” • Networks run by different institutions • Autonomous System (AS) • Collection of routers run by a single institution • ASes have different goals • Different views of which paths are good • Interdomain routing is what reconciles those views • To compute end-to-end paths through the Internet Wonderful problem setting for game theory and mechanism design

3. An Open Question Evolvable Protocols (under-specified, programmable) ? Global Properties (stability, scalability, reliability, security, managability, …) Autonomy (autonomous parties, with different economic objectives) Can we have all three? Under what conditions?

4. Autonomous Systems (ASes) Path: 6, 5, 4, 3, 2, 1 4 3 5 2 6 7 1 Web server Client

5. Interdomain Routing: Border Gateway Protocol • ASes exchange info about who they can reach • Destination: block of IP addresses (an “IP prefix”) • AS path: sequence of ASes along the path • Policies configured by the AS’s network operator • Path selection: which of the paths to use? • Path export: which neighbors to tell? “I can reach d via AS 1” “I can reach d” 2 3 1 data traffic data traffic d

6. Interdomain Routing Convergence Challenges • Must scale • Address blocks: 300,000 and growing • Autonomous Systems: around 35,000 • Must support flexible policy • Path selection: which path your AS wants to use • Path export: who can send packets through your AS • Must converge, and quickly • Routing convergence can take several minutes • … and the system doesn’t necessarily converge at all! Goal: Guaranteed convergence of the global routing system with purely local control.

7. 2 3 d 2 d 2 1 2 d 1 d d 1 3 1 d 3 d 3 Stable Paths Problem (SPP) Model • Model of routing policy • Each AS has a ranking of the permissible paths • Model of path selection • Pick the highest-ranked path consistent with neighbors • Flexibility is not free • Global system converges slowly, or not at all • Depending on the way the ASes rank their paths

8. Better choice! Only choice! Top choice! Better choice! Only choice! Only choice! Conflicting Policies Cause Convergence Problems 1 2 0 1 0 1 0 3 1 0 3 0 2 3 0 2 0 3 2 Pick the highest-ranked path consistent with your neighbors’ choices.

9. Global Control is Not Workable • Create a global Internet routing registry • Keeping the registry up-to-date would be difficult • Require each AS to publish its routing policies • ASes may be unwilling to reveal BGP policies • Check for conflicting policies, and resolve conflicts • Checking for convergence problems is NP-complete • Link/router failure may result in an unstable system Need a solution that does not require global coordination.

10. Think Globally, Act Locally • Key features of a good solution • Flexibility: allow diverse local policies for each AS • Privacy: do not force ASes to divulge their policies • Backwards-compatibility: no changes to BGP • Guarantees: convergence even when system changes • Restrictions based on AS relationships • Path selection rules: which route you prefer • Export policies: who you tell about your route • AS graph structure: who is connected to who

11. advertisements traffic Customer-Provider Relationship • Customer pays provider for access to the Internet • Provider exports its customer’s routes to everybody • Customer exports provider’s routes only to downstream customers Traffic to the customer Traffic from the customer d provider provider customer d customer

12. advertisements traffic Peer-Peer Relationship • Peers exchange traffic between their customers • AS exports only customer routes to a peer • AS exports a peer’s routes only to its customers Traffic to/from the peer and its customers peer peer d

13. Hierarchical AS Relationships • Provider-customer graph is a directed, acyclic graph • If u is a customer of v and v is a customer of w • … then w is not a customer of u w v u

14. Valid and Invalid Paths Valid paths: “6 4 3 d” and “8 5 d” Invalid paths: “6 5 d” and “1 4 3 d” Valid paths: “1 2 d” and “7 d” Invalid path: “5 8 d” 1 4 3 2 d 5 6 Provider-Customer 7 Peer-Peer 8

15. Act Locally, Prove Globally • Route export • Do not export routes learned from a peer or provider • … to another peer or provider • Global topology • Provider-customer relationship graph is acyclic • E.g., my customer’s customer is not my provider • Route selection • Prefer routes through customers • … over routes through peers and providers • Guaranteed to converge to unique, stable solution

16. Our Local Path Selection Rules • Classify routes based on next-hop AS • Customer routes, peer routes, and provider routes • Rank routes based on classification • Prefer customer routes over peer and provider routes • Allow any ranking of routes within a class • E.g., can rank one customer route higher than another • Gives network operators the flexibility they need • Consistent with traffic engineering practices • Customers pay for service, and providers are paid • Peer relationship contingent on balanced traffic load

17. Solving the Convergence Problem • Result • Safety: guaranteed convergence to unique stable solution • Inherent safety: holds under failures and policy changes • Definitions • System state: current best route at each AS • Activating AS: re-do decision based on neighbor choices • Sketch of (constructive) proof • Find an activation sequence that leads to a stable state • Any “fair” sequence (eventually) includes this sequence

18. Rough Sketch of the Proof • Two phases • Walking up the customer-provider hierarchy • Walking down the provider-customer hierarchy 1 4 3 2 d 5 6 Provider-Customer 7 Peer-Peer 8

19. Economic Incentives Affect Protocol Behavior • ASes already follow our rules, so system is stable • High-level argument • Export and topology assumptions are reasonable • Path selection rule matches with financial incentives • Empirical results • BGP routes for popular destinations are stable for ~10 days • Most instability from failure/recovery of a few destinations • ASes should follow our rules to make system stable • Need to encourage operators to obey these guidelines • … and provide ways to verify the network configuration • Need to consider more complex relationships and graphs

20. Playing One Condition Off Against Another • All three conditions are important • Path ranking, export policy, and graph structure • Allowing more flexibility in ranking routes • Allow same preference for peer and customer routes • Never choose a peer route over a shorter customer route • … at the expense of stricter AS graph assumptions • Hierarchical provider-customer relationship (as before) • No private peering with (direct or indirect) providers Peer-peer

21. Extension to Backup Relationships • Backups: more liberal export policies, and different ranking • The motivation is increased reliability • …but ironically it may cause routing instability! • Generalize rule: prefer routes with fewest backup links • Need to maintain a count of the # of backup links in the path Backup Provider Peer-Peer Backup [RFC 1998] provider primary provider backup path failure backup path backup provider failure peer

22. Results Hold Under More Complex Scenarios • Complex AS relationships • AS pair with different relationship for different prefixes • AS pair with both a backup and a peer relationships • AS providing transit service between two peer ASes • Stability under changing AS relationships • Customer-provider to/from peer-peer • Customer-provider to/from provider-customer

23. Extensions of the Work • Influence of AS relationships on BGP convergence • Algebraic framework and design principles for policy languages • Fundamental limits on relaxing the assumptions • Application of the idea to internal BGP inside an AS • Sufficient conditions for iBGP convergence inside an AS • “What-if” tool for traffic engineering inside an AS • AS-level analysis of the Internet topology • Inference of AS relationships and policies from routing data • Characterization of AS-level topology and growth • Practical applications of knowing AS relationships • Analyzing your competitors’ business relationships • Identifying BGP routes that violate export conditions

24. A Case For Customized Route Selection • ISPs usually have multiple paths to the destination • Different paths have different properties • Different neighbors may prefer different routes Shortest latency Most secure Bank VoIP provider School Lowest cost

25. Neighbor-Specific Route Selection • A node has a ranking function per neighbor is node i’s ranking function for neighbor node j.

26. Stability Conditions for NS-BGP • Surprisingly, NS-BGP improves stability! • Neighbor-specific selection is more flexible • Yet, requires less restrictive stability conditions • “Prefer customer” assumption is not needed • Choose any “permissible” route per neighbor • That is, need just two assumptions • No cycle of provider-customer relationships • Do not export routes learned from one peer/provider to other peers/providers

27. Why Do Weaker Conditions Work? • An AS always tells its neighbor a route • If it has any route that is permissible for that neighbor 1 2 0 1 0 1 0 3 1 0 3 0 2 3 0 2 0 3 2

28. Deploying NS-BGP • An AS can deploy NS-BGP alone • Without upgrading their routers • Without coordinating with all their neighbors • Three aspects to the solution • Disseminating extra BGP routes • Customized route selection • Directing traffic from ingress to egress • Can be done exploiting existing mechanisms • Designed for Virtual Private Networks (VPNs)

29. Disseminating Extra BGP Routes • Advertising more than one BGP route • Route distinguisher feature for VPNs • Multiple internal BGP sessions • ADD-PATHs extensions to internal BGP

30. Customized Route Selection R3’s forwarding table (FIB) entries D: (red path): R6 D: (blue path): R7 • Multiple virtual routing and forwarding tables • Cisco: Virtual Routing and Forwarding (VRF) • Juniper: Virtual Router

31. Directing Traffic from Ingress to Egress ? • Tunnels from ingress to egress • IP-in-IP tunneling • MPLS

32. Customized Route Selection • Customized route selection as a service • Select a different best route for different neighbors • Different menu options • Cheapest route (e.g., “prefer customer”) • Best performing routes, or most secure routes • Routes that avoid undesirable ASes (e.g., censorship) • Nice practical features of NS-BGP • An individual AS can deploy NS-BGP alone • … without compromising global stability!

33. Conclusions • Avoiding convergence problems • Hierarchical of provider-customer relationships • Export policies based on commercial relationships • (Path ranking based on AS relationships) • Salient features • No global coordination (locally implementable) • No changes to BGP protocol or decision process • Guaranteed convergence, even under failures • Guidelines consistent with financial incentives

34. References Related to This Talk • “The stable paths problem and interdomain routing” • Tim Griffin, Bruce Shepherd, and Gordon Wilfong • http://portal.acm.org/citation.cfm?id=508332 • “Stable Internet routing without global coordination” • Lixin Gao and Jennifer Rexford • http://www.cs.princeton.edu/~jrex/papers/sigmetrics00.long.pdf • “Inherently Safe Backup Routing with BGP” • Lixin Gao, Tim Griffin, and Jennifer Rexford • http://www.cs.princeton.edu/~jrex/papers/infocom01.pdf • “Neighbor-Specific BGP: More flexible routing policies while improving global stability” • Yi Wang, Michael Schapira, and Jennifer Rexford • http://www.cs.princeton.edu/~jrex/papers/nsbgp_sigmetrics09.pdf

35. Other Related Research Papers • Inherently Safe Backup Routing with BGP • http://www.cs.princeton.edu/~jrex/papers/infocom01.pdf • Design Principles of Policy Languages for Path Vector Protocols • http://conferences.sigcomm.org/sigcomm/2003/papers/p61-griffin.pdf • Implications of Autonomy for the Expressiveness of Policy Routing • http://conferences.sigcomm.org/sigcomm/2005/paper-FeaBal.pdf • Meta-routing • http://conferences.sigcomm.org/sigcomm/2005/paper-GriSob.pdf • An Algebraic Theory of Interdomain Routing • http://portal.acm.org/citation.cfm?id=1103561 • Searching for Stability In Interdomain Routing • http://www.cs.yale.edu/homes/schapira/PID808559.pdf

36. Related Papers With Game Theory • Interdomain Routing and Games • http://www.cs.huji.ac.il/~mikesch/routing_games-full.pdf • Rationality and Traffic Attraction: Incentives for Honest Path Announcements in BGP • http://ccr.sigcomm.org/online/?q=node/395 • Incentive-Compatible Interdomain Routing • http://cs-www.cs.yale.edu/homes/jf/FRS.pdf • Mechanism Design for Policy Routing • http://cs-www.cs.yale.edu/homes/jf/FSS.pdf • The Complexity of Game Dynamics: BGP Oscillations, Sink Equlibria, and Beyond • http://www.cs.berkeley.edu/~alexf/papers/fp08.pdf • Specification Faithfulness in Networks with Rational Nodes • http://www.eecs.harvard.edu/econcs/pubs/podc04.pdf • Distributed Algorithmic Mechanism Design • http://cs-www.cs.yale.edu/homes/jf/AGTchapter14.pdf • Partially Optimal Routing • http://www.stanford.edu/~rjohari/pubs/por.pdf

37. Background on Interdomain Economics • http://drpeering.net/a/Home.html • http://www.fcc.gov/Bureaus/OPP/working_papers/oppwp32.pdf • http://www.potaroo.net/papers/1999-6-peer/peering.pdf • http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac201/about_cisco_ipj_archive_article09186a00800c83a5.html • http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac200/about_cisco_ipj_archive_article09186a00800c8900.html • http://www.vjolt.net/vol3/issue/vol3_art8.html