New Interdomain Routing Architectures

1. New Interdomain Routing Architectures Jennifer Rexford

2. Well, BGP Has Some Problems • Instability • Not guaranteed to converge to a unique, stable solution (e.g., oscillation and bi-stable solutions) • Slow convergence • Path exploration can take a very long time • Non-linearity • Small changes can have large effects (e.g., intradomain changes leading to BGP changes) • Feature interaction • Unexpected side effects (e.g., the interaction of route-flap damping and path exploration)

3. Well, BGP Has Some More Problems • Scalability challenges • Large memory, processing, and message-passing overhead, dependent on behavior in other ASes • Security vulnerabilities • BGP update messages are relatively easy to forge with bogus prefix, AS path, or other attributes • Difficult to configure • Operators must express their (many) goals in an indirect and contorted manner • Difficult to troubleshoot • Hard to infer the root cause of a routing change, or even the AS(es) responsible

4. But Wait, There’s More! • No performance guarantees • BGP announcements do not include any information about the performance of the path • Limited flexibility for path selection • Only single-path routing on destination prefix • Limited control over path selection • Chosen path is a byproduct of the composition of local routing policies in many different ASes • Forwarding path may not match AS path • Routers may deflect packets to other paths (e.g., route aggregation, misconfiguration, and malice)

5. And, Perhaps the Biggest Problem of All… • Hard to change • BGP is the glue that holds the disparate parts of the Internet together • So, hard to change BGP in a fundamental ways • … or is it?

6. Why is BGP Such a Mess? • Absence of underlying models • Protocol designed without a design for the decision process or policy language • BGP models (e.g., SPP) came much later • Incremental evolution of the protocol • Many additions to BGP over the years • New route attributes and decision-process steps • Rapid growth of the Internet • Many ASes, and much more complex topology • Commercialization of the Internet • More complex routing policies and competition • Heightened importance of security concerns

7. What’s a Networking Researcher to Do? • Characterize the existing routing system • Measurement, modeling, simulation • Improved operational practices • Best common practices for configuration • Timer settings, route filtering, features to use, … • Methods and tools for complicated tasks • Traffic engineering, config checking, root-cause analysis • Incremental changes to BGP • Better implementations of the protocol • New route attributes for greater flexibility • Graceful handling of failures, config changes, …

8. What’s a Networking Researcher to Do? • Build on top of the existing routing system • Overlay networks • Propose new architectures anyway • Identify and evaluate new and better solutions • … even if we can’t readily deploy them today • Explore incremental-deployment approaches • Meta solutions that enable new architectures • With the goal of leading to a better architecture • Create models of the fundamental limits • Maybe the problem BGP is solving is really hard…

9. Key Ingredients of Architectural Proposals • More control at the edge • Source routing and multi-path routing • Negotiation between ASes • Explicit coordination for path selection • Design for the common case • Handle common routing policies well (e.g., HLP) • Servers computing the routes • Greater visibility and control than any one router • Multiple simultaneous architectures • Virtualization to support customized solutions

10. Source Routing • The ultimate in flexibility • Sender determines path for each packet • At the router level, or at the AS level • At the cost of… • Overhead of propagating topology information • Need for fast path changes after a failure • Lost control for intermediate routers/ASes B C ABEF A F D ADECF E

11. Source Routing Deployment Challenges • ASes have little incentive to cooperate • No business relationship with ASes far away • Don’t want to relinquish control over routing • So, source routing is typically disabled • Policy-compliant source routing • Limiting what sources routes can be used • Progress on limited forms of source routing • Overlay networks • Source routing on an overlay topology • Multi-homing route control • “One hop source routing”

12. Multipath Routing • Expose more paths to ASes • Multiple paths through same next-hop AS • Giving ASes greater control over path selection • Extreme case: export all paths • High protocol overhead • Lost control for intermediate ASes • Compromise: export selective paths • Under control of intermediate ASes • Based on their routing policies, pricing, etc.

13. BCF Exposing Extra Paths • Example: AS A doesn’t like AS E • Default routes both go through E • Good to learn alternate route that avoids E BEF*BCF CF*CEFCBEF B C ABEF*ADEF A F F* D E DEF*DABEF EF *ECF

14. Encapsulation to Use Non-Default Paths • Direct packet along alternate route • Destination-based forwarding not enough • Encapsulate the packet to egress point e e d d B C A F D E

15. Inter-AS Negotiation • Directives • Tell other AS what to do • E.g., which link to use • Suggestions • Tell other AS your wishes • Which link you prefer used • Cooperation • Negotiate where to send • Across flows and time • Inbound and outbound • Trade small losses for larger gain (“win win”) Customer B Provider B multiple peering points Early-exit routing Provider A Customer A

16. Inter-AS Negotiation • But, how to do it? • What info to exchange? • How to prioritize the many choices? • How prevent cheating? • Open research territory • Some initial work on exchanging preferences • E.g., ASes exchange preferences, and iterates Customer B Provider B multiple peering points Early-exit routing Provider A Customer A

17. Revisiting the Division of Labor • Routers • Forward packets: yes • Compute paths: maybe not • End users or ASes • Dictate path properties: yes • Control path selection: maybe not • Intermediate ASes (e.g., ISPs) • Forward packets: yes • Business relationships: yes • Compute end-to-end paths: maybe not

18. Removing Routing From Routers • Should routers forward packets: yes • Must be done at high speed • … in line-card hardware on fast routers • So, needs to be done on the routers • Should routers compute routes: no • Hard to do in a distribution fashion • Difficult to make load-sensitive routing stable • Lacking complete information for good decisions • Not flexible enough for end users • Difficult to extend over time

19. Routing As a Service (RAS) • Goal: third parties pick end-to-end paths for clients to satisfy diverse user objectives • Forwarding infrastructure • Basic routing (e.g., default routing) • Primitives for inserting forwarding-table entries • Routing Service Provider (RSP) • Contracts ISPs for service (e.g., virtual links) • Selects end-to-end routes on behalf of clients • Competes with other selectors for customers • End host • Queries RSP to pick path & install forwarding state

20. Routing As a Service (RAS) RSP ISP ISP user ISP

21. eBGP eBGP Inter-AS Protocol RCP RCP RCP iBGP RCP RCP iBGP iBGP AS 1 AS 2 AS 3 Physical peering Routing Control Platform (RCP) • Goal: Move beyond today’s artifacts, while remaining compatible with the legacy routers • Incentive compatibility: phased evolution • Intelligent route reflector in a single AS • Learning BGP routes directly from neighbor ASes • Interdomain routing between RCPs • Backwards compatibility: BGP to the routers • Using BGP to “push” answers to the routers • No need to change the legacy routers at all • Keep message format and router implementation

22. How Does This Differ From Overlays • Overlays: circumventing the underlay • Host nodes throughout the network • Logical links between the host nodes • Active probes to observe the performance • Direct packets through good intermediate nodes • Routing services: controlling the underlay • Servers collect data directly from the routers • Servers compute forwarding tables for the routers • Data packets do not go through the servers • Like an overlay for managing the underlay Maybe some combination of the two makes sense?

23. Concurrent Architectures Better than One • Multiple customized architectures in parallel • Multiple virtual routers on a single platform • Resource isolation in CPU, forwarding table, bandwidth • Programmability for different protocols and mechanisms (routing, forwarding, addressing, …)

24. Cabo: Applications Within an Single ISP • Customized virtual networks • Security for online banking • Fast-convergence for VoIP and gaming • Specialized handling of suspicious traffic • Testing and deploying new protocols • Evaluate on a separate virtual network • Rather than in a dedicated test lab • Large scale and early-adopter traffic • Leasing virtual components to others • ISPs have unused node and link capacity • Can allow others to construct services on top

25. Cabo: Enabling Economic Refactoring • Infrastructure providers:Maintain routers, links, data centers, other physical infrastructure • Service providers:Offer end-to-end services (e.g., layer 3 VPNs, SLAs, etc.) to users Infrastructure Providers Service Providers Today: ISPs try to play both roles, and cannot offer end-to-end services

26. Key Ingredients of Incremental Deployability • Backwards compatibility • Technically possible to deploy the solution • E.g., change anything but the message format • Incentive compatibility • Offer concrete benefits to early adopters • Generate incentives for others to follow • Do not rely on complete participation • QoS, multicast, secure routing, IPv6, … • Need creativity on incremental deployment

27. Lessons on Incremental Deployment • Adding on top: tunneling in the data plane • Circumvent destination-based forwarded • Traverse routers • Adding on top: servers in the control plan • Tricking the routers into doing the server’s bidding • Implementing new functionality on the servers • Sneaking in on the side: virtualization • Running additional virtual networks in parallel • Offering new functionality for special applications • … while continuing to support “legacy” Internet

28. Discussion • Tussles between stake-holders • Users and ISPs • Division of function • Data, control, and management planes • End host, edge routers, and core routers • How much better could BGP be? • While still being an interdomain protocol with control distributed across Autonomous Systems? • With an even cleaner slate than that? • Where should the economics be???