How to Multi-Home

1. How to Multi-Home Avi Freedman VP Engineering AboveNet Communications

2. What is Multi-Homing? • Multi-homing is the process of selecting, provisioning, and installing a redundant connection to the Internet. • Could be the same provider, or a different provider.

3. Why Multi-Home? • Slow is 1,000,000% better than dead. • You may be out of bandwidth. • And • Telco circuits die. • Routers die. • Providers’ networks fail. • Different networks have better performance to different sites.

4. A Multi-Homed Architecture • Ideally, take advantage of the opportunity to multi-home to remove all single points of failure in your network. • Use - • Multiple providers, unless your current provider will let you have cheap backup • Multiple routers • Multiple telco vendors

5. Multi-Homed Architecture • Two routers, each with a different WAN connection from a different telco vendor. • Use HSRP or VRRP internally to make both routers look like one “virtual” router. • Eventually, multiple providers. • Upcoming Boardwatch article with configs.

6. How the Internet Works • Well, it breaks more than it works but when it does work - • The Internet is a network of networks. • Each network (called Autonomous System) on the Internet announces “routes”, which are lists of the IP addresses of the boxes on their network. • You need to be able to send packets *to*, and get packets *from*, everywhere.

7. Inbound Traffic - Routes • Routes are announced via BGP4 (the Border Gateway Protocol) • Routers are announced to BGP peers. • Each “BGP peer” can be a “network peer” or a “transit peer”. • Network peers exchange just lists of customer routes. • Each route is tagged by the ASNs it passes through.

8. Inbound Traffic - Routes • So when AboveNet and UUNET peer, only AboveNet and UUNET routes are exchanged. No Sprint, PSI, etc... • Transit peers - • Announce to their customers all of the routes on the ‘net (AboveNet, UUNET, Sprint, PSI, and the 60,000+ routes on the ‘net). • Announce to their peers all routes heard via transit.

9. Inbound Traffic - Routes • So if you advertise 207.106.96.0/19 to AboveNet, - • If you’re a network peer, they only re-announce 207.106.96.0/19 to customers (and use it internally); • If you’re a transit peer/customer, they announce 207.106.96.0/19 to all of their network peers. • That’s how you get global *inbound* reachability.

10. Address Space Issues • Noone wants to hear a route for you unless - • You are multi-homed (even then, some people don’t want to hear routers), or • You have your own direct IP space allocation from ARIN, RIPE, or APNIC. • So, when you’re single-homed without your own space, your IPs are reachable because they’re part of your provider’s “aggregate” block.

11. Address Space Issues • For example, your provider has 207.8.128.0/17. • You have 27.8.197.0/24 from them. • You’re single-homed. • The only route on the ‘net for you is the 207.8.128.0/17 route, “originated” by your provider’s ASN (and you don’t have to do anything special).

12. Address Space Issues • If you have your own CIDR block and are single-homed, your provider will originate it. • So, if you have 219.190.64.0/19, it’ll be visible as an announcement by your provider, originated into the BGP mesh with your provider’s ASN as the “origin”.

13. Address Space Issues • If you have your own IP space and want to multi-home, addressing issues are simple. • Your other provider will start also originating your IP blocks. • Or you’ll start speaking BGP, originate your IP blocks, and your providers will re-advertise them to the world.

14. Address Space Issues • If you don’t have your own IP space, it’s a bit more complicated. • So, normally your ISP will only be advertising 207.8.128.0/17 if you have 207.8.200.0/23. • If you’re multi-homed, your other provider will have to advertise 207.8.200.0/23. • But *so will your first provider*. • Why?

15. Address Space Issues • Routes are chosen first by specificity. • That is, to how many IP addresses they refer. • The route “covering” the fewest IP is the most specific, and wins. • (Otherwise default would always win and nothing would work.)

16. Address Space Issues • So, if ISP 1 advertises only 207.8.128.0/17 and ISP 2 advertises only 207.8.200.0/23, all inbound traffic from the ‘net will come in on ISP2. • So, ISP 1 needs to “blow a hole in their filters” to “leak” the more specific 207.8.200.0/23 route.

17. Address Space: Filtering • Some ISPs do or did filter on routes smaller than (more specific than) /19s in > 205.0.0.0 space. • But it doesn’t matter as long as your two upstreams have good connectivity. • Why?

18. Address Space: Filtering • If Sprint doesn’t see 207.8.200.0/23 from ISP1 or ISP2, they’ll still see your provider’s 207.8.128.0/17 route. • So if your connectivity to ISP1 (the owner of 207.8.128.0/17) goes down, all will be well as long as ISP1 still sees 207.8.200.0/23 from ISP2. • Sprint -> ISP1 -> ISP2 • This is why people don’t let you take IPs...

19. Load-Balancing Outbound • You can use static default routes to control outbound packets. • ip route 0.0.0.0 0.0.0.0 serial0/0 • ip route 0.0.0.0 0.0.0.0 serial1/0 • If they’re equal-cost (no metric at the end), it’ll load-balance based on *destination*, by default.

20. Load-Balancing Outbound • Why load-balance based on destination? • For internal networking, sometimes per-packet-load balancing makes sense. • But if you’re trying to talk to England and one provider has a 60ms path and the other has a 150ms path, packets will arrive out of order and TCP and UDP apps get unhappy and slow.

21. How it works, Single-Homed • Outbound (easy): • Use a default route to your provider. • Inbound: • Your provider originates a large (aggregate) BGP route, and gives you some space from inside it; and/or • Your provider originates BGP routes for your ARIN/RIPE/APNIC CIDR blocks as well.

22. How it Works, Multi-Homed, Static • Outbound (easy): • Load-balance default routes to deal with outbound packets. • Inbound: • Your providers both originate BGP routes for just the address space you’re using, even if it’s out of one provider’s space; and/or • Your providers both originate BGP routes for your ARIN/RIPE/APNIC CIDR blocks as well.

23. How it Works, Multi-Homed, Static • Special note: • When providers configure BGP for single-homed customers, they will generally “nail up” your routes (even your directly-issued) CIDR blocks, so that if your connection goes down and up and down and ..., they don’t have to flap that route out to the whole Internet. This is a good thing.

24. How it Works, Multi-Homed, Static • Special note (ctd): • But you NEED to make sure, when you’re multi-homed, that the providers are NOT nailing your routes up. • Why? • Because if they do, when one T1 goes down, that provider will still advertise you to the world, thus “blackholing” you.

25. How it Works, Multi-Homed, BGP • Topic of next talk. • You either load-balance outbound with statics, or take full routes from your providers (if you can). • You originate advertisements under your ASN for your directly-issued CIDR blocks, AND for the parts of your providers’ space that you’re using (with their permission).

26. The Transition: Static Routing • To transition: • Turn up the other T1/T3/Ethernet. • Put IPs on the interface. • Run tests end-end. • Start load-balancing default to the new T1. • Then, in the middle of the night, have the new provider start advertising your IP space. Make sure you have reachability to every other ISP you can think of afterwards.

27. The Transition: Static Routing • To transition (ctd): • After testing it live, turn off your other transit pipes and make sure that, after a few minutes, you still have connectivity.

28. The Transition: BGP Routing • To transition: • Turn up the other T1/T3/Ethernet. • Put IPs on the interface. • Run tests end-end. • Start load-balancing default to the new T1. • Then, undo that and bring up a BGP session that permits no routes either way. • Then start taking routes, and watch outbound traffic.

29. The Transition: BGP Routing • To transition (ctd): • Then, start announcing your routes. • Then, in the middle of the night, have your ISP take out the static route and BGP announcement they were making. • Make sure your route is propagating. • Test reachability. • Turn off your other pipes. • Test reachability.

30. BGP or no? • Advantages of doing static - • Cheaper/smaller routers (less true nowadays) • Simpler to configure • Advantages of doing BGP - • More control of your destiny (have providers stop announcing you) • Faster/more intelligent selection of where to send outbound packets. • Better debugging of net problems (you can see the Internet topology now)

31. Same Provider or Multiple? • If your provider is reliable and fast, and affordably, and offers good tech-support, you may want to multi-home initially to them via Frame, SMDS, or some backup path (slow is 1,000,000% better than dead). • Eventually you’ll want t multi-home to different providers, to avoid failure modes due to one provider’s architecture decisions.

32. Questions? • avi@freedman.net • inet-access mailing list

33. Nailing routes