Community Attribute Use in Internet2 CPS

1. Community Attribute Use in Internet2 CPS Caren Litvanyi lead network engineer peering team Internet2 NOC GigaPoP Geeks BOF January 2008 Honolulu, Hawai’i

2. Outline • Quick review of what CPS is. • Quick notes on how to connect. • So if I’m connected to CPS, how can I use communities to control how my routes are advertised? • Discussion and feedback from all you GPG’s.

3. CPS Background CPS is Internet2’s “Commercial Peering Service”. “Through CP Service members can leverage their existing Internet2 Network investments to help serve their commercial Internet needs, thereby saving money on commodity Internet charges.” CPS is included in the base connection fee, so it is available for Internet2 Network connectors at no additional cost. http://www.internet2.edu/network/cp/

4. CPS Background • Implemented as an “overlay” on the existing Internet2 Layer3 Network. • Uses MPLS Layer3 VPN, VRF on same T640 routers. • Currently 4 commercial public peering sites: • PAIX New York 10GE • Equinix Chicago 10GE • PAIX Palo Alto 10GE • SIX (Seattle) 1GE • Also PNI (private peerings). • Today has over 76,000 unique commercial prefixes from approximately 50 peers, and advertises about 850 connector prefixes.

5. Connecting to CPS - brief • Call up the Internet2 NOC, open a ticket to connect to Internet2 CPS. • You will need to enable 802.1Q VLAN encapsulation on your Ethernet connection to Internet2, or frame-relay encapsulation on a SONET connection. Add an additional VLAN (or DLCI) to carry CPS traffic. • Assign IP addresses (/30 or /31) in the usual way. MTU is 1500. • Supply the NOC with a list of ASs behind you, or reference an AS-SET object you maintain. • Supply the NOC with a prefix list of what you will advertise to CPS, or agree to use your existing Internet2 prefix lists, or provide a diff. • Set up the BGP peering - it’s with AS 11537 (same as R&E network). • Pad towards your direct commercial providers as desired to shift inbound traffic away from them, letting end commercial networks see the CPS path as “better”. • Local-pref CPS higher, so your outbound traffic prefers CPS over your direct commodity providers, as desired.

6. Of course, we didn’t cover… • Analyzing your current commercial traffic patterns, if any, to get a baseline. • Checking that your circuit to Internet2 can handle the additional load without affecting R&E traffic. • Figuring out how, or if, this will interact with your existing bandwidth shapers, firewalls, etc., if any. • Considering how a circuit failure to Internet2 or one of your other commercial providers will be handled. • Figuring out how you will distribute this to downstream connectors, how/if it will be measured/charged… • Educating your downstream connectors.

7. But I want more control! • We give you some! It not perfect, but it’s pretty easy to understand and implement. • You can attach certain communities to your prefixes you advertise to Internet2 CPS that in turn, affects how we advertise your prefix to commercial peers. (details next slide) • You can, if you like, configure your network policy to allow your downstreams to do this themselves. • Combining this with adjustments to your import policy, gives you better control. • We also support blackhole routing for up to /24s.

8. Using communities in CPS • Inbound traffic, outbound route policy: • If you do not want CPS peer network X to send traffic to you over Internet2 CPS, you can tag your prefixes with the BGP Community 65000:<foo> where ”<foo>" is the BGP ASN of peer network X. • CPS has an outbound policy specific to each commercial peer that will prevent the advertisement of your prefix to it appropriately, CPS-AS<foo>-OUT. • Note we do NOT do this “per location”, eg, “advertise my route to Shaw (AS6327) in New York but not Seattle”.

9. Using communities in CPS • For example: • Suppose your downstream customer <bar> has called you up to say they don’t want YouTube to send traffic destined to their dorm network over CPS, ever, not even as a last resort. Though they want other traffic to come across CPS destined to that network. • In your BGP policy with CPS, apply policy outbound that tags that dorm network prefix (or prefixes) with 65000:36561, since 36561 is YouTube’s AS number. • When CPS processes what it advertises to AS36561, it will leave out those prefixes. • YouTube will not have a path across Internet2 CPS to your downstream’s dorm network. Traffic will not come in that way. • That dorm network prefix will still be advertised to all other CPS commercial peers.

10. Of course, this is not perfect… • For example: • YouTube is now moving behind the Google AS (AS15169). • CPS has peerings with Google and with YouTube. • Does this mean you should tag that dorm network prefix with 65000:15169 and 65000:36561? Maybe, Maybe not. • Therefore, in some ways, this is better for “traffic engineering/balancing/management” than to “ensure” certain traffic doesn’t come across CPS. • For example, if you already have a decent path to LimeLight, and you need to keep up a certain minimum bandwidth usage, you might want to tag all your prefixes with 65000:22822 towards CPS. So CPS will not advertise your prefixes to LimeLight at all.

11. But, I’d like CPS as a backup • Inbound traffic, outbound route policy: • If you want this “globally” regarding CPS, you can of course simply pad towards CPS in hopes of influencing CPS commercial peers. • What if you want CPS to be your primary commercial path (for the routes CPS offers), EXCEPT for traffic from peer Y -- for peer Y, you want them to send your traffic across CPS only as a “last resort”? • The CPS outbound policy specific to each commercial peer can pad your prefix with the Internet2 AS (AS11537) one, two, or three times, whichever you choose. • This may cause peer Y to see the path across CPS to your prefix as less desirable, leaving it as a backup. • Note we do NOT do this “per location”.

12. Using communities in CPS • Specifically, if you want traffic from peer network Y to prefer a different path, but want to use the Internet2 CPS path as a backup, you can tag your prefixes with 65001:<bar>, 65002:<bar>, or 65003:<bar> where ”<bar>" is the BGP ASN of network Y. • These communities will cause Internet2 to pad the AS-PATH towards peer network Y 1, 2 or 3 times respectively (using AS11537 for the pad) for those prefixes you tag.

13. Using communities in CPS • For example: • Suppose you see traffic from Akamai is preferring Internet2 CPS. For whatever reason, you’d rather they get to you over a different path, leaving the CPS path as a backup. • In your BGP policy with CPS, apply policy outbound that tags your prefixes with 65001:20940, since 20940 is Akamai’s AS number. • When CPS processes what it advertises to AS20940, it will pad your prefixes with one additional “11537” in the AS-PATH. • If you find that is not enough to influence the inbound traffic from Akamai, you can successively try 65002:20940 and 65003:20940. • Your prefixes will not be padded towards any other CPS peers. • Of course, this is not perfect either, and additionally peers may not be letting decisions fall to AS-PATH length for their own reasons.

14. Using communities in CPS • And there’s always the blackhole community: • We check it’s your prefix first of course. • only allowed /32 to /24. • 11537:911. • sets next-hop to discard

15. Discussion • Comments? • Suggestions?

16. Thank you! http://noc.net.internet2.edu/i2network/commercial-peering-service.html http://www.internet2.edu/network/cp/ litvanyi@grnoc.iu.edu network@internet2.edu