Why “better” common practices?

Multicast “Better” Common PracticesCaren Litvanyilitvanyi@grnoc.iu.eduMost work by Matt Davy, mpd@grnoc.iu.eduJoint Techs MeetingBurnaby, BC17 July 2005

Why “better” common practices? • Well, not because this is “better” information than anything else in particular -- it just doesn’t really qualify for “BEST”. • Many practices are suggestions, and YMMV. Also some are dependent on our experience with particular vendors, which may not apply to you. • Vendors change their software. Standards get modified. Experience happens. • There's a lot of uncollected experience out there – compilation and documentation of BCPs is badly needed! • I had a limited amount of time, and this is far, far, from complete. It’s just a start. Hopefully we will improve it.

Turning on Multicast (border router) • The steps, on a Cisco router, are generally: • if this is your RP, define a separate loopback for your RP address and check for route distribution. • Meta-step: define the appropriate filters for all protocols. • enable multicast routing at the global level. • If this is a switch, put mls ip multicast on all interfaces. • If this is not your RP, configure the RP address, and make sure it has a route to it. Use the “override” option! • Enable PIM on all interfaces. • Enable MSDP. • Enable MBGP. • On a Juniper, there is no global multicast command, but there is RIB-group configuration to do.

Turning on Multicast • Is there a logic to this order? -- Yes, sort of. • As with most protocols, it’s best to have your filters in place ahead of time. You don’t want your router falling over the minute you turn this stuff on. • Have your interfaces set to use switching for same reason. • A helpful practice, especially in highly multi-homed networks, is to configure MBGP last, because you don’t want things RPFing towards that nice new connection you’re bringing up if there is, say, no PIM configured. This creates “blackholes” of sorts. • Verify that PIM, MSDP, etc. are working before you turn on MBGP. • Objectives are to protect your CPU, your caches, and not create blackholes.

Design Issues • Does it matter where I put my RP/MSDP speaker? • As with most things, “it depends”. • It use to be common practice to put this at the “top of the tree”. It still is. • However, due to the load imposed by MSDP, sometimes this is now put “near the top, but off to the side”. This way, if your MSDP speaker falls over, you don’t lose everything. However, other protections may be the better strategy. • Anycast RP is recommended. • Can’t I use my <regional, gigapop, provincial, etc.> network’s RP instead of running my own RP? • Yes, you can. From experience though, life seems a lot easier if your PIM domain matches your AS. We recommend you try to align these.

Design Issues • What about firewalls? • Used to be, you had to static mroute around these, do all kinds of tricks.... • There are firewall products which support full participation in multicast routing. We suggest going this route, and have gotten some input that this works. • This is not an area I have much familiarity with. • Discussion?

Design Issues • Other random suggestions • Seem to run into less “issues” if you keep your multicast subnets “relatively small”. Giant, flat, switched networks with thousands of hosts are difficult to troubleshoot and can easily run into performance and resource exhaustion issues. • Unfortunately, the “easy” solution was to disable multicast! • Design subnet/vlan scope and size with multicast in mind. • If possible, heavy external multicast users may benefit by being topologically “closer to the border”. Less switches to cascade through, in particular. • Multicast users behind a long series of switches of different make/model/software-version/generation seem to often be plagued with weird problems. Try to avoid this.

Design Issues • More random suggestions • Many people want to “try” multicast on a very limited segment of the network. • This is understandable to start. However, as things grow, the balance can tip rather quickly to where it is more trouble than it is worth to try to limit where multicast goes. • If possible, plan to go from your “limited test” to “general deployment”, rather than planning on implementing multicast a segment at a time where needed. • Of course this is not always possible. • Others?

BCPs for Multicast TCAM Problems • First make VLAN ACLs to block ICMP and TCP to 224/4. This will at least catch poorly-written address scanners that are either pinging or trying to opena TCP connection. • On 6500 do "mls ip multicast threshold" to limit the packet rate at which multicast entries get added to the TCAM. This way, slow scans don't take up TCAM slots. • Note you need to be careful here - things below the threshold get routed in software. The tradeoff is not taking up a switch-cache entry. 5 pps? • Graph the mroute MIB to see what ASM groups are popular. You'll be floored by how many belong to Norton Ghost. 229.55.150.208 is one of them.

BCPs for Multicast TCAM Problems A note on the sup720 from Cisco docs: By default, a Supervisor Engine 720 supports 32,000 multicast routes in sparse mode. The following default settings apply for maximum number of multicast routes: • 32,000 for PIM-SM/DM/SSM for ingress- or egress-replication mode • 32,000 for IPv4 bidirectional PIM ingress-replication mode • 10,700 for IPv4 bidirectional PIM egress-replication mode • By entering the mls ip multicast max-routes command, you can increase the maximum number of multicast routes to 64,000 for PIM-SM/DM/SSM with ingress- and egress-replication mode.

Multicast TCAM Problems • Set up multicast boundaries for admin scoped groups like 239.255/16 to be link local unless you really care about them working campus-wide. These include Microsoft SSDP and Apple's discovery protocols. • Block IAPP (Inter Access Point Protocol) if you aren't doing so already. (deny ip any host 224.0.1.76) • See www.abilene.iu.edu/mccook.html#MSDP_Filter_contentsand www.abilene.iu.edu/mccook.html#furtherfor other bogons to block. • AARNet has some great examples: http://www.aarnet.edu.au/engineering/guides/multicast/member-implementation.html

A multicast boundary example (from AARnet) ip access-list standard MULTICASTRNOADMIN remark block Cisco auto-RP deny 224.0.1.39 deny 224.0.1.40 remark reserved rfc2365 deny 239.0.0.0 0.127.255.255 deny 239.128.0.0 0.63.255.255 remark member only usage deny 239.224.0.0 0.15.255.255 remark reserved rfc2365 deny 239.253.0.0 0.0.255.255 deny 239.254.0.0 0.0.255.255 deny 239.255.0.0 0.0.255.255 remark allow all remaining multicast remark including RNO and AARNet scope permit 224.0.0.0 15.255.255.255

But I’m getting ahead of myself...Some BCPs forMulticast Security

What’s Unique About Multicast? • By simply sending an IP packet, any host can: • create control plane state in routers & switches • force routers & switches to generate & process protocol packets • flood a large number of hosts with a large traffic stream

Why Is This a Problem? • Hosts can intentionally or unintentionally generate a DoS attack on multicast-enabled routers & switches by overloading the control plane. • Worms which scan 224/4 are the most common problem. • Several worms have unintentionally disrupted many multicast enabled networks (ramen, slammer, etc.)

Sender-Based Vulnerabilities: ASM • When host sends a packet to a 224/4 address: • The first router (aka the PIM DR)... • creates a multicast route (S,G) • result = memory allocation on RP/RE (rib) and forwarding hardware (fib) - potential for memory exhaustion • encapsulates data packet inside PIM register and sends to RP • result = processor cycles on the DR & RP - potential for CPU exhaustion

Sender-based Vulnerabilities: ASM • The PIM RP... • receives PIM Register [processor] • creates (S,G) state [memory] • de-encapsulates the data packets [processor] • forwards the packets down the shared tree [processor] • sends PIM join towards source [processor]

Sender-Based Vulnerabilities: ASM • If it’s also an MSDP speaker, the RP... • creates MSDP SA state [memory] • sends MSDP SA with encapsulated data to all MSDP peers [processor] • Note: MSDP SAs are flooded to every MSDP speaker on the Internet !

Sender-Based Vulnerabilities: ASM • Every MSDP speaker on the Internet... • receives the MSDP SA and deencapsulates the data packet [processor] • creates MSDP SA state and forwarding state [memory] • forwards the data packet down shared tree [processor]

Does SSM Solve This Problem? • SSM does not have sender-based vulnerabilities ! • First hop router simply drops packets if no forwarding state (hopefully in ASIC) • No PIM Registers = no data packets inside control plane packets • No MSDP = packets can’t reach all MSDP speakers & no data packets inside control plane packets • SSM still has receiver-based vulnerabilities

Receiver-Based Vulnerabilities: SSM & ASM • When a host joins a multicast group, it sends an IGMP host report packet to a multicast group • Ethernet switches often snoop IGMP packets [memory & processor] • The first hop router... • creates (*,G) and/or (S,G) state if necessary [memory] • sends PIM join towards RP (ASM) or towards source (SSM) [processor]

Receiver-Based Vulnerabilities: SSM & ASM • Every router in the path... • receives a PIM join packet [processor] • creates forwarding state as necessary [memory] • Unintentional receiver-based attacks are unlikely

Protection Options: Sender-Based • On first-hop routers, filter multicast packets to “unusable” groups. See: • http://www.iana.org/assignments/multicast-addresses • http://www-fp.mcs.anl.gov/~nickless/draft-nickless-ipv4-mcast-unusable-02.txt • Prevents creation of forwarding state and PIM register processing for unusable groups

A Bit on “Unusable” Groups • Ethernet MAC overlaps with 224.0.0.0/24 (225.0.0.0/24, 225.128.0.0/24, etc) • Should not use, but a few people are • What about “reserved” addresses ? • 224.3.0.64 - 224.251.255.255 • 225.0.0.0 - 231.255.255.255 • 234.0.0.0 - 238.255.255.255 • Might reduce impact of worms significantly by eliminating use of this address space.

Protection Options: Sender-Based • On PIM RP, filter register packets. Only allow packets from your source addresses and “usable” group addresses. • ip pim accept-register list pim-register-filter • This prevents unnecessary register processing and forwarding state creation on the RP. • Redundant if all DRs have same filters, but... can’t hurt. • Consider rate-limiting registers: • ip pim register-rate-limit <x>

Protection Options: Sender-Based • On all MSDP speakers... • filter SAs by source, group, & RP as appropriate (see “unusable” addresses). • Only allow your GLOP space going out, assuming you’re non-transit. • set limits on total SAs from each peer. • For peers, filter out “unusuable groups”. • Recall reserved: • 224.3.0.64 - 224.251.255.255 • 225.0.0.0 - 231.255.255.255 • 234.0.0.0 - 238.255.255.255

Glop review GLOP addresses: • Provides globally available private Class D space • 233.x.x/24 per AS number • RFC 2770 How? • Insert the 16-bit AS number into the middle two octets of the 233/8 • Online GLOP calculator:www.shepfarm.com/multicast/glop.html

Protection Options: Sender-Based • On all MSDP speakers... • set per-source SA limits (Juniper); cool feature. need per-source PIM Register limits too. • set per-instance SA limits. • rate-limit all MSDP traffic destined to router. • turn off data encapsulation for MSDP?

term bad-groups { from { route-filter 224.0.1.2/32 exact; route-filter 224.0.1.3/32 exact; route-filter 224.0.1.8/32 exact; route-filter 224.0.1.22/32 exact; route-filter 224.0.1.24/32 exact; route-filter 224.0.1.25/32 exact; route-filter 224.0.1.35/32 exact; route-filter 224.0.1.39/32 exact; route-filter 224.0.1.40/32 exact; route-filter 224.0.1.60/32 exact; route-filter 224.0.2.1/32 exact; route-filter 224.0.2.2/32 exact; route-filter 224.77.0.0/16 orlonger; route-filter 225.1.2.3/32 exact; route-filter 226.77.0.0/16 orlonger; route-filter 229.55.150.208/32 exact; route-filter 234.42.42.40/30 orlonger; route-filter 234.142.142.42/31 orlonger; route-filter 234.142.142.44/30 orlonger; route-filter 234.142.142.48/28 orlonger; route-filter 234.142.142.64/26 orlonger; route-filter 234.142.142.128/29 orlonger; route-filter 234.142.142.136/30 orlonger; route-filter 234.142.142.140/31 orlonger; route-filter 234.142.142.142/32 exact; route-filter 239.0.0.0/8 orlonger; route-filter 232.0.0.0/8 orlonger; } then reject; } term bad-sources { from { source-address-filter 10.0.0.0/8 orlonger; source-address-filter 127.0.0.0/8 orlonger; source-address-filter 172.16.0.0/12 orlonger; source-address-filter 192.168.0.0/16 orlonger; } then reject; } term allow { then accept; }

term bad-sources { from { source-address-filter 10.0.0.0/8 orlonger; source-address-filter 127.0.0.0/8 orlonger; source-address-filter 172.16.0.0/12 orlonger; source-address-filter 192.168.0.0/16 orlonger; } then reject; } term allow { then accept; }

Protection Options: Sender-Based • On all multicast routers... • rate-limit total PIM traffic to the router. • rate-limit all 224/4 traffic to the router. • You can rate-limit multicast in and out on a per-interface basis. • block mcast packets to “unusable” groups.

Protection Options: Sender-Based • On all multicast routers... • only allow udp to 224/4; exceptions for PIM, OSPF, etc. • disable sdr/sap • set forwarding table limits (Juniper) • set routing-options multicast forwarding-cache

Protection Options: Receiver-Based • On all multicast routers... • rate-limit PIM and IGMP packets • per-interface multicast route limits would be useful • Per-port MAC limits in switches; not sure if this applies to IGMP snooping. If it doesn’t, it should.

Last thoughts • SSM solves sender-based vulnerabilities. Will ASM cease to be supported for inter-domain? • Blocking reserved groups would help a lot. • So would turning off data encapsulation for MSDP. • So would per-source PIM and MSDP limits. • Need more features from vendors to protect multicast-enabled routers & switches.

Information Online • http://multicast.internet2.eduespecially the tutorial-style paper at http://multicast.internet2.edu/almeroth.pdf • www.ncne.nlanr.net/documentation/faq/mcast_eng_faq.html • http://dast.nlanr.net/projects/Beacon/ • www.multicasttech.com/faq/multicast_faq.html andwww.multicasttech.com/status/ • Greg Shepherd’s multicast site: www.shepfarm.com/multicast/ • www.cisco.com/warp/public/cc/pd/iosw/tech/ipmu_ov.htm and ftp://ftpeng.cisco.com/ipmulticast.html • www.sprint.net/multicast/faq.html

Where to ask Questions • The Internet2 Multicast Working Group mailing list is a good place to share experiences and get help. http://multicast.internet2.edu/wg-multicast-list.shtml

Why “better” common practices?