1 / 32

Hardening a Cisco Core Network

Cisco Security Deployment Forum 2003. Hardening a Cisco Core Network. Nicolas Fischbach (nico@colt.net) Senior Manager - IP Engineering/Security Andreas Friedrich (andreasf@de.colt.net) Marc Binderberger (mbind@colt.net) IP Network Architecture 2. April 2003 (v1.1).

elsie
Download Presentation

Hardening a Cisco Core Network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cisco Security Deployment Forum 2003 Hardening a Cisco Core Network Nicolas Fischbach (nico@colt.net) Senior Manager - IP Engineering/Security Andreas Friedrich (andreasf@de.colt.net) Marc Binderberger (mbind@colt.net) IP Network Architecture 2. April 2003 (v1.1) 2. April 2003

  2. Agenda / What is this talk about? • Background • Paradigm: networks should transport DoS • optimized to be able to do so. Fault isolation • Distributed router architecture • queuing, memory, hardware details of 7500/GSR • Using ACLs to block DoS • performance, hardware dependencies • Discussion about Netflow • Conclusion & future work • Q&A 2. April 2003

  3. Denial of Service attacks (I) • Limited resources • link bandwidth • CPU cycles and memory • forwarding performance vs “received” packets processing • Trend: attacks shifting from end systems towards core devices/infrastructure. • why ? • default free routing in the core (magnet) ? • data-center vs core infrastructure approach 2. April 2003

  4. Denial of Service attacks (II) • Detection • ACLs and queue counters • netflow • NMS (CPU, interface counters, etc) • customers • Mitigation • ACLs and CAR • null0 routing, sinkhole routing, traffic rerouting and “cleaning” • de-aggregate block and stop to announce specific prefixes • mark with special community • Traceback • ACLs, netflow, source-tracking, null0 if counters, etc. 2. April 2003

  5. Denial of Service attacks (III) • Community • nsp-security mailing-list • honeybot approach • watch IRC/P2P/etc based communications • run bots in “safe mode” • (Latest) type of attacks • attacks • “special” small packets vs large packets • propagation speed • built-in “intelligence” (random vs targeted propagation) • network/routing protocol stability • Active bots and botnets 2. April 2003

  6. CPU cycles on 7500/GSR • RP/RSP often not very powerful • compared to your PC’s Intel CPU • usually no need, fine for BGP, IS-IS, … • but then SNMP, ICMP attacks easy to deploy • VIP or LC “misuse” is normal • overloading VIP2/50 (>STM-1) works fine but … • engine0/1/2 optimized for forwarding, not for edge features - but before E3 no alternative! • “slow path” via CPU for E2/3/4/4+ • candidates: ACL, IP options, ICMP, mcast 2. April 2003

  7. Tips • block network addresses at boundary • you have a structured address plan, right? ;-) • or: spoofing filter + received ACL (12.0(21)ST) ip receive access-list • but: this is CPU-based, use compiled ACL  Make ACL at boundary part of your design • Watch your VIP/LC CPU usage! • via SNMP since 12.0(21)ST. Or use scripts 2. April 2003

  8. More configuration tips • TTL processing causes trouble • not protected by ip icmp rate-limit ! • stop TTL processing (MPLS networks) mpls ip ttl-expiration pop 1 no tag-switching ip propagate-ttl forwarded • protects the core but not the edge/peering • Options: ignore (or drop?) them (12.0(23)S) ip options drop|ignore • RSVP router-alert option *sigh* 2. April 2003

  9. Forwarding performance • Mainly a problem with engine0/1 • experience: keep LC CPU <60% utilization • “50% rule” - use only half of your LC ports • Engine2 is a backbone LC • features consume PSA cycles, < 4 MPPS • VIP2/50: one STM-1 in total (Cisco advice) • often (?) violated. • Difference PIR  CIR causes the problems 2. April 2003

  10. Feature performance • ACL on top • again: use compiled ACL for VIP, E0 • engine1 forwards with SALSA, may filter with CPU (rev. 3)  activating ACL “dangerous” execute-on slot N show contr L3 | incl ASIC • when rev is >= 4: access-list hardware salsa • Watch performance decrease on E0/1/2 2. April 2003

  11. Reminder: 7500/GSR architecture DMA • Typical problems: • HOL • oversubscribtion • multiple interfaces, one memory VIP / Line Card 2. April 2003

  12. Memory management on VIP/LC • Memory setup: • One buffer pool per VIP/LC, not per port • Queue length “unlimited” • send more traffic than output link capacity output queue grows until out-of-memory • affects whole VIP/LC, other ports get blocked • backpressure through fabric or bus to ingressVIP/LC (for engine0/1) 2. April 2003

  13. Backpressure on 7500 • Problem is Rx-side buffering • when the outbound interface is congested • and the queueing strategy of the outbound interface is First in, First out (FIFO) • Command to check status of Rx-side buffering:show controllers vip accumulator • number of output interfaces present in the router • how many packets VIP has Rx-buffered for these interfaces • why the VIP has Rx-buffered • VIP runs at 99% CPU utilization • five seconds: 99%/97%; one minute: 70%; five minutes: 69% 2. April 2003

  14. Rx buffer on 7500 • Check Rx-side buffering router#show controllers vip 2 accumulator show vip accumulator from Slot 2: Buffered RX packets by accumulator: ..... Serial10/0: MEMD txacc 0x00B2: 544980 in, 2644182 drops (126 paks, 378/376/376 bufs) 1544kbps No MEMD acc: 544980 in, 2644182 limit drops, 0 no buffer No MEMD buf: 0 in, 0 limit drops, 0 no buffer 2. April 2003

  15. Configuring 7500 queues • Avoid Rx-side buffering • queuing and dropping at outbound interface • =< 2Mbit/s fair-queue no dWRED • > 2Mit/s fifo random-detect • rate-limit fifo no dWRED • Advantage • fix overload and dropping to related outgoing interface • avoid possible reaction to other interfaces • see drop counter at related outgoing interface 2. April 2003

  16. Reminder: naming DMA • Naming conventions: • Incoming: “Rx”, “tofab”, “ingress” • Outgoing: “Tx”, “frfab”, “egress” VIP / Line Card 2. April 2003

  17. GSR Tx congestion • When one interface is congested queue grows until #Qelem is zero (or very small) • no memory available to accept packets from fabric • one memory pool, so all interfaces affected • depending on engine: backpressure! • Solution: control maximum queue size • RED is one possible fix • (Sorry: no congestion snapshot available ;-) 2. April 2003

  18. GSR: frfab memory example LC-Slot3#show controllers frfab queues 65534/65534 buffers specified/carved Qnum Head Tail #Qelem LenThresh ---- ---- ---- ------ --------- 26174/26174 (specified/carved), 39.93%, 80 byte 1 16866 20991 26174 65535 19630/19630 (specified/carved), 29.95%, 608 byte 2 43701 45521 19629 65535 13087/13087 (specified/carved), 19.96%, 1568 byte 3 53158 50364 13087 65535 6543/6543 (specified/carved), 9.98%, 4544 byte 4 65086 65085 6543 65535 2. April 2003

  19. GSR config example (I) cos-queue-group stm4-tx precedence all random-detect-label 1 random-detect-label 1 1550 5167 1 exponential-weighting-constant 12 ! interface POS1/0 […] tx-cos stm4-tx • keyword precedence also covers MPLS EXP • CPU impact for engine0 2. April 2003

  20. GSR config example (II) interface GigabitEthernet3/0 […] tx-queue-limit 10000 • engine1 doesn’t support (W)RED • use tx-queue-limit instead • alternative for RED on engine0 to control max queue length (BMA, no CPU impact) 2. April 2003

  21. tofab queues • RED on egress everywhere - we are safe :-) • Hold on! Backpressure may still happen • Potential for overloading the BMA of egress E0 LC with small packets • (250kPPS when WRED is enabled) 2. April 2003

  22. backpressure: just theory? router#execute-on slot 14 show contr tofab queu 262140/262140 buffers specified/carved Qnum Head Tail #Qelem LenThresh ---- ---- ---- ------ --------- 112678/112678 (specified/carved), 42.98%, 80 byte 1 234767 80256 0 262143 83853/83853 (specified/carved), 31.98%, 608 byte 2 171796 151548 2 262143 41926/41926 (specified/carved), 15.99%, 1568 byte 3 36144 198618 0 262143 23583/23583 (specified/carved), 8.99%, 4544 byte 4 24107 240492 0 262143 2. April 2003

  23. backpressure: just theory? router#execute-on slot 14 show contr tofab queu […] ToFab Queues: Dest Slot 0 0 0 0 262143 1 221233 201638 0 262143 2 172267 181071 0 262143 3 205470 232182 262024 262143 4 232695 225979 0 262143 5 0 0 0 262143 6 0 0 0 262143 7 136211 178716 0 262143 8 0 0 0 262143 […] • Head-of-line blocking!  Need to configure queues! 2. April 2003

  24. tofab & virtual queues Rx Line Card Tx Line Card A Tx Line Card C Tx Line Card B Rx Line Card A C A C A A B B • No HOL - when queues are independent • free packet buffers must exist to ensure this Incoming Packets Datagrams Multiple output queue 2. April 2003

  25. GSR config example (III) rx-cos-slot 1 stmX-rx rx-cos-slot 2 stmX-rx […] slot-table-cos stmX-rx destination-slot all stm16-tx ! cos-queue-group stm16-tx precedence all random-detect-label 0 random-detect-label 0 6369 21231 1 exponential-weighting-constant 14 • config per slot (linecard), not per interface • keep it simple: stm-16 profile for all slots 2. April 2003

  26. ACL details on GSR • Already mentioned: • engine0: only software based • engine1: careful, use rev. 4 or newer • engine2: fine - depends on the details • engine3: should be no problem at this time • high performance filtering at exchange points:3port GigE (engine2)  lets look into E2 details 2. April 2003

  27. ACL details on GSR • Avoid output ACL if possible (E0/1/2) • effectively “hidden input” ACL (!) • thus a global (per router) configuration • potential conflicts with PSA code (E2) • Input ACL: avoid following (on E2) … • source port • log/log-input keyword • netmask with non-contigous 0s and 1s • protocol IGMP • ToS 2. April 2003

  28. ACL on engine2 router#execute-on slot 8 show access-list psa sum PSA ACL Configured: yes, Running: yes Access list limits: 1 ingress, 5 egress (max 128 lines each) ACL in microcode configured in output direction (Output ACL microcode loaded) Mic Output ACL: 0 1 2 3 4 ACL total lines: 52 34 0 53 0 Lines on cpu: 1 0 0 3 0 Access List : 120 160 - 140 - • Hits the CPU (R5000/200 on OC48) • Limit per router! • Watch Micro-code dependencies 2. April 2003

  29. ACL on engine2 CPU LC-Slot8#show access-lists psa table output 3 | begin ACL Action ACL Action 3----------------- 0-0: hits: 0, flags: 0x1 (permit mic) 1-1: hits: 130067, flags: 0x3 (permit cpu) 2-2: hits: 902992, flags: 0x3 (permit cpu) 3-41: hits: 0, flags: 0x1 (permit mic) 42-42: hits: 0, flags: 0x3 (permit cpu) 43-52: hits: 0, flags: 0x1 (permit mic) 53-127: hits: 0, flags: 0x0 (deny mic) 128: hits: 0, flags: 0x0 (deny mic) 1&2: permit udp any eq domain host … gt 1023 42: access-list 140 permit ip any host … log 2. April 2003

  30. ACL on E2 and BGP • Constrained by 2x 64MB lookup memory (PLU/TLU) for PSA • Memory for ACL tables is reserved in advance • changing from 128 to 448 line code means … • 115k prefixes on E2 with 448 line output ACL • CEF stops working, leaving LC non-functional (!) • Avoid output ACLs! 2. April 2003

  31. Netflow • Impact on 72xx, 75xx and 12k • “Normal” vs sampled Netflow • Netflow vs. mini-protocol analyzer approach pros and cons permit tcp any any (2243368035 matches) permit udp any any (16108809 matches) permit icmp any any (2183318365 matches) permit gre any any (2784997418 matches) permit ahp any any (42819960 matches) permit esp any any (3048927559 matches) permit ip any any (46610829 matches) 2. April 2003

  32. Conclusion • Conclusion • Related work • IP Backbone Security (BlackHat Briefings US 2002) : http://www.securite.org/presentations/secip/ • Future work • Q&A • Thank you! 2. April 2003

More Related