Securing and Monitoring 10GbE WAN Links

1. Securing and Monitoring 10GbE WAN Links • Steven Carter • Center for Computational Sciences • Oak Ridge National Laboratory

2. Disclaimer • Oak Ridge National Laboratory does not endorse any particular product. This presentation merely details our experience and chosen course of action (i.e. I am not a patsy for Force10).

3. Requirements • Wire rate intrusion detection (i.e. 20Gb/s) • Little or no latency • Low administrative/development overhead • Flexible (used for IDS and protocol monitoring) • Scalable (We have 5+ 10G links that we would like to monitor) • Affordable

4. Approaches • Divide and Conquer: Use a piece of network equipment (e.g. Juniper Router) to divide the stream of packets by some attribute (e.g. destination port) into smaller, more easily handled streams for processing.

5. Approaches (Cont.) • Host intensive: Send the full (or possibly filtered) stream to the host CPU for inspection. • NIC intensive: The NIC does the packet inspection.

6. The Contenders • Intel, Neterion, Chelsio 10G NICs • Endace DAG 6.2SE • Force10 P-Series (formally MetaNetworks)

7. Initial Pros/Cons • Standard 10G NICS • Inexpensive • Single host unable to keep up with full rate, full duplex connection • Endace DAG 6.2SE • Offload allows single host to inspect more traffic (~13Gb/s), but you need a beefy host. • Timestamps • Only available with 1310nm optics • Expensive

8. Initial Pros/Cons (cont) • Force10 P-Series • Less expensive • Compete offload • Scalable • Can block packets if used in-line • Supports too few snort rules (700 shared between 2 channels) • Long compile time • PCI Bus (1Gb/s b/w the card and the host)

9. Initial Test Setup DAG Port Mirror Optical Tap Host Host Switch Switch Host Host Host Host P-Series Saturating Traffic (~10Gb/s) Simulated Nefarious Traffic

10. DAG Results • Circular Buffer started overflowing ~5Gb/s (could likely be tuned better) • Not a generic network interface (Either use the provided dag* utilities or a special version of libpcap) • Only one tool can be used at a time

11. P-Series Results • Able to handle full rate (~10Gb/s) • Interface presented as generic interface (i.e. can run Bro, Snort, and tcpdump simultaneously) • Supports too few snort rules (700 shared between 2 channels)... you have to choose well • Long compile time (long test cycles)

12. Our Decision • The DAG 6.2SE is way too expensive for what you get. We could not afford to use it on 5+ links • The Force10 P-Series had the best strategy and would scale best to fit our needs. Although the card doubled in price, the next generation is slated to have stateful firewall features, more real estate, and a PCI-X (should be PCIe) interface. This makes for a very cost effective, flexible, firewall, IPS, and protocol analysis solution.

13. Working Around the Rule Limitation • Send known low-rate traffic (ICMP, DNS, HTTP, etc.) to the host CPU to be compared against full complement of Snort rules. • Send the first few packets of every connection to the host CPU to be compared against full compliment of Snort rules (either via state register or through the API). • Use the rules on the card for high-rate traffic.

14. Final Setup • 3U Dual 2.8Ghz Opteron • 8 GB RAM • 3TB of internal RAID 5 storage • 2 P-Series cards (room for a third)

15. Final Testing “Real” Internet Traffic Border Router Switch P-Series Host Host Saturating Traffic (~9Gb/s)

16. Conclusion • The Force10 P-Series takes a good approach to the problem. It allows us to secure and monitor several 10G links for a reasonable price. The next generation is even more promising allowing the merging of IPS with firewalling capabilities.

17. Questions?