Firewalls at Rhodes

1. Firewalls at Rhodes David Siebörger System Administrator, I.T. Division

2. Why use firewalls, where? • Open source firewall technologies • About ipfw • About dummynet • About pf • About altq • Lessons learnt from ResNet

3. Where the firewalls are

4. Why firewall the Internet? • Force users to use our HTTP proxies and SMTP relays • Protect users’ PCs from the Internet • Prevent users from using certain applications or providing services • Monitor traffic use

5. ResNet firewalls • Each residence is a separate subnet on a separate VLAN. • A FreeBSD machine routes between subnets, so its firewall can control all inter-res traffic.

6. Why firewall ResNet? • Stop the spread of viruses/worms that use network exploits to propagate. • e.g., Nimda, Code Red, Sasser • Though we try to force students to use antivirus software and SUS, many don’t use it, or don’t keep it up-to-date. • So when a virus breaks out, it’s worst on ResNet. • We can’t predict when the next outbreak will happen, but we’re better prepared for it.

7. Why firewall ResNet? • Keep untrusted users away from important machines. • Prevent users running their own proxy servers. • Allows more detailed monitoring.

8. FreeBSD as the operating system Firewalls: ipfw pf Traffic shaping: dummynet Monitoring: home-grown scripts + rrdtool net-snmpd + rrdtool + Cacti What we’re using

9. ipfw • Appeared in FreeBSD 2.0 (1994) • Quite easy to use: • Numbered ruleset; first match wins • deny tcp from any to me http • Disadvantages: • IPv6 support isn’t quite there yet. • Some strange new syntax.

10. dummynet • Originally intended as a network simulator. • Creates pipes with bandwidth, delay, packet loss parameters. • ipfw rules direct traffic flows through the pipes. • For high-bandwidth (ab)users, we force their traffic through a low-bandwidth.

11. pf • OpenBSD dropped ipf, and developed pf to replace it. pf has developed substantially since. • pf has since been imported into FreeBSD. • Slightly counter-intuitive: last match wins. • Otherwise elegant, regular syntax. • block proto tcp to self port http

12. pf • Interesting features: • pfsync – synchronises dynamic rules across two firewalls. • carp – two firewalls bind the same IP address (a patent-free alternative to VRRP). • Combined, they allow failover within seconds. • pfflowd – provides NetFlow traffic accounting. • authpf – network access after authentication. • scrubbing – tidy up fragments, etc. • All-in-one configuration file, with macros, etc.

13. altq • Handles queuing on particular interfaces. • Has a number of algorithms: • CBQ: Class-Based Queuing • RED: Random Early Detection • RIO: Random Early Drop • HFSC: Hierarchical Packet Scheduler • PRIQ: Priority Queuing

14. Firewall performance: a case study • The problem: our ResNet firewall is a choke-point (by design), and we have lots of bandwidth-hungry users on ResNet. • When the firewall can’t handle the traffic, network bandwidth lies idle and everyone suffers. • It’s most noticeable for interactive applications • And unfair: the SSH users aren’t causing the problem.

15. Relevance in other situations • But does firewall performance matter when your Internet connection is < 10 Mbps? • Yes: • Local users might be able to DoS your firewall. • Some day, we will have more bandwidth (SANREN?) and we’d rather design our firewalls now to handle higher volumes.

16. Possible solutions • Ignore the problem. We’d be providing poor service to our paying customers. • Reduce the demand for bandwidth. It means creating restrictive rules, and it starts an arms-race. • Increase the available bandwidth. It’s taken some work, but we’re ahead of demand now.

17. Flat-topped all day at ~13.5 kpps in both directions Flat-topped at ~22 kpps in either direction Results • From… (April 2005)

18. Results • To… (now) No flat-topping: short spikes up to ~75kpps in some cases. Roughly bell-shaped over the day.

19. Lessons learnt • Monitor traffic for informed decisions • Look for bottlenecks elsewhere • Watch for interface errors • Reduce interrupt rate… • … but increase HZ • Consider PCI bandwidth • Reduce rule set size • Stateful rules save CPU • Performance testing is hard • Design for scalability

20. 1. Monitor traffic to inform decision making • We’ve used a home-grown system and, more recently, Cacti to monitor network traffic flows. • Cacti gathers data from a number of sources (primarily SNMP) and stores and graphs the data using rrdtool. • http://www.cacti.net/ • http://www.rrdtool.org/

21. 1. Monitor traffic to inform decision making • For the ResNet firewall, we graph • bits/second, • packets/second, • errors/second on the Ethernet interface, • bits/second on each VLAN, • CPU utilisation, • number of connected PCs. • As well as the aggregation switches. • (and much more)

23. 2. Look for bottlenecks elsewhere • We found that some aggregation switches on 100 Mbps uplinks were saturating their uplinks. • We’ve upgraded a number of them to 1 Gbps.

24. 3. Watch for interface errors • We were seeing up to 800 input errors per second on the Ethernet interface. • The machine wasn’t processing packets fast enough, so the NIC was discarding incoming packets. systems@slug$ netstat -I em0 Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll em0 1500 <Link#1> 00:0c:f1:c3:78:99 2065687990 8457 1547628913 0 0

25. 3. Watch for interface errors • Packet loss causes TCP to throttle back large transfers, and makes SSH really jittery. • With every performance improvement we’ve made, we’ve checked to see that the number of input errors decreases.

26. 4. Reduce interrupt rate… • Generating an interrupt per packet is too inefficient at > 100 Mbps. • Can lead to live-lock. • Device polling sets the NIC to not generate interrupts, and the OS polls on every clock interrupt. • Certain device drivers (fxp, em, etc.) can delay interrupt generation. Sometimes called interrupt mitigation.

27. 4. Reduce interrupt rate… • The result is that the OS can receive multiple frames from the NIC and process them as a batch. • Which saves context switch overheads.

28. 5. … but increase HZ • Clock interrupt frequency is determined by the “HZ” value compiled into the FreeBSD kernel. • In FreeBSD 5.X, the default is 100. • We found that polling reduced CPU utilisation, but maximum PPS was still limited. • Either because the NIC’s buffers were full, • Or the OS would only fetch so many at a time.

29. 5. … but increase HZ • In FreeBSD 6.X, the default has been increased to 1000. Consensus seems to be that 100 is too low for modern computers, never mind routers. • We’ve increased HZ to 2500 on our firewall. • Though we may be able to lower that now since we’ve made other improvements.

30. 6. Consider PCI bandwidth • Using an Intel 1000baseSX NIC, we found that we were limited to ~ 400 Mbps inbound and ~ 400 Mbps outbound (simultaneously). • The NIC has a 64-bit, 66 MHz PCI-X interface… • But the Intel S875WP1 board only has 32-bit, 33 MHz PCI slots.

31. 6. Consider PCI bandwidth • 32 × 33000000 = 1056 Mbps • That’s the theoretical maximum, but bandwidth is shared with everything else on the bus. • The onboard NIC is connected via a “Communications Streaming Architecture” link – effectively a dedicated 32-bit, 66 MHz PCI bus. • Using that interface has allowed us to peak at up to 730 Mbps in both directions – possibly higher.

32. 7. Reduce rule set size • In ipfw, every packet is evaluated against every rule until it matches. • Which uses a lot of CPU time when you’re handling tens of thousands of packets a second. • Reduce the number of rules. • Optimise the rule set by placing frequently matched rules higher, and/or skipping over less frequently matched rules.

33. 7. Reduce rule set size • We started with 200 rules… • Using skipto rules, we reduced the number of rules that most packets would be tested against to about 50. • We’re now down to 28 rules, and almost all packets will have matched after about 12 evaluations.

34. 8. Stateful rules save CPU time • Stateful/dynamic firewall rules create a dynamic rule to allow a specific connection as that connection is established. • The firewall can search the dynamic rule set far quicker than it can evaluate the static rules. • So the firewall only evaluates the static rules once per connection.

35. 9. Performance testing is hard • Generating traffic with iperf, netperf, tcpblast, etc. doesn’t do a very good job of simulating the usage patterns of 1000s of PCs. • We have to try our changes on the live system (but be fairly careful).

36. 10. Design for scalability • Sooner or later, our one machine will reach its limits (probably PCI) • We’re about to deploy a second firewall. • The current firewall will advertise certain prefixes into iBGP; the new firewall will handle the rest. • And we get better redundancy too.