CS419 – Computer Security

1. CS419 – Computer Security Vinod GanapathyTopic: Intrusion Detection & Prevention

2. Security Intrusion & Detection Security Intrusion a security event, or combination of multiple security events, that constitutes a security incident in which an intruder gains, or attempts to gain, access to a system (or system resource) without having authorization to do so. Intrusion Detection a security service that monitors and analyzes system events for the purpose of finding, and providing real-time or near real-time warning of attempts to access system resources in an unauthorized manner.

3. Principles of Intrusion Detection • Characteristics of systems not under attack • User, process actions conform to statistically predictable pattern • User, process actions do not include sequences of actions that subvert the security policy • Process actions correspond to a set of specifications describing what the processes are allowed to do • Systems under attack do not meet at least one of these

4. Example • Goal: insert a back door into a system • Intruder will modify system configuration file or program • Requires privilege; attacker enters system as an unprivileged user and must acquire privilege • Nonprivileged user may not normally acquire privilege (violates #1) • Attacker may break in using sequence of commands that violate security policy (violates #2) • Attacker may cause program to act in ways that violate program’s specification

5. Goals of IDS • Detect wide variety of intrusions • Previously known and unknown attacks • Suggests need to learn/adapt to new attacks or changes in behavior • Detect intrusions in timely fashion • May need to be be real-time, especially when system responds to intrusion • Problem: analyzing commands may impact response time of system • May suffice to report intrusion occurred a few minutes or hours ago

6. Goals of IDS • Present analysis in simple, easy-to-understand format • Ideally a binary indicator • Usually more complex, allowing analyst to examine suspected attack • User interface critical, especially when monitoring many systems • Be accurate • Minimize false positives, false negatives • Minimize time spent verifying attacks, looking for them

7. IDS Requirements • run continually • be fault tolerant • resist subversion • impose a minimal overhead on system • configured according to system security policies • adapt to changes in systems and users • scale to monitor large numbers of systems • provide graceful degradation of service • allow dynamic reconfiguration

8. Intrusion Techniques • objective to gain access or increase privileges • initial attacks often exploit system or software vulnerabilities to execute code to get backdoor • e.g. buffer overflow • or to gain protected information • e.g. password guessing or acquisition

9. Intrusion Detection Systems • classify intrusion detection systems (IDSs) as: • Host-based IDS: monitor single host activity • Network-based IDS: monitor network traffic • logical components: • sensors - collect data • analyzers - determine if intrusion has occurred • user interface - manage / direct / view IDS

10. Models of Intrusion Detection • Two models of algorithms used for intrusion detection: • Signature-based schemes (also called misuse detection) • Characterize what is “bad” • Everything else is “good” • Anomaly detection-based schemes • Characterize what is “good” • Everything else is “bad”

11. IDS Architecture • Basically, a sophisticated audit system • Sensor: gathers data for analysis • Analyzer: it analyzes data obtained from the sensor according to its internal rules • Notifier obtains results from analyzer, and takes some action • May simply notify security officer • May reconfigure agents, director to alter collection, analysis methods • May activate response mechanism

12. Sensors • Obtains information and sends to analyzer • May put information into another form • Preprocessing of records to extract relevant parts • May delete unneeded information • Analyzer may request agent send other information

13. Example • IDS uses failed login attempts in its analysis • Sensor scans login log every 5 minutes, sends director for each new login attempt: • Time of failed login • Account name and entered password • Analyzer requests all records of login (failed or not) for particular user • Suspecting a brute-force cracking attempt

14. Analyzer • Reduces information from sensors • Eliminates unnecessary, redundant records • Analyzes remaining information to determine if attack under way • Analysis engine can use a number of techniques, discussed before, to do this • Usually run on separate system • Does not impact performance of monitored systems • Rules, profiles not available to ordinary users

15. Notifier • Accepts information from director • Takes appropriate action • Notify system security officer • Respond to attack • Often GUIs • Well-designed ones use visualization to convey information

16. Example GUI • GUI showing the progress of a worm as it spreads through network • Left is early in spread • Right is later on

17. Evaluating IDSes • Effectiveness depends on • False positive rate • False negative rate • Supposing you have a false positive rate of 0.0001% Is that “good”? How do we evaluate “good”? • The Base-rate Fallacy

18. The Base-Rate Fallacy • Pr[Intrusion] = Chance of a real intrusion • Pr[Alert] = Chance that the IDS raises alert • What are these quantities? • Pr[Alert|Intrusion] • Pr[Alert|No Intrusion] • Pr[No Alert|Intrusion] • Pr[No Alert|No Intrusion] • (we will abbreviate “Alert” with “A” and “Intrusion” with “I”)

19. The Base-Rate Fallacy • Bayes’ Rule says: • Pr[I|A]*Pr[A] = Pr[A|I]*Pr[I] • So we have:

20. The Base-Rate Fallacy • Let’s plug in some values: • Suppose that P(I) = 2/100000. Then

22. Signature-based Detection • observe events on system and applying a set of rules to decide if intruder • approaches: • rules identify known penetrations / weaknesses • often by analyzing attack scripts from Internet • supplemented with rules from security experts

23. Misuse Modeling • Determines whether a sequence of instructions being executed is known to violate the site security policy • Descriptions of known or potential exploits grouped into rule sets • IDS matches data against rule sets; on success, potential attack found • Cannot detect attacks unknown to developers of rule sets • No rules to cover them

24. Anomaly Detection • assume intruder behavior differs from legitimate users • observe deviations (“anomalies”) from past history

25. Anomaly Detection • threshold detection • checks excessive event occurrences over time • alone a crude and ineffective intruder detector • must determine both thresholds and time intervals • Model-based based • characterize past behavior of users / groups • then detect significant deviations • based on analysis of audit records • gather metrics: counter, guage, interval timer, resource utilization • analyze: mean and standard deviation, multivariate, markov process, time series, operational model

26. Threshold Metrics • Counts number of events that occur • Between m and n events (inclusive) expected to occur • If number falls outside this range, anomalous • Example • Windows: lock user out after k failed sequential login attempts. Range is (0, k–1). • k or more failed logins deemed anomalous

27. Model-based anomaly detection • Model the behavior of a program as it “typically” executes • E.g., make a model of the system calls that the program can issue during “normal” runtime. • Watch for new (anomalous) system calls at runtime.

28. Network-Based IDS • network-based IDS (NIDS) • monitor traffic at selected points on a network • in (near) real time to detect intrusion patterns • may examine network, transport and/or application level protocol activity directed toward systems • comprises a number of sensors • inline (possibly as part of other net device) • passive (monitors copy of traffic)

29. NIDS Sensor Deployment

30. Network-Based Sensors • Detects network-oriented attacks • Denial of service attack introduced by flooding a network • Monitor traffic for a large number of hosts • Examine the contents of the traffic itself • Agent must have same view of traffic as destination • TTL tricks, fragmentation may obscure this • Example: “finger root” attacks. • End-to-end encryption defeats content monitoring • Not traffic analysis, though

31. Network Issues • Network architecture dictates agent placement • Ethernet or broadcast medium: one agent per subnet • Point-to-point medium: one agent per connection, or agent at distribution/routing point • Focus is usually on intruders entering network • If few entry points, place network agents behind them • Does not help if inside attacks to be monitored

32. SNORT • lightweight IDS • real-time packet capture and rule analysis • passive or inline

33. SNORT Rules • use a simple, flexible rule definition language • with fixed header and zero or more options • header includes: action, protocol, source IP, source port, direction, dest IP, dest port • many options • example rule to detect TCP SYN-FIN attack: Alert tcp $EXTERNAL_NET any -> $HOME_NET any \ (msg: "SCAN SYN FIN"; flags: SF, 12; \ reference: arachnids, 198; classtype: attempted-recon;)

34. Host-Based IDS • specialized software to monitor system activity to detect suspicious behavior • primary purpose is to detect intrusions, log suspicious events, and send alerts • can detect both external and internal intrusions

35. Host-Based Sensors • Obtain information from logs • May use many logs as sources • May be security-related or not • May be virtual logs if agent is part of the kernel • Very non-portable • Sensor generates its information • Scans information needed by IDS, turns it into equivalent of log record • Typically, check policy; may be very complex

36. Buffer-overflow defenses: Non-Executable Stack NX bit on every Page Table Entry Some applications need executable stack Does not defend against return-to-libc exploits Overwrite return address with the address of an existing library function (can still be harmful) …nor against heap and function pointer overflows

37. Embed “canaries” in stack frames and verify their integrity prior to function return Any overflow of local variables will damage the canary Choose random canary string on program start Attacker can’t guess what the value of canary will be Terminator canary: “\0”, newline, linefeed, EOF String functions like strcpy won’t copy beyond “\0” Buffer Overflow defenses: “Canaries” on the stack canary buf sfp ret addr Frame of the calling function Top of stack Return execution to this address Pointer to previous frame Local variables

38. Canary Implementation Requires code recompilation Checking canary integrity prior to every function return causes a performance penalty For example, 8% for Apache Web server This defense can be defeated! Phrack article by Bulba and Kil3r

39. Run-Time Checking: Safe libraries Dynamically loaded library Intercepts calls to strcpy(dest,src) Checks if there is sufficient space in current stack frame |frame-pointer – dest| > strlen(src) If yes, does strcpy; else terminates application topofstack src buf sfp ret-addr sfp ret-addr dest main libsafe

40. Encrypting pointers in memory Attack: overflow a function pointer so that it points to attack code Idea: encrypt all pointers while in memory Generate a random key when program is executed Each pointer is XORed with this key when loaded from memory to registers or stored back into memory Pointers cannot be overflown while in registers Attacker cannot predict the target program’s key Even if pointer is overwritten, after XORing with key it will dereference to a “random” memory address

41. Normal Pointer Dereference CPU 2. Access attack code referenced by corrupted pointer 1. Fetch pointer value Corrupted pointer 0x1234 0x1340 Attack code Memory Data 0x1234 0x1340 CPU 2. Access data referenced by pointer 1. Fetch pointer value Pointer 0x1234 Memory Data 0x1234

42. Dereference with encrypted pointers CPU Decrypts to random value 2. Access random address; segmentation fault and crash 0x9786 1. Fetch pointer value Decrypt Corrupted pointer 0x7239 0x1340 Attack code Memory Data 0x9786 0x1234 0x1340 CPU 0x1234 2. Access data referenced by pointer 1. Fetch pointer value Decrypt Encrypted pointer 0x7239 Memory Data 0x1234

43. Issues with encrypted pointers Must be very fast Pointer dereferences are very common Compiler issues Must encrypt and decrypt only pointers If compiler “spills” registers, unencrypted pointer values end up in memory and can be overwritten there Attacker should not be able to modify the key Store key in its own non-writable memory page PG’d code doesn’t mix well with normal code What if PG’d code needs to pass a pointer to OS kernel?

44. Introducing Artificial Code Diversity Buffer overflow and return-to-libc exploits need to know the (virtual) address to which pass control Address of attack code in the buffer Address of a standard kernel library routine Same address is used on many machines Slammer infected 75,000 MS-SQL servers using same code on every machine Idea: introduce artificial diversity Make stack addresses, addresses of library routines, etc. unpredictable and different from machine to machine

45. Address Space Randomization Randomly choose base address of stack, heap, code segment Randomly pad stack frames and malloc() calls Randomize location of Global Offset Table Randomization can be done at compile- or link-time, or by rewriting existing binaries Threat: attack repeatedly probes randomized binary Several implementations available: PaX

46. Hybrid Network- & Host-Based IDS

47. Combining Sources • Neither network-based nor host-based monitoring sufficient to detect some attacks • Attacker tries to telnet into system several times using different account names: network-based IDS detects this, but not host-based monitor • Attacker tries to log into system using an account without password: host-based IDS detects this, but not network-based monitor. • Many modern “botnet” detection schemes follow this hybrid model

48. Audit Records • a fundamental tool for intrusion detection • two variants: • native audit records - provided by O/S • always available but may not be optimum • detection-specific audit records - IDS specific • additional overhead but specific to IDS task • often log individual elementary actions • e.g. may contain fields for: subject, action, object, exception-condition, resource-usage, time-stamp

49. Honeypots • are decoy systems • filled with fabricated info • instrumented with monitors / event loggers • divert and hold attacker to collect activity info • without exposing production systems • initially were single systems • more recently are/emulate entire networks

50. Honeypot Deployment