Formal Methods for Intrusion Detection

1. Formal Methods for Intrusion Detection Presented by Brian Kellogg CSE 914: Formal Methods for Software Development Michigan State University December 11th, 2002

2. Purpose and Method • Find intrusion detection methods that utilize formal methods • Analyze strengths and weaknesses of each method • Compare the methods and see if they can be combined in such a way to improve one another • Found three research papers on intrusion detection that used formal methods for different purposes

3. Intrusion Detection Quickie • The SANS institute defines intrusion detection as “the art of detecting inappropriate, incorrect, or anomalous activity” • Two types: • Host-based: detects intrusions on a specific host • Network-based: detects intrusions on a network • Two (main) methods: • Knowledge-based • Determine vulnerabilities and attempts to detect vulnerabilities • Low false alarm rate • Attacks not specified are not detected • Behavior-based • Determines normal system activity • High false alarm rate • Able to detect many intrusions (even ones not previously known)

4. Intrusion Detection Continued • Why use intrusion detection, why not just prevent the attacks? • Firewalls can prevent many attacks, but have no power over the internal network • Certain network activities that have legitimate uses can also signify an attack (e.g. port scans) • What should an intrusion system do when it detects an attack? • Responses range from e-mails to reconfiguring the network • Just because the system detects an intrusion, may be legitimate • Severe (or even simple) responses can be utilized by attackers to create new attacks

5. Yasinsac Paper (Motivation) • “An Environment for Security Protocol Intrusion Detection” • Traditional methods of protocol analysis not fool proof or complete • Different protocols running concurrently can create new exploits • Shift to “tunneling” paradigm in networks • Sensitive data sent over same links as non-sensitive data • Cryptographic techniques must be applied at higher layer (application layer)

6. Yasinsac Paper (Method) • Take knowledge gained from formal analysis of security protocols and make them in to intrusion signatures • Uses both knowledge-based and behavior-based intrusion detection • Knowledge-based: signature an ordering of activity traces • Behavior-based: surveys taxonomies and protocol principles to determine profile strategies and behavior recognition • State-based attack recognition

7. Yasinsac Paper (Method) • IKE protocol: • AB: HDR1, SAA, KEA, NA, A • BA: HDR2, SAB, KEB, NB, B, {prf(KAB, (KEB, KEA, KEB, KEA, B))}KB • Exploit: • AB: HDR1, SAA, KEA, NA, A • IB: HDR1, SAA, KEA, NA, I • BI: HDR2, SAB, KEB, NB, B, {prf(KAB, (KEB, KEA, KEB, KEA, B))}KB

8. Yasinsac Paper (Architecture) • Central monitor, each principal communicates with monitor through secure channel

9. Pouzol Paper • Motivation: • Algorithm that detects attacks in a declarative IDS is a black box • Partial instances of attacks can choke an IDS • Wants to give more power to security officer to choose which attack instances are important • Method: • Formally specify intrusion signatures and detection rules • Create a lattice used to define equivalence classes that defines a signature • Choose an equivalence relation that can reduce the number of instances reported

10. Т {U1, U2, T1, T2, T3} {U1, U2, T3} {U1, U2} {U2, T3} {U1} {U2} {T3} { } Pouzol Lattice U1U2T3: In this equivalence class, every instance that has a unique pair of users and a third time stamp will be reported. This is an example of a good choice. This class will resist the choking attack, and will report all completed instances of an attack. Having the final timestamp means that the last part of the attack occurred, thus only a completed attack is being reported.

11. NetSTAT Paper (Motivation) • “NetSTAT: A Network-based Intrusion Detection Approach” • Motivated by the increase of network reliance and attacks • Host-based intrusion detection fails to detect these attacks • Firewalls do an excellent job of preventing external intrusions, but internal threats are left unchecked

12. NetSTAT Paper (Method) • NetSTAT is a network-based intrusion detection system • Wants to solve: • Networks generate large amounts of data • Some attacks occur only in a certain portion of a network • Too much communication between IDS components can clog a network • Networks can grow very large • Able to work with host-based methods • Four components: • A network fact base • A state transition scenario database • Many general purpose probes • An analyzer

13. NetSTAT Paper (Method) • Network fact base • Stand alone application that describes network topology and network services • Contains interfaces, hosts, and links • Represented as a hypergraph • Interfaces are nodes, hosts and links are edges • This is a formal model, adds benefits: • Well defined semantics • Supports reasoning and automation • Topological properties described in expressive way

14. NetSTAT Paper (Method) • State transition scenario database • Contains signatures of attacks • Attacks are sequences of states (snapshots) • States are described by assertions that return Boolean values • Example: i.link.type==”ATM”; • Probes • Sensors that are strategically placed in a network but are also full blown intrusion detection system • Made up of: • Filter that only collects data of interest • Inference engine contains attack scenarios • Decision engine issues response according to information collected in the inference engine, or reports info to the analyzer

15. NetSTAT Paper (Method) • Analyzer • Takes as input a network fact base and a state transition scenario • Tells security officer where probes are needed • Sets up the probes • It determines: • Events to be monitored, • The network topology • State information it requires to verify state assertions

16. NetSTAT Paper (Architecture)

17. Analysis: Yacinsac • Advantages • Able to find flaws in protocols that get past formal analysis • Able to detect flaws in concurrently running protocols • Architecture is cheap and versatile • Disadvantages • How do you choose the sources for signatures? • How many signatures is too many? • Architecture • Every single principal required to run software to report to central authority • Intruders can disable software • Network attacks can still occur unnoticed

18. Analysis: Pouzol • Advantages • Allows security officer to specify an equivalence relation to prevent choking attacks on the IDS • Formal specification of signatures and detection rules proven sound and complete • Disadvantages • Has not been implemented in any IDS • Complexity of algorithm may create choking attacks • Equivalence relations can be dangerous if configured incorrectly

19. Analysis: NetSTAT • Advantages: • Can detect intrusions on multiple sub-networks and total network • Scalable to large networks • Formal methods allow expressiveness and automation • Disadvantages • Not yet fully implemented • Analyzer does ad hoc configuring of probes

20. Combination • Pouzol’s technique to prevent choking attacks can be used by Yasinsac (and NetStat) • Two full intrusion detection architecture • Which one is best? NetSTAT! • Yasinsac’s knowledge base can be used by NetSTAT (and all IDS)

21. Conclusion • Formal methods and intrusion detection can work together to make networks more secure • There are many different areas where formal methods can be applied • Neither is a silver bullet to network security • Attackers are always evolving new techniques to attack a network, and as security experts, so must we

22. Main References • A. Yasinsac. An Environment for Security Protocol Intrusion Detection. Special edition of the Journal of Computer Security, 2001 • J. Pouzol and M. Ducassé. Formal Specification of Intrusion Signatures and Detection Rules. 15th IEEE Computer Security Foundations Workshop, June 2002 • G. Vigna and R. Kemmerer. NetSTAT: A Network-based Intrusion Detection Approach. Computer Security Applications Conference, 1998