Language-Based Generation and Evaluation of NIDS Signatures

1. Language-Based Generation and Evaluation of NIDS Signatures Shai Rubin Somesh Jha Barton P. Miller University of Wisconsin, Madison

2. Misuse Network Intrusion Detection System (NIDS) Problem: A single attack might have many forms: • Ptacek and Newsham, 1988 • Handley and Paxson, 2001 • Marty, 2002 • Mutz, Vigna, and Kemmerer, 2003 • Vigna, Robertson, and Balzarotti, 2004 • Rubin, Jha, Miller, 2004 • And others... TYPE A \n LIST \n CWD ... Attacker Network NIDS Signature database “TYPE A \n (.)*CWD <long arg>” “TYPE A \n CWD <long arg>\n” Rubin, Jha, Miller

3. Problem: Accurate Signatures • Today, we construct signatures in an ad-hoc manner • Challenges: complex protocols, redundancy • Questions: • Can we systematically construct an accurate signature? • Can we systematically evaluate a signature? • Can we systematically compare signatures? TYPE A \n LIST \n CWD ... Attacker Network NIDS Signature database “TYPE A \n (.)*CWD <long arg>” Rubin, Jha, Miller

4. Contributions • Practical: provide signature writers a methodology and a tool that enables them to systematically construct a signature, evaluate its accuracy, and compare it to other signatures • Conceptual: • a session signature, • a semantic model for an attack protocol, • a language-base approach for signature construction Rubin, Jha, Miller

5. A NIDS Signature TCP Streams • Attack: a set of TCP streams • Signature: a set of TCP streams Sig A Rubin, Jha, Miller

6. A NIDS Signature TCP Streams • Attack: a set of TCP streams • Signature: a set of TCP streams • A prefect signature: Sig=A Sig A Sig=A Rubin, Jha, Miller

7. A A NIDS Signature TCP Streams • Attack: a set of TCP streams • Signature: a set of TCP streams • A prefect signature: Sig=A • Problem: most of the time A is unknown. Difficult to: • construct accurate a signature • evaluate changes to the signature • compare signatures Sig Rubin, Jha, Miller

8. A A NIDS Signature TCP Streams • Attack: a set of TCP streams • Signature: a set of TCP streams • A prefect signature: Sig=A • Problem: most of the time A is unknown. Difficult to: • construct accurate a signature • evaluate changes to the signature • compare signatures Sig Rubin, Jha, Miller

9. Language-Based Approach • Attack: the language Aghost • Signature: the language Lsig • Goal: compare the language • Problem: difficult to determine containment Aghost. • Ideas: • Abstraction: over-approximate Aghost, such that it is easy to determine containment • Automation: Use an automatic tool to compare Lsig and Ainv TCP Streams Ainv Lsig Aghost Rubin, Jha, Miller

10. Language-Based Signature Construction TCP Streams Ainv  fn Lsig  fp Aghost Rubin, Jha, Miller

11. Language-Based Signature Construction TCP Streams Ainv  fn Lsig  fp Aghost Rubin, Jha, Miller

12. Language-Based Signature Construction TCP Streams Ainv Ainv  fn Lsig  fp Aghost Rubin, Jha, Miller

13. Language-Based Signature Construction TCP Streams Ainv Ainv  fn Lsig  fp Aghost Rubin, Jha, Miller

14. Language-Based Signature Construction TCP Streams Ainv Ainv  fn Lsig  fp Aghost  sp Rubin, Jha, Miller

15. Language-Based Signature Construction TCP Streams Ainv Ainv  fn Lsig  fp Aghost  sp Rubin, Jha, Miller

16. Outline • Goal: develop methodology to construct and evaluate signatures • Main idea: use a formal language to approximate Aghost and automatically compare this language to Lsig • The languages • The signature construction process Rubin, Jha, Miller

17. Lsig: A Syntactic Representation of the Attack • Our signature is a regular language • Alphabet: application-level events. For example, FTP commands • A session signature: a string in the language represents the entire attack. • Each signature is a concatenation of three languages: preparation (Lpre), exploitation (Lexp), and confirmation (Lconf) Rubin, Jha, Miller

18. ftp-cwd [CAN-2002-0126] • Preparation: FTP login L L Q logout login Q Rubin, Jha, Miller

19. ftp-cwd [CAN-2002-0126] • Preparation: FTP login • Exploitation: A CWD command with a long argument L L Q logout login Q A such that (length>100 && data (.)*/bin/sh(.)* C login attack Rubin, Jha, Miller

20. Lftp-cwd: ftp-cwd Session Signature A,IR,L • Non-recursive hierarchical state machine • Constructed automatically • Can be analyzed IR A,L 1 attack intrusion IR,L C  L Q C accept  reject Q Q A logout 2 start A,C,IR,Q C Rubin, Jha, Miller

21. Lftp-cwd: Vs. Snort A,IR,L • Non-recursive hierarchical state machine • Constructed automatically • Can be analyzed IR A,L 1 attack intrusion IR,L C  L Q C accept  reject Q Q A logout 2 start A,C,IR,Q C Rubin, Jha, Miller

22. Language-Based Signature Construction TCP Streams Ainv Ainv  fn  fp Session Signature Aghost  sp Rubin, Jha, Miller

23. Ainv: Semantic Representation of the Attack • Another regular language • Models semantics properties: • “Requires FTP login” • “Requires ASCII FTP mode” • “Requires HTTP 1.1” • Using an FSM we model the semantics of the application-level protocol that the attack uses Rubin, Jha, Miller

24. FTP Semantic Model FTP State variables FTP Transitions Rubin, Jha, Miller

25. Language-Based Signature Construction TCP Streams Signature Semantic Model Semantic model  fn Spin  fp Session Signature Aghost String/ NULL FN or FP SP Manual refinement (currently) Automatic comparison Rubin, Jha, Miller

26. Constructing a Signature for ftp-cwd TCP Streams Semantic Model Signature login=1 Spin String/ NULL Rubin, Jha, Miller

27. Constructing a Signature for ftp-cwd TCP Streams Semantic Model Signature login=1 L1 Spin FP1 String FP1 Rubin, Jha, Miller

28. Constructing a Signature for ftp-cwd TCP Streams Semantic Model Signature login=1 L1 Spin FP1 String/ NULL Rubin, Jha, Miller

29. Constructing a Signature for ftp-cwd TCP Streams Semantic Model Signature login=1 L1 Spin L2 FP1 FP2 String FP2 Rubin, Jha, Miller

30. Constructing a Signature for ftp-cwd TCP Streams Semantic Model Signature login=1 L1 Spin L2 FP1 FP2 String/ NULL Rubin, Jha, Miller

31. Constructing a Signature for ftp-cwd TCP Streams Semantic Model Signature login=1 L1 Spin L2 FP1 FP2 L3 FP3 String FP3 Rubin, Jha, Miller

32. Constructing a Signature for ftp-cwd TCP Streams Semantic Model Signature login=1 L1 Spin L2 FP1 FP2 L3 FP3 NULL Rubin, Jha, Miller

33. Constructing a Signature for ftp-cwd TCP Streams login=1 L1 L2 FP1 FP2 L3 FP3 L4 • Comparing signature: • It is possible to show that L4 does not miss more attacks than L1 (under certain assumptions) L1L2L3L4 More false positives Less false positives Rubin, Jha, Miller

34. Constructing a Signature for pro-ftpd TCP Streams login=1 TYPE=‘A’ Rubin, Jha, Miller

35. Constructing a Signature for pro-ftpd TCP Streams login=1 TYPE=‘A’ FN1 Two signatures based on the configuration of the FTP server Rubin, Jha, Miller

36. Lessons to Take Home TCP Streams • A methodology to construct and evaluate signatures • Able to detect loopholes in signatures, loopholes that we did not anticipate • The accuracy of the signature depends of the accuracy of the semantic model Ainv Ainv  fn  fp Session Signature Aghost  sp Rubin, Jha, Miller