NETWORK TRAFFIC ANALYSIS OPTIMIZATION FOR SIGNATURE-BASED INTRUSION DETECTION SYSTEMS

1. NETWORK TRAFFIC ANALYSIS OPTIMIZATION FOR SIGNATURE-BASED INTRUSION DETECTION SYSTEMS Dmitry S. Kazachkin student, Computational systems lab at CMC MSU Dennis Y. Gamayunov scientific advisor, PhD, Computational systems lab at CMC MSU

2. INTRODUCTION • High-performance traffic analysis at gigabit channels • Optimization of a given signature set matching • Packet – the unit of analysis • Operations number minimization for a single packet

3. SIGNATURE FORMAL DEFINITION (1) • Packet – pair {H, P} • Header (H) – {H1,H2,...,Hn}, Hi belongs to finite space • Payload (P) – text line of unrestricted length • State vector (C) – {С1,С2,...,Сk}, Ci belongs to finite space

4. SIGNATURE FORMAL DEFINITION (2) • Header condition – logical predicate; Hi – arguments • Payload condition – set of functions: • CondPi(P,C) — logical predicate; payload and state vector are arguments • Effectsi = {Effecti,j(P,C)} — represents side effect on variable Сi, performed on CondPi(P,C) success • Reaction – element of event classes finite set

5. SIGNATURE FORMAL DEFINITION (3) • Signature – triplet <SH,SP,R>, where • SH – set of header conditions • SP – ordered set of payload conditions • R – reaction. • RES(<SH,SP,R>,H,P,C): • {R}, if all the header conditions return ‘true’ and all the payload conditions successfully evaluate consecutive. • Æ, else

6. SNORT SIGNATURESSTRUCTURE • Action – action performed on rule activation (usually, ‘alert’) • Header – context-independent conditions on packet header • Options – context-dependent conditions on packet payload • Info – rule info and message, generated at rule activation

7. SNORT SIGNATURES FORMALIZATION (1) • Header • H1 — protocol type (TCP / UDP / ICMP / other IP) • H2 and H3 — IP and port of packet source • H4 — packet direction (from server/to server) • H5 and H6 — IP and port of packet destination • Payload – packet payload byte string • State vector – C = {C1}, C1 – position of text marker

8. SNORT SIGNATURES FORMALIZATION (2) • CondHk(H) – logical expression, operating {Hi}only • <CondPk(P,C), Effectk(P,C)> – pair of functions representing pattern matching performed after marker • Reaction–finite set of identifiers, which represent action and info sections

9. R-LANG LANGUAGEFORMAL SIDE • Automaton - (K, PS, T, PT, s0, I, g q) • S – states set; • PS – state predicates set; • T – transitions set; • PT – transition conditions set; • s0 – initial state; • I – automaton exemplars set; • g – global environment; • q – global timer queue • automaton exemplar – series of slices NKx L x Q x E • NK – current state; • L – local environment; • Q – local timer queue; • E – local events queue

10. R-LANG LANGUAGEDECLARATIVE SIDE • Scenario • Events subscription • Variables declaration – scenario fields • Functions declaration – scenario methods • States declaration • One is marked as initial • Transitions declaration • Transition type – consuming/nonconsuming/unwinding • Condition – logical function • Body – code block

11. SIGNATURE CONVERSIONTO R-LANG (1) S.T. Eckmann, "Translating Snort rules to STATL scenarios“ • Typified events on every packet • SrcIP, SrcPort, Direction, DstIP, DstPort • Payload • Scenario field variable Current • Header conditions – logical expressions on {Hi} • Payload conditions – boolean function calls, with side-effect on Current • Reactions– alert sending code blocks

12. SIGNATURE CONVERSIONTO R-LANG (2) scenario sc(<event corresponding to packet type> ev)‏ { <Secondary variables definition> initial state st0; consuming transition st0->st0 event <event corresponding to packet type> ( <Header condition 1> && … && <Header condition N> ) { <Restoring state vector> if(<Payload condition 1>)‏ … if(<Payload condition M>)‏ <Reaction>; } };

13. HEADER-BASED OPTIMIZATION (1) • Conditions alternative — a set of pairs <SPi,Ri>, where • SPi – ordered set of payload conditions • Ri — reaction • Alternative-containing signature – a pair <SH, SA>, where • SH — set header conditions • SA — condition alternative. • Set of signatures with the same header are converted to alternative-containing signature this way: • Alternative-containing signature evaluation result is defined as:

14. HEADER-BASED OPTIMIZATION (2) • R-lang transition body of the alternative-containing signature model: <Restoring state vector> if(<Series of payload conditions 1>)‏ <Reaction 1>; <Restoring state vector> if(<Series of payload conditions 2>)‏ <Reaction 2>; … <Restoring state vector> if(<Series of payload conditions N>)‏ <Reaction N>; • We get the same RES set with less operation by grouping signatures with the same header conditions set. • 519 different header condition sets for Snort base of 6372 signatures grants a good performance speed-up

15. HEADER-BASED OPTIMIZATION (3) Tree-style order provides even more economy. consuming transition st0->st0 event <event corresponding to packet type>(true){ if(<Header condition 1>) { if(<Header condition 2>) <predicate alternative for header granted by 1,2> if(<Header condition 3>) if(<Header condition 4>) <predicate alternative for header granted by 1,3,4> } } Snort analysis engine use that header optimization only, that does not allow achieving further speed-up on a fixed signature set.

16. PREDICATE TREE • Fact: Condition chains contained in conditions alternative can have the same beginnings. • Predicate tree ST – tree: • the edges are marked with a payload condition • the nodes are marked with a reaction set, possibly empty • the leafs are marked with non-empty reaction sets

17. PREDICATE TREE EVALUATION • Predicate tree evaluation • recursive traversal of tree from root node • conditions control sub-trees evaluation • reaction performing if current node is marked • context restoration after sub-tree evaluation • RES(ST, P,C) – {all the reaction achieved} • Statement: Let predicate tree ST is converted from condition alternative SA. Then RES(ST,P,C)=RES(SA,P,C).

18. ADDITIONAL OPTIMIZATIONS Synonymic bunch – set of leafs: • the same {R} • the same edges parent • PCRE union • Aho-Corasick algorithm

19. STATIC RESULT ANALYSIS • Tree profit = edges number in the tree - total number of payload conditions • Relational tree profit = tree profit / total number of payload conditions • For 2 signature groups (Samba): • ~2000 signatures • tree profit > 11000 • relational tree profit > 80% • Overall relational tree profit ~62%. • 40 pairs of identical signatures

20. DYNAMIC RESULT ANALYSIS • N seconds of flood with legal packets, a bad packet after them • Test 1 – legal traffic is ignored by most rules • FTP “HELP” command • Test 2 – legal traffic is checked by lots of rules • specially constructed HTTP request

21. CONCLUSION • Proposed methods provides significant speed-up of packet analysis at experimental IDS. • Though speed-up is significant, it is stillnot enough. There is a need for other optimizations, e.g. parallel signature matching.

22. REFERENCES • Snort IDS, http://www.snort.org • S.T. Eckmann, "Translating Snort rules to STATL scenarios" presented at the 4th International Symposium on Recent Advances in Intrusion Detection (RAID 2001), Davis, CA, October 2001, LNCS 2212, pp. 69-84. • M. Norton, "Optimizing Pattern Matching for Intrusion Detection," white paper, Sourcefire Inc., 2004 [Online] Avaliable: http:// docs.idsresearch.org/OptimizingPatternMatching/ForIDS.pdf. • D.U. Gamayunov, “Network objects behavior analysis for detecting computer attacks” PhD thesis, Faculty of Computational Math and Cybernetics, Moscow State University, Moscow, 2007. • S.T. Eckmann, G. Vigna, and R. A. Kemmerer. “STATL: An Attack Language for State-based Intrusion Detection” Dept. of Computer Science, University of California, Santa Barbara, 2000. • G. Vigna, R. Kemmerer, "NetSTAT: A Network-based Intrusion Detection Approach." Proceedings of the 14th Annual Computer Security Application Conference, Scottsdale, Arizona, December 1998. • M. Roesch. "Writing Snort Rules: How To write Snort rules and keep your sanity" [Online] Avaliable: http://www.snort.org.

23. Thank you for attention