1 / 15

Security event processing with SEC (Simple Event Correlator)

Security event processing with SEC (Simple Event Correlator). Risto Vaarandi Estonian Defence Forces. Outline. Event correlation – what it is and why do we need it? Simple Event Correlator (SEC) – motivation, features and performance. Sample event correlation scenarios.

asher-mcgee
Download Presentation

Security event processing with SEC (Simple Event Correlator)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Security event processing with SEC (Simple Event Correlator) Risto Vaarandi Estonian Defence Forces

  2. Outline • Event correlation – what it is and why do we need it? • Simple Event Correlator (SEC) – motivation, features and performance. • Sample event correlation scenarios. • SEC application experience and further reading.

  3. Event correlation – what it is? • Commonly used formal definition (Jakobson and Weissman, 1995) – event correlation is an interpretation procedure where a new meaning is assigned to a set of events that occur within a predefined time interval. • A simple example – if a network link down event is not followed by network link up within 5 seconds, emit an output event network link broken; otherwise create an internal event network link short outage. If more than 10 network link short outage events have been observed within 3 hours, emit an output event network link quality degradation. • Please note that event correlation is a real-time (or near-real-time) event processing technique!

  4. Event correlation – why do we need it? • Event correlation produces events that are more meaningful to a human. • Event correlation can significantly reduce the number of events that are presented to the human. These considerations are important, because in today’s IT systems millions of events can occur in every hour – no hope for a human to follow the events and react to them in a timely manner!

  5. Example - centralized event management infrastructure Central log server Application event correlator event receiver Notifications to the monitoring console Server events Router network Event logs Applications, servers, and network devices send their events to the central log server. Events are stored and correlated on the central log server, and output alerts from the event correlator are sent to the central monitoring console.

  6. Simple Event Correlator (SEC) – motivation • Existing event correlation products are mostly heavyweight and expensive solutions: • many products can only be deployed as a client-server system with a requirement for a dedicated high-end event correlation server, • some products have been designed for specific tasks only (like network fault management), • several products depend on other complex and expensive software packages (e.g., Oracle), • there are products that can’t be deployed without the assistance from vendor consultants! • there exist only a few free and open-source event monitoring tools that have some support for event correlation. • SEC implements a completely different event correlation paradigm – lightweight and platform independent event correlation with an open source tool

  7. SEC – design and features • Employs rule-based approach for event correlation – events are correlated according to rules written by a human analyst. • Lightweight design – the size of SEC is < 300KB and < 9000 lines, runs as a single process. • Modest resource requirements – e.g., with 10,000+ event correlation operations and contexts active, < 20MB memory is consumed. • Platform independence – SEC is written in Perl, thus runs on almost any UNIX flavor and Windows (with CygWin Perl). • No dependencies on other software (Perl only). • Reads input events from one or more textual event logs (plugins can be written for binary formats). • Input events are matched with regular expressions, Perl subroutines, string patterns, and truth values. • Produces output by executing custom command lines (e.g., sends an SNMP trap or an e-mail), writing to files and FIFOs, etc. • Licensing – GNU GPLv2 (free and open-source software).

  8. Snort IDS alert consolidation I # The following ruleset processes Snort IDS syslog messages: # Apr 4 10:10:55 snorthost.mydomain [auth.alert] snort[18800]: # [1:2528:14] SMTP PCT Client_Hello overflow attempt [Classification: # Attempted Administrator Privilege Gain] [Priority: 1]: # {TCP} 192.168.5.43:28813 -> 192.168.250.44:25 # # If a Snort priority 1 alert has been seen for a certain source IP address, create # a context for this IP, in order to consolidate all events for the same source IP; # also, send a short note to the local admin that an attack has begun type=Single ptype=RegExp pattern=snort\[\d+\]: \[[\d:]+\].*\[Priority: 1\]: \S+ ([\d\.]+):?\d* -> [\d\.]+:?\d* context=!ATTACK_FROM_$1 continue=TakeNext desc=Priority 1 attack started from $1 action=create ATTACK_FROM_$1; \ pipe '%s' mail -s 'Snort: priority 1 attack from $1 (alert)' root

  9. Snort IDS alert consolidation II # If a Snort priority 1 alert is observed for a certain source IP address, then: # 1) store it to the context for that IP (the context has been created by the previous rule) # 2) extend the context lifetime for 5 minutes. # # If the context lifetime ends (in other words, there have been no more events from # the given source IP), send all events from the context event store to the local admin # as a single mail. type=Single ptype=RegExp pattern=snort\[\d+\]: \[[\d:]+\].*\[Priority: 1\]: \S+ ([\d\.]+):?\d* -> [\d\.]+:?\d* context=ATTACK_FROM_$1 desc=Priority 1 incident from $1 action=add ATTACK_FROM_$1 $0; \ set ATTACK_FROM_$1 300 ( report ATTACK_FROM_$1 \ mail -s 'Snort: priority 1 attack from $1 (report)' root )

  10. User account probe detection I # Process sshd PAM authentication messages from Solaris system log, in order to detect repeated # probing of different user accounts: # Apr 3 14:20:19 myhost sshd[25888]: [ID 800047 auth.error] error: # PAM: Authentication failed for risto from myhost2 # Apr 3 14:20:23 myhost sshd[25888]: [ID 800047 auth.info] Accepted # keyboard-interactive/pam for risto from 192.168.27.69 port 9729 ssh2 # # If a user fails to login with SSH and this is not followed by a successful login within 30 seconds, # generate an event LOGIN_FAILED_FOR_username type=PairWithWindow ptype=RegExp pattern=sshd\[\d+\]: \[ID \d+ auth\.error\] error: PAM: Authentication failed for (\S+) from \S+ desc=PAM authentication failed for $1 action=event LOGIN_FAILED_FOR_$1 ptype2=RegExp pattern2=sshd\[\d+\]: \[ID \d+ auth\.info\] Accepted keyboard-interactive/pam for ($1)\ from \S+ port \d+ ssh2 desc2=PAM authentication successful for $1 action2=logonly window=30

  11. User account probe detection II # Count LOGIN_FAILED_FOR_username events from the previous rule – if 10 distinct usernames # have been seen within 10 minutes, alert the local admin; then switch to silent mode for 1 hour type=SingleWithThreshold ptype=RegExp pattern=LOGIN_FAILED_FOR_(\S+) context=!USER_$1_ALREADY_COUNTED && !COUNTING_OFF continue=TakeNext desc=Ten login failures for distinct users have been observed action=pipe '%s' mail -s 'PAM alert' root; create COUNTING_OFF 3600 window=600 thresh=10 # Set up a context for the username after it has been counted, in order to prevent repeated counting type=Single ptype=RegExp pattern=LOGIN_FAILED_FOR_(\S+) context=!USER_$1_ALREADY_COUNTED && !COUNTING_OFF desc=Set up the "count once" context for user $1 action=create USER_$1_ALREADY_COUNTED 600

  12. User account probe detection III PairWithWindow Feb 18 12:10:23 PAM: authentication failed for risto Feb 18 12:10:33 Accepted keyboard-interactive/pam for risto Feb 18 12:15:11 PAM: authentication failed for john Feb 18 12:15:56 ntpd: ….. Feb 18 12:16:49 PAM: authentication failed for john doesn’t produce a match anymore at 12:17:19 logonly: PAM authentication successful for risto (at 12:10:33) create: USER_john_ALREADY_COUNTED Single event: LOGIN_FAILED_FOR_john LOGIN_FAILED_FOR_john SingleWithThreshold 10 LOGIN_FAILED events for distinct users within 10m neither USER_john_ALREADY_COUNTED nor COUNTING_OFF context exists at 12:15:41, so start the counting operation with the window of 10 minutes and set event counter to 1 pipe: send mail alert create: COUNTING_OFF (for getting 1 mail per 1 hour)

  13. SEC – performance • SEC’s event processing speed depends on a number of factors – the size and nature of the rule base, the nature of input data, and the underlying hardware. • Experiment1: Linux workstation with a 3.0GHz P4 CPU, one rule with a regular expression pattern, 1% of input lines were matching and written to a file by the rule – event processing speed was 17,000+ events per second. • Experiment2: Linux server with a 3.2GHz P4 CPU, 100+ rules with complex regular expression patterns and non-trivial output actions (at least 2-3 actions executed per second), banking card events with a complex structure as input – event processing speed was 3,000 events per second.

  14. SEC – experience • Has been developed by Risto Vaarandi for 8+ years. • Employed by many institutions – large North-American telecom companies, government and financial institutions, universities, medical and car industry, etc. • Has been used for security management, network management, system monitoring, etc. • Has been employed for correlating events from many sources – Snort IDS, HP OpenView, CiscoWorks, BMC Patrol, Nagios, syslogd/syslog-ng, various routers and firewalls, modems and other network equipment, etc. • Has been packaged for major Linux and BSD UNIX distributions – RedHat, Fedora, Debian, Ubuntu, Gentoo, OpenBSD, FreeBSD. • SEC mailing list has 300+ members.

  15. SEC – references • SEC home page: http://simple-evcorr.sourceforge.net. • John P. Rouillard, University of Massatchusetts at Boston, “Real-time Logfile Analysis Using the Simple Event Correlator (SEC)”, USENIX 2004 System Administration Conference (see also http://www.cs.umb.edu/~rouilj/sec/). • Jeff Becklehimer and Cathy Willis, Cray Inc. and Josh Lothian, Don Maxwell, and David Vasil, Oak Ridge National Laboratory (ORNL), “Real Time Health Monitoring of the Cray XT Series Using the Simple Event Correlator (SEC)“, 2007 Cray User Group Conference (see alsohttp://www.nccs.gov/wp-content/uploads/2007/08/becklehimer_paper1.pdf). • James Turnbull, “Hardening Linux” (Chapter 5), Apress 2005 (see also http://www.apress.com/book/view/1590594444). • Risto Vaarandi, “Simple Event Correlator for real-time security log monitoring”, Hakin9 Magazine 1/2006 (6) (see also http://en.hakin9.org/attachments/pdf/hakin9_05_2006_10_EN_str28-39.pdf).

More Related