Security Monitoring Thru Log Analysis

1. Security Monitoring Thru Log Analysis Sam.NG@PISA

2. Once upon a time An email server of my company was HACKED! A user “tmp” with uid 0 and no password was created

3. What the hacker did • Startup script was modified, a line is added to the rc.local • ./usr/bin/t0x5mm (I can’t remember the exact name) • “ls” the file, nothing showed • I tried to remove the file by “rm”, “rm” ran without any error • Many months later, I knew that’s something call “rootkit”

4. Log was gone! • Some of the log files are truncated • I wanted to know how the hacker got in, and I redirected syslog to “/dev/lp0” and ran for a couple of weeks • Several inches of paper printed, but there is more than I could handle • I can’t remember how, but finally I knew the hacker got in by exploiting an IMAP4 vulnerability

5. Lesson Learnt • You will need the log in sometime • And better still, store it in a safe place • Write once read many (WORM) is good, but searching capability is even more important • And the most important: you have to prepare it beforehand

6. But how can I use the log for security monitoring?

7. The Challenge “I don’t have the staff to handle 140,000 alerts. I don’t even have the staff to handle 12,000 alerts” David MacLeod, Ph.D., CISSPThe Regence Group CISO

8. Reference: Counterpane’s Security Monitoring Service Bruce Schneier

9. Block Diagram Phase I Phase III Phase IV Phase II Log Collection Noise Reduction Data Mining Alert/Ticket Management

10. Alert/Ticket Management Noise Reduction Log Collectors Data Mining

11. Log Collectors • Send your log to DATABASE • Can be direct (open database connection from the client) or in-direct (e.g. post data to a web application) • Each different log may need a different log collector • Many logs have built-in support to store data in database

12. Unix syslog • syslog can be redirected • Can be redirected to printer! (“/dev/lp0” in Linux) • Can be redirected to remote syslog host • But redirect to program (pipe) seems not working!!?? • syslog-ng support logging directly to database • Win32 solutions to accept remote syslog records and log to database • Plain text format, easy to write a program to watch the data and then send to database

13. Demo • A smiple “tail” base perl script to monitor new records in syslog and send it to remote database

14. Windows Eventlog • Microsoft logparser can log to database • WMI interface allows you to query remote eventlog • Many third party solutions support logging to database

15. Demo • A smiple C# program to monitor new records in Eventlog and send it to remote database

16. Quiz 1 • Windows NT • Windows 2000 • Windows 2003 • None of the above Which of the following Windows Server will pass Microsoft Baseline Analyzer (MSBA) 2.0 auditing requirements with default install?

17. DO YOU KNOW? • You can pass MSBA 2.0 if you enabled Success and Failure auditing for the Account Logon Events • But indeed, it recommends you to enable Success and Failure auditing: Account Logon Events, Account Management, Policy Change, System EventsFailure auditing: Directory Service Access, Object Access • Data volume may be quite high, a server config in this way may generate a eventlog for every 2 seconds (actual volume depends on your server)

18. Microsoft IIS Web Server Native support log to ODBC (but seems to be only available on server platform only)

19. Microsoft IIS Web Server (2) • Microsoft does not recommend logging to database if the IIS is busy (Q245243) • But nowadays most web pages are generated with SQL queries • Test ODBC logging on your own environment • PrepWebLog (Q296093) convert lIS log to SQL insertion statements in plain text format, but still have to run these SQL insertion by some other means • Still, the best would be realtime logging to database • Writing a “tail -f” similar program in IIS is difficult

20. DO YOU KOWN? • IIS log file is updated in 64k chunks. On servers that do not have a high usage rate, the statistics will not be up-to-date because of the delayed write (Q142557) • When it flush, may be only first 20k of the chunk contains data, the rest 44k data is ‘\0’ and will be filled in next flush • Obviously a mmapped file, may be due to performance consideration… but

21. Suggestions to programmer • In C/C++, STDERR, cerr are not buffered • In fact, error log should be send out immediately, should not be cached/buffered # man stderr …… …… CONSIDERATIONS The stream stderr is unbuffered. The stream stdout is line- buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a newline is printed. This can produce unexpected results, especially with debugging output. …… ……

22. J2EE Application Server • Most application server support log4j or java.util.logging (JDK 1.4+) • log4j natively support logging to database thru JDBCAppender

23. My Experience • I have a program developed since JDK1.3 • At that time, Java don’t have java.util.logging • And I don’t know much about log4j • I wrote my own log handling routine (similar to syslog) to insert my own records into database • But is not configurable/adaptable, and is not consolidated with the Application Server’s log • Since logging is widely spread all over the codes, it is very difficult for me to change the program to use these new technology • Log architecture should be planned in DESIGN PHASE

24. Snort IDS • Comes with database output plugin to send alerts to database • The packet analysis thread is also responsible for database insertion

25. My Experience • I have experience in using Snort to monitor a ~20M Internet link, with database output plugin, default rules • CPU is just about 30%, seems good  • When I changed to log to local file, CPU sharp jump to 100% • Barnyard seems to be a solution but it does not support database!!

26. And many others • Checkpoint FW1: thru LEA enabled product http://www.opsec.com/ • Microsoft ISA: default support ODBC logging (KB838710) • Microsoft Exchange: same as Eventlog • Apache httpd: mod_log_sql

27. Noise Reduction • Very important • Noise will kill the system • And to improve overall performance • Can be done on several layers • ignore certain message at the log collection sender • ignore certain message at the log collection receiver end • delete (mark as ignore) at the database • fine tune your IDS rule, firewall logging policy etc.. • exception cases for data mining phase

28. Data Mining “Data processing using sophisticated data search capabilities and statistical algorithms to discover patterns and correlations in large preexisting databases; a way to discover new meaning in data” From http://www.tfd.com

29. Data Mining (2) • The heart of Security Monitoring • A board term, a general concept • Utilize database queries to get the information you want • Can be an external program, can be a SQL server schedule job

30. Data Mining Techniques

31. Attack Definition GET /..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /MSADC/root.exe?/c+dir GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir GET /_vti_bin/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\ GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir GET /_vti_bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /_vti_bin/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /_vti_cnf/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /adsamples/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /cgi-bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir+c:/ GET /cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:/ GET /cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /cgi-bin/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:/ GET /iisadmpwd/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /iisadmpwd/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /msadc/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\ GET /msadc/..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /msdac/root.exe?/c+dir+c:\ GET /scripts/..%e0%80%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /scripts/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:\ IIS Unicode Directory Traversal (cmd.exe) attack strings

32. Vulnerability characteristic ≠attack characteristic • CodeRed I [/default.idq?NNNNN……] • CodeRed II [/default.idq?XXXXX……] • In fact, the vulnerability can be exploited if the variable name is around 240 bytes • Regular Expression [\/default\.idq\?.{240,}] • Difficult to write an effective and accurate definition • And not all attack leave audit trail

33. Event Correlation • E.g. 1000 login failures followed by ONE successful login from the same IP • E.g.IF http_response_code = 500; THEN find_all_other_url_accessed();ENDIF

34. Anomaly Detection But what is NORMAL??

35. Normal Definition • You define what is normal and then monitor it • E.g. Operators login time should be corresponding to their shift duty • E.g. All server services should not be restarted unless necessary (ignore service start within 3minutes of system startup)

36. Statistical Analysis • E.g. on average a event occurs 10 time a day, with a standard deviation of 2.3. But today we have 2000 records. • E.g. anything happens more than 200 times in the past 30 minutes • E.g. a event never seen in the past 7 days • E.g. “TOP 10” events/users/hosts, etc…

37. Alert/Ticket Management • Works like Bug Tracker • Save the alert as a “Ticket” • A ticket is a something like an outstanding job • Assign the ticket to a staff to follow up • Escalate it if remain unresolved for some time • And don’t forget People Management(Time and Skill)

38. Alert/Ticket Management (2) • According to my experience, at least 90% of the alerts generated by data mining are still FALSE ALARMS • People will get use to it and tend to think EVERY alerts are false alarms • If possible, fine tune the system to eliminate the false alarm from occurring again Noise Can Kill

39. Couterpane 2003 Results

40. DO YOU KNOW? • TSL providing Email to pager service for about HK$80/mth

41. How should I start? • Do it step by step, phase by phase, EventLog, syslog are easy to start with • A group brain storming section would give you at least 10 such data mining rules, and is a good starting point • Security Monitoring is a (long term) process, do not regard it as a single one-shot install and forget project

42. Q & A Contact: samngms@yahoo.com