CECR (“Seeker”) - Centralized Event Correlation and Response

1. CECR (“Seeker”) - Centralized Event Correlation and Response Ramon Kagan, Chris Russel York University, Toronto

2. Agenda • How and why Automated Incident Response enhances an Information Security program • Initial Phase: Detection and Compliance systems • Implementation of Centralized Event Correlation and Response (CECR) system • Detection and Compliance • Correlation and Classification • Automated Response

3. Context: York University • Located in Toronto • Canada’s third largest University • 60,000+ students • 10,000+ staff and faculty • 25,000+ network drops • In 2003, 2 FT information security staff positions (now 3)

4. “Traditional” Incident Response • Preparation • Identification • Containment • Eradication • Recovery • Lessons Learned (From SANS Step-by-Step Incident Response)

5. August 2003

6. Short-Circuited Incident Response • Information Security becomes “Worm Management Services” • No time for normal response procedures • We adapted and made it through, but… • Is this really security? • What are we missing in the noise and mayhem of constant worm attacks?

7. Prevention • Don’t let these things happen in the first place • Lots of products to buy • Firewalls, IPS, Anti-Virus, Silver Bullets, etc. • These are all good things but not without their challenges

8. Prevention Challenges in the Academic Environment • Porous and increasingly fuzzy perimeter • Dialup, Wireless, VPN, Mobile devices, etc.Where does the firewall go now? • Political hurdles to implement controls • I want my dancing pigs! • Increase in operational management overhead • $$$++

9. Detection and Response are Essential Too • Why? • Prevention measures require increasing amounts of money and strong policy, diminishing returns • They cannot prevent everything • What if they fail? • How useful is a bank vault without an alarm and police response? • Ultimately it can only buy time

10. Automated Detection and Response • Improving detection and response speed • Makes best use of and complements existing prevention measures • Better ROI than additional prevention? • Allows a 24/7/265 response absent staff • Frees up incident handlers to focus on less obvious/potentially more serious matters

11. Where Automated Detection and Response Matter • BotNets • compromised host waits for commands • Detect it first and take it out before it spreads behind your perimeter • Spyware (Marketscore, etc) • Leveraged/Low and Slow Hacking • Automated correlation can help detect things otherwise below the radar • Large virus/worm infestation • Can scale to greatly assist with a future large-scale event

12. Detection • Gather as much information as possible from anywhere you can • Syslog • Flow logs • IDS/IPS/Firewall logs • Honeypots

13. Syslog • Login attempts • Port scans • Local exploits • Anti-virus alerts

14. Flow logs • Network traffic patterns • Scanning detection • Anomaly detection • Historical context and forensic information

15. IDS/IPS/Firewall Logs • Scanning • Invalid access • Hacking attempts

16. Honeypots • Great for internal detection • No need for expensive hardware • much cheaper than gigabit (multi-gig?) IDS sensors at every router • By definition, very few false positives • Darknets or Honeynets

17. Compliance • Agent-based compliance detection • Network-side vulnerability scanning • Nessus or other commercial tools • NOXscan: FAST scanner for Microsoft vulnerabilities used by many worms (MS04-007, MS04-011, MS05-039) http://infosec.yorku.ca/tools/

18. Correlation and Reaction • Map events to an IP or MAC • Map IP or MAC to user, support group, network drop, etc. • Initiate a response as appropriate to the incident type, severity and context • Do this very fast! • Enter CECR… large drop in incidents within 3 months after implementation

19. Implementation

20. Lots of info, so what • All this great information being gathered • How to sift through it • How to react to it • How to record our actions

21. Manual Handling • Manual correlation • Manually enter each incident (ELOG) • Basic reporting available • Very time consuming to enter all the tickets

22. Manual Handling • Needed to increase correlation speed • Needed better reporting • Needed a way to distinguish incident types more easily • Needed a tool that portrayed a process • Needed a way to enter incidents automatically

23. Impetus for Change • In a single word - LAZINESS • September 2004 - Outbreak of virus activity on our dialup network • Two problems • Mapping users to IP/Mapping IP to network segment - time consuming • Entering all those tickets - even more time consuming and oh the pain

24. CECR v1.0 • Shell script designed to accomplish two menial tasks • Correlate incidents to users • Submit tickets to RTIR automagically • Great first step for dealing with mass breakouts • Allowed for initial automation of specific triggers

25. CECR v1.0 • Limitations • Not abstracted and difficult to manipulate for extension • Haphazard script to ease the pain • Wasn’t really designed for more central usage • Unable to effectively take actions based on incident severity

26. CECR v2.0 • Rewritten in Perl • Designed for extension and real-time updating • Able to conduct many more tasks • Different actions depending on severity • Plugins can be added at any time • Exclusions now possible • Repeat notification removed - limited to once daily • Automated contact to end-users/support groups

27. Framework of CECR v2.0 Sensors Reporting Process Central Processor Correlation Plugins Logging and Ticket Creation Action Plugins Automated Notification

28. Components of CECR v2.0 • Reporting Process • Wrapping scripts around some IDS sources • Argus not “tail-able” • Vulnerability scanner results • Logsurfer+ for real-time processing of others • Pix log trends - context cognition • snort

29. Components of CECR v2.0 • Central Process • Perl script - the coordinator • Param: incident type, IP, time, port (optional) • Two configuration files • Actions - what action to take per incident Incident type:Action:RTIR Category:Reason Tag:Email file:Exclusion List • Contacts - whom to contact for non-user access Regex domain:email:RTIR support group

30. Components of CECR v2.0 • Correlation plugins • 6 plugins • Correlate IP (depending on connection): • Username • MAC • Port • TTY (dialup)

31. Components of CECR v2.0 • Action Plugins • 5 plugins • Conduct various tasks including • Account lockout • Deregistration • Disconnection from network • Quarantine

32. Components of CECR v2.0 • Automated notification • Template based emails by incident type • Contact either username (LDAP verified) or contact listed in contacts file • Notification sent to infosec group of incident • In event of no contact information, infosec email states as such

33. Components of CECR v2.0 • Logging & Ticket Creation • All actions and decisions are logged via syslog for audit purposes • E-mail notification to RTIR to automagically create tickets in appropriate queues • Time based record of event maintained to limit repeat notification

34. RTIR • CERT sponsored add-on to RT from Best Practical - opensource with support availability • Queues helped define process • Manual insertion still required, but contributions existed for e-mail ticket creation - the light!!

35. CECR v2.0 • Net Results • Extendable framework for ever changing landscape • Force multiplier allowing handlers to worry about more significant events • 24x7x365 monitoring of known issues • Automated tracking of events - allows for statistics

36. Questions?