Network Security Monitoring SearchSecurity webcast: 4 Dec 02

1. Network Security MonitoringSearchSecurity.com webcast: 4 Dec 02 Richard Bejtlich, Foundstone richard.bejtlich@foundstone.com Robert Visscher, Ball Corporation rvissche@ball.com

2. Detection – Overview • This module will address: • What is detection? • Why perform it? • How is it accomplished? • When and where should detection occur? • Who performs it? • Recommendations

3. Detection – What is detection? • Detection means identifying intrusions • Detection is best implemented through network security monitoring (NSM) • NSM is the collection, analysis, and escalation of indications and warnings (I&W) to detect and respond to intrusions • NSM is an “industry best practice” not implemented by most enterprises • This material is seldom taught elsewhere

4. Detection – What is detection? • Military indications and warning (I&W) examples: • Training/exercise • Construction • Force deployment • Vehicle convoy formation • Ordinance stockpiling • Communications • Movement of leadership • Foreign assistance • Imagery: http://globalsecurity.org

5. Detection – What is detection? • NSM relies upon: • Products, because human brains have trouble interpreting raw network traffic and memory registers • People, because machines cannot assess intent and computers lack real-world situational awareness • Processes, because unvalidated and unactionable I&W is worthless • “IDS” is only a product; NSM is an operation incorporating products, people, and processes

6. Detection – Why perform it? • You should want to detect and respond to an intrusion to mitigate damage to your: • Finances • Intellectual property • Reputation • Computing resources • Individual liberty – if you go to jail for breaking the law!

7. Detection – Why perform it? • Detection always occurs! • A customer reports being charged for goods he never ordered from your e-commerce business • A competitor notifies you that your computers are launching attacks against his network • A user complains her computer is slow and odd background processes are running • These real-world examples show someone else detecting intrusions before you do! • Detecting an intrusion using in-house or outsourced resources is more proactive than waiting for bad news from customers, competitors, or users

8. Detection – Why perform it? • Beyond identifying intrusions, NSM: • Provides digital forensic data to support post-compromise law enforcement actions • Accelerates response and recovery actions • Identifies host and network misconfigurations • Improves management and customer understanding of the Internet's inherent hostility • Validates acceptable use policies and access control lists • May be required by law or best industry practices – now or in the near future

9. Detection – How is it accomplished? • If NSM is the collection, analysis, and escalation of indications and warnings (I&W) to detect and respond to intrusions, then: Products collect and generate I&W Peopleanalyze and validate I&W Processesescalate validated I&W and shape response actions

10. Detection – How is it accomplished? • Detection starts with interpreting activities • All network traffic and computer processes fall in one of three categories: • Normal - Web surfing, FTP sessions, sending email • Abnormal but not malicious - Odd protocol manipulation by peer-to-peer clients, load balancing by Web servers, proprietary applications • Malicious - Recon, intrusions, worms, viruses • Properly classifying activities is difficult, but the result is identifying intrusions

11. Detection – How is it accomplished? • To help assess activity, products collect and generate I&W • Two intrusion detection system (IDS) types: • Network-based IDS (NIDS) monitors network traffic for signs of misuse • Host-based IDS (HIDS) monitors computer processes for signs of misuse • So-called "hybrid" systems may do both, but for a single host

12. Detection – How is it accomplished? • Ideally, products generate three types of I&W data: • Event – a summary of an observed activity; an “alert” • Session – a summary of conversations seen by NIDS • Full Content – complete collection of information related to one or more activities • In practice, most only generate event data • Vendors (and many customers) don’t appreciate session data • Full content data is expensive to collect and store

13. Detection – How is it accomplished? • Is collecting this data legal? We are not lawyers, but... • 18 U.S.C. 2511(2)(a)(i) offers the Provider Protection Exception • Interception is allowed “while engaged in any activity which is a necessary incident totherendition of service or the protection of the rights or property of the provider of the service” • Ref: http://www.cybercrime.gov/usc2511.htm • Consent Exception, implemented through banners, gives more explicit legal cover for full collection

14. Detection – How is it accomplished? • Sample NIDSevent data: Snort alert on a Microsoft SQL Server attack using stored procedures [**] [1:687:3] MS-SQL xp_cmdshell - program execution [**] [Classification: Attempted User Privilege Gain] [Priority: 1] 04/02-12:46:58.109453 172.16.86.36:3544 -> 192.168.46.111:1433 TCP TTL:107 TOS:0x0 ID:18073 IpLen:20 DgmLen:182 DF ***AP*** Seq: 0x5D4A696 Ack: 0x7ACAAC20 Win: 0x3F10 TcpLen: 20

15. Detection – How is it accomplished? • Sample HIDSevent data: Microsoft IIS web server logs on Unicode directory traversal: #Software: Microsoft Internet Information Services 5.1 #Version: 1.0 #Date: 2002-09-19 20:34:38 #Fields: time c-ip cs-method cs-uri-stem sc-status 20:36:16 127.0.0.1 GET /msadc/..%5c../..%5c../..%5c../winnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%5c..%5c..%5c..%5cwinnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%5c../..%5c../..%5c../winnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%5c..%5c..%5c..%5cwinnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%5c../..%5c../..%5c../winnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%2c..%2c..%2c..%2cwinnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%2f..%2f..%2f..%2fwinnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%5c..%5c..%5c..%5cwinnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%5c..%5c..%5c..%5cwinnt/system32/cmd.exe 404 20:36:16 127.0.0.1 GET /msadc/..%5c..%5cwinnt/system32/cmd.exe 404

16. Detection – How is it accomplished? • Comments on event data: • All vendors try to reduce “false positive” event data, but the term is a misnomer • There is no such thing as a false positive! • All events are I&W and represent computing activities • Every single packet on a network, and process on a computer, tells the intrusion detector something about the state of those resources • “Blinking red lights” cannot confirm intrusions because machines cannot assess intent and computers lack real-world situational awareness

17. Detection – How is it accomplished? • Sample session data: proprietary code summarizing multiple TCP connections Time | Source IP | Port | Destination | Port | SP | SB | DP | DB ---------+----------------+------+----------------+------+-----+--------+-----+-------- 12:46:57 | 172.16.86.36 | 3544 | 192.168.46.111 | 1433 | 9 | 654 | 8 | 6648 12:46:58 | 192.168.46.111 | 2267 | 172.173.86.248 | 21 | 24 | 1144 | 22 | 3433 12:47:00 | 172.173.86.248 | 20 | 192.168.46.111 | 2268 | 7 | 2047 | 4 | 164 12:47:01 | 172.173.86.248 | 20 | 192.168.46.111 | 2269 | 365 | 511444 | 242 | 9684 12:47:11 | 172.173.86.248 | 20 | 192.168.46.111 | 2271 | 17 | 18608 | 11 | 444 12:47:13 | 172.16.86.36 | 3550 | 192.168.46.111 | 1433 | 5 | 438 | 4 | 611 NOTE: SP is Source Packets; SB is Source Bytes; DP is Dest Packets; DB is Dest Bytes

18. Detection – How is it accomplished? • Commentson session data: • Session data is based on military signals intelligence collection practices • Session data can be generated even when encryption foils collection of event and full content data • It is always useful to know to whom and for how long your systems communicate • Generating session data is much less intrusive than full content collection • Hardly any vendors produce session data • Generally a NIDS (not HIDS) concept

19. Detection – How is it accomplished? • Sample full content data: tethereal showing packet contents of Microsoft SQL Server attack using stored procedures 0000 01 01 00 8e 00 00 01 00 45 00 58 00 45 00 43 00 ........E.X.E.C. 0010 20 00 6d 00 61 00 73 00 74 00 65 00 72 00 2e 00 .m.a.s.t.e.r... 0020 2e 00 78 00 70 00 5f 00 63 00 6d 00 64 00 73 00 ..x.p._.c.m.d.s. 0030 68 00 65 00 6c 00 6c 00 20 00 22 00 66 00 74 00 h.e.l.l. .”.f.t. 0040 70 00 2e 00 65 00 78 00 65 00 20 00 2d 00 76 00 p...e.x.e. .-.v. 0050 20 00 2d 00 6e 00 20 00 2d 00 73 00 3a 00 5c 00 .-.n. .-.s.:.\. 0060 66 00 74 00 70 00 2e 00 74 00 78 00 74 00 20 00 f.t.p...t.x.t. . 0070 31 00 37 00 32 00 2e 00 31 00 37 00 33 00 2e 00 1.7.2...1.7.3... 0080 38 00 36 00 2e 00 32 00 34 00 38 00 22 00 8.6...2.4.8.”. • Note: ‘tethereal’ is the text-based version of ethereal

20. Detection – How is it accomplished? • Comments on full content data: • Full content shows exactly what happens on systems, as long as the IDS collects both sides of the conversation • Ease of interpretation of NIDS full content data varies • Trivial for text-based protocols like telnet • Moderately difficult for inter-process and file-sharing communications (NFS, RPC) • Very difficult for graphical protocols (X) • HIDS full content data is analogous to contents of /proc on UNIX systems, but rarely collected

21. Detection – How is it accomplished? • Network-based intrusion detection pros: • Highest return on investment, as one sensor can potentially monitor dozens to hundreds of targets • Recognize attacks upon infrastructure and provides a larger field-of-view • Network-based intrusion detection cons: • Encryption may degrade network visibility • IDS and target system may handle packets differently (http://online.securityfocus.com/data/library/ids.ps describes ‘insertion’ and ‘evasion’ attacks)

22. Detection – How is it accomplished? • Example NIDS: Snort (snort.org) • Key benefits: • Signatures can be rapidly updated by admins • Provides supporting event and full content data needed to verify the significance of alerts • Large user community tracks intrusions world-wide • Major weakness: • Snort is a detection engine, not an enterprise suite • Proper use may require administrator knowledge exceeding that needed for commercial products

23. Detection – How is it accomplished? • Host-based intrusion detection pros: • Offers greater ability to understand processes on hosts, including success or failure of attacks • A single event log can effectively replace interpretation of hundreds of network packets • Host-based intrusion detection cons: • Difficult to manage more than a few systems • Host owners blame HIDS for problems

24. Detection – How is it accomplished? • Example HIDS: • Tripwire (file integrity verification) • System security event logs • Application event logs • Dedicated host-based agents by commercial vendors • All help, but... • Customers and admins for box platform with HIDS tend to blame HIDS agents for any problems • Some HIDS are little more than event log readers

25. Detection – How is it accomplished? • Remote event logging is highly recommended • After enabling logging on your devices, forward copies of event log entries to a secure log host • Configure logger to accept messages from selected machines only, and dedicate the log host to logging • Syslog can be used, and syslog generators exist for Windows architectures • http://ntsyslog.sourceforge.net/ • http://www.eventreporter.com/en/ • http://www.kiwisyslog.com/ • http://www.winsyslog.com/en/

26. Detection – How is it accomplished? • The cardinal rule of all intrusion detection: • You must have enough supporting data to verify the impact of I&W. If not, why bother monitoring? • Every time you must physically inspect a target to verify the impact of an alert, response time, cost, and effectiveness are an order of magnitude worse Confused? Call Foundstone’s IRT!

27. Detection – How is it accomplished? • Questions to ask • What could cause the activity in question? • What could the IDS have missed? • How does reality differ from textbooks? • Would the community benefit from sharing? • SecurityFocus Incidents list (http://online.securityfocus.com/archive/75) offers forums for exchanging IDS "sightings" “The Bible”... for the ’90s only?

28. Detection - When and where should detection occur? • Network-based intrusion detection tools: • Router logs • Firewall logs • Dedicated sensors (appliances or PCs) • All should be used, just as defense in depth requires routers to screen some traffic and firewalls to screen other traffic

29. Detection - When and where should detection occur? • Where should one place the NIDS? • Some say outside the firewall is attack detection • Some say inside the firewall is intrusion detection • Some say both inside and outside is needed • In an environment where a single office or individual is responsible for all security, and owns the routers, firewalls, and NIDS, inside the firewall is acceptable • Otherwise, outside is preferred

30. Detection - When and where should detection occur? • Here, a NIDS interface sits before the router and firewall and another in the DMZ • HIDS sits on critical servers • Who watches the wireless segment?

31. Detection – Who performs it? • Timeliness of detection is the issue • How quickly do you want to be able to contain and recovery from intrusion? • How much abuse are you willing to sustain before your reputation and assets are destroyed? • Subtle, expert compromise is difficult to detect • The longer one waits the greater the damage

32. Detection – Who performs it? • Four options • Aperiodic in-house monitoring • Periodic in-house monitoring • Continuous (24X7) in-house monitoring • Continuous (24X7) outsourced monitoring • Your choice is usually based on: • Budgetary constraints • Appreciation of threats • Quality and quantity of technical staff

33. Detection – Who performs it? • Aperiodic in-house monitoring: react when notified by informal means • Advantages • Low cost – system/network admins responsible • “Ignorance is bliss” • Disadvantages • Most likely to be victimized and remain so • Response requires forensic consulting on victim hosts • High probability of long-term, systematic compromise

34. Detection – Who performs it? • Periodic in-house monitoring: regularly consult logs and IDS (if any) • Advantages • Moderate cost – 1+ security admins responsible • May strike best balance for small enterprises • Disadvantages • Friday PM intrusions not noticed until Monday AM • Data may not be of sufficient fidelity to aid response • Narrow field of view causes network “tunnel vision”

35. Detection – Who performs it? • Continuous (24X7) in-house monitoring: you perform NSM or best approximation • Advantages • Fastest response time mitigates impact of intrusion • Highest fidelity data reduces need for host forensics • Proactive NSM can prevent some intrusions • Disadvantages • Requires dedicated equipment and trained personnel • High cost usually only justified at global enterprises

36. Detection – Who performs it? • Continuous (24X7) outsourced monitoring: vendor performs NSM or best approximation • Advantages • Like in-house, plus low cost from economies of scale • Wider field of view and higher analyst expertise • Disadvantages • Hardly any vendors understand NSM principles; most perform “device management” • Most vendors have poor validation capabilities and rely on collecting “syslog” messages from devices

37. Detection – Who performs it? • Free global I&W info: Internet Storm Center • Grew from SANS Y2K Global Incidents Analysis Center (GIAC – the original meaning of the acronym) • Useful for observing trends and corroborating I&W collected locally • http://isc.incidents.org

38. Detection – Who performs it? • Free global I&W info: CERT/CC Current Scanning Activity • Tends to be more static than ISC • Convenient links to CERT/CC advisories • http://www.cert.org/ current/scanning.html

39. Detection – Who performs it? • Free global I&W info: defacement mirrors • Mirrors sharing OS and service info for victims gives clues to hacks-du-jour • ‘nmap’ feature shows ports open on victims • defaced.alldas.org

40. Detection - Recommendations • Product issues: • Coordinate product requirements with analysts (the people) and decision makers (the processes) • Balance the product’s capability to: • Update rules – frequency, reliability, timeliness • Manage multiple platforms • Detection strategy – signature- or anomaly-based? • Scale beyond initial deployment plans • IDS wire monitoring speed is not the primary issue! • Choose sensors to collect the quantity and quality of data to needed to assess the impact of an event

41. Detection - Recommendations • People issues: • Training and reading are absolutely essential • Bejtlich’s reading lists: http://taosecurity.com/books.html • Analysts typically deployed in tiered infrastructure • Tier one: entry level to 12/18 months experience; 2+/shift • Tier two: 12/18 months to 3 years experience; 1/shift • Tier three: 3 years+ experience; 1+/operation • Entry level analysts (tier one) screen alerts and forward what they can’t handle to higher tiers • NSM operation only as good as the tier three on duty

42. Detection - Conclusion • Process issues: • Analysts must take responsibility for the events they interpret • Accountability allows managers to separate will problems from skill problems • Escalation procedures require knowing who to contact when an intrusion is found and how to do it • Analysts must have clear guidance on how to proceed when intrusions are discovered • Pursue and monitor • Remediate and recover