CSC 382/582: Computer Security

1. CSC 382/582: Computer Security Integrity Management CSC 382/582: Computer Security

2. Topics • Host Integrity • Anti-virus Software • Malware self-protection techniques. • Malware detection techniques. • Personal Firewalls • Host Intrusion Detection Systems • Host Intrusion Prevention Systems CSC 382/582: Computer Security

3. C Host Integrity I A Ensuring a host adheres to a security policy. • Misuse Detection and Prevention • Know a set of attacks against a host. • Attempt to detect and prevent these attacks. • Anti-virus, anti-spyware, personal firewall. • Policy-based Tools • Security policy describes good state of system. • Attempt to detect deviations from good state. • Host Intrusion Detection Systems (HIDS). CSC 382/582: Computer Security

4. Misuse Detection • Anti-virus tools most widely used host misuse detection software. • Signature database primary method of detecting known attacks on host. • Most anti-virus tools also offer • Intrusion Prevention: detect virus during d/l. • Recovery: eradicate detected viruses. • Anti-virus software is often deployed on network proxy servers in addition to hosts. CSC 382/582: Computer Security

5. Malware Self-Protection Anti-debugging Detect/disable debuggers when used to analyze code. Attack anti-malware tools Disable anti-malware tools upon infection. Kill processes or destroy/modify signatures. API checksums Avoid having UNIX/Win32 API calls in code. Store checksums of API names and search for match. Code obfuscation Use unusual tricks and unused code to avoid dissassembly and prevent quick analysis of purpose. Self-modifying code. CSC 382/582: Computer Security

6. Self-Protection Compression Code looks almost random; size is smaller. Use unusual executable packers to avoid analysis. Data encryption Encrypt strings, hostnames, IP addresses to avoid detection. Embedding Embed infection in one format inside a document inside an archive file. Scanners have to understand and have time to parse and decompress each file format. CSC 382/582: Computer Security

7. Self-Protection Entry-Point Obscuring Changing initial code or entry point easy to notice. Alter program code to gain control randomly. Host morphing Alter host file during infection to prevent removal. CSC 382/582: Computer Security

8. Self-Protection: Encryption Encrypt all code except small decryptor. • Note that copy protected files will have similar decryptors to prevent analysis too. • Often uses multiple decryptors. • Change encryption key dynamically. Random Decryption Algorithm (RDA) • Choose random key for encryption. • Brute force search for key to decrypt. • Slows VMs/debuggers used for analysis. CSC 382/582: Computer Security

9. Self-Protection: Polymorphism Alter malware code with each infection. • Cannot be detected by signature scanning. • May alter decryptor only or entire code. • Insert junk instructions that do nothing. • Fragment and rearrange order of code. • Alternate sets of instructions for the same task. • Ex: SUB -1 instead of ADD 1 • Randomize names in macro viruses. CSC 382/582: Computer Security

10. Case Study: Zmist EPO, encrypted, polymorphic virus. Code integration Decompiles PE files to smallest elements. Inserts virus randomly into existing code. Rebuilds executable. Polymorphic decryptor Inserted as random fragments linked by JMPs. Randomizes self with ETG engine. CSC 382/582: Computer Security

11. Virus Detection Signature-based • Look for known patterns in malicious code. • Defeated by polymorphic viruses. Smart scanning • Skips junk instructions inserted by poly engines. • Skips whitespace/case changes in macro viruses. Decryption • Brute-forces simple XOR-based encryption. • Checks decrypted text against small virus sig to decide whether has plaintext or not. CSC 382/582: Computer Security

12. Virus Detection Code Emulation • Execute potential malware on VM. • Scan VM memory after certain # iterations. • Watch instructions for decryptor profile. Code Optimization. • Optimize away junk instructions and odd techniques used by polymorphic viruses. CSC 382/582: Computer Security

13. Virus Detection Heuristics • Code execution starts in last section. • Suspicious code redirection. • Suspicious section ACLs or size. • Suspicious library routine imports. • Hard-coded pointers into OS kernel. Neural Network Heuristics • IBM researchers trained neural net to recognize difficult polymorphic viruses. • Released in Symantec antivirus. CSC 382/582: Computer Security

14. Limits of Malware Detection • Assume you have a perfect malware detector D(p) that takes a program p as input and returns True or False. • Create a program P(q) that incorporates your malware detector D: if D(q): Do nothing else Become malware • What would D report if given P to analyze? CSC 382/582: Computer Security

15. Personal Firewall • Firewall configured to protect single host. • Used on servers, desktops, and laptops. • Why use a personal firewall? • Configuration can closely match single host’s needs without considering entire network. • Can protect on a per-application basis. • Can protect mobile hosts when outside the organization’s network firewall. CSC 382/582: Computer Security

16. Host Intrusion Detection Systems Monitors host state for signs of intrusion: • Files • File metadata changes (access time, perms) • File data changes (checksums) • Configuration • Log file entries • User accounts and groups • Runtime • Logins and logouts • Running processes • Open network connections • Kernel modules and status CSC 382/582: Computer Security

17. Why HIDS? • October 25, 2000 • Microsoft detects passwords being mailed out of company to e-mail address in Russia. • Electronic logs show source code downloaded. • How did it start? • Employee received e-mail carrying Qaz trojan. • Qaz copied itself to Notepad.exe, moved Notepad to Note.com. • Qaz spread itself across network, d/led tools. • Attacker used tools to acquire passwords. CSC 382/582: Computer Security

18. Advantages of HIDS • HIDS can associate data with specific users, while NIDS cannot do this. • HIDS has access to data that’s sent to the host in a network encrypted form (SSL, VPN.) • HIDS is immune to NIDS evasion techniques. NIDS doesn’t know how host stack interprets packets, but HIDS looks at data after stack has processed it. CSC 382/582: Computer Security

19. HIDS Types • File Integrity Checkers • Tripwire, AIDE, anti-virus software • Log Watchers • logwatch, swatch • Network monitors • portsentry, BlackICE • Host Integrity Monitors • Osiris, Samhain CSC 382/582: Computer Security

20. File Integrity Checkers • Perform baseline scan of filesystem. • Metadata: ownership, permissions, times • Cryptographic checksums of contents • Periodically scan filesystem • Compare current state to baseline state. • Notify admin if changes discovered. CSC 382/582: Computer Security

21. File Integrity Checkers Configuration required to avoid false +’s • Ignore temporary files in /tmp and elsewhere. • Ignore log file checksums, but permissions and ownerships are important and size should not decrease. • Update checksums when binaries or libraries updated. CSC 382/582: Computer Security

22. HIDS Architecture A HIDS consits of three components: • Agentruns on host and gathers data. • Director periodically polls hostsand aggregates data • Sends requests to agents to scan hosts. • Receives data from agents. • Decides on whether to act on data. • Notifier acts on director results. • May simply notify security officer. • May reconfigure agents or director. • May activate response mechanism. CSC 382/582: Computer Security

23. Agent • Scans host and transmits data. • Attackers target agent to avoid detection • Kill agent process. • Replace agent with their own code. • Install rootkit. • Protection against subversion • Self integrity check. • Privilege separation. • Encrypted communication with director. • Configuration data and baselines stored on director. • Scan data transmitted to director host only. CSC 382/582: Computer Security

24. Director Director Functionality • Polls agents to request scan data. • Sends configuration data to agents. • Receives scan data from agents. • Stores scans in database. • Compares scans with baselines. • Updates baselines if Administrator requests. Protecting the Director • Use a dedicated host with limited access. • Run an agent on the director host to monitor itself. CSC 382/582: Computer Security

25. Example HIDS: Tripwire CSC 382/582: Computer Security

26. Running a HIDS • When should I collect baselines? • Ideally before the host is connected to network. • If later, knowngoods.org maintains checksums of common Linux distributions. • How often should I poll hosts? • Depends on performance goals. • How often should I monitor alerts? • Serious alerts should automatically notify you. • Read logs daily. CSC 382/582: Computer Security

27. Host Intrusion Prevention Systems • Attacks usually cause programs to initiate new behaviors: • New network sockets opened. • New files modified, etc. • Create a system call model of a program. • Compiler extracts call graph model of program. • HIPS monitors program during execution, checking to see if execution matches model. CSC 382/582: Computer Security

28. Example: System Call Models CSC 382/582: Computer Security

29. Key Points • Integrity Management Approaches • Misuse detection and prevention. • Policy-based management. • Detection Issues • Problems: encryption, polymorphism • Solutions: compiler-based, VM simulations • Host Integrity Management Systems • System Types: HIDS, HIPS • Monitor Files, Configuration, Runtime data. • HIDS provides local view with greater detail than NIDS. CSC 382/582: Computer Security

30. References • Matt Bishop, Computer Security: Art and Science, Addison-Wesley, 2003. • Ted Bridis and Rebecca Brucman, “Microsoft hacked! Code stolen?”, http://news.zdnet.com/2100-9595_22-525083.html, October 26, 2000. • Simson Garfinkel, Gene Spafford, and Alan Schartz, Practical UNIX and Internet Security, 3/e, O’Reilly & Associates, 2003. • Lap Chung Lam, Wei Li, and Tzi-cker Chiueh, “Accurate and Automated System Call Policy-Based Intrusion Prevention,” Proceedings of the International Conference on Dependable Systems and Networks (DSN'06), 2006. • Stephen Northcutt, Lenny Zeltser, Scott Winters, Karen Kent and Ronald Ritchey, Inside Network Perimeter Security, Second Edition,Sams Publishing, 2005. • Ed Skoudis and Lenny Zeltser, Malware: Fighting Malicious Code, Prentice Hall, 2003. • Ed Skoudis, Counter Hack Reloaded 2/e, Prentice Hall, 2006. • Peter Szor, The Art of Computer Virus Research and Defense, Addison-Wesley, 2005. • Brian Wotring, Host Integrity Monitoring Using Osiris and Samhain, Syngress, 2005. CSC 382/582: Computer Security