Effective and efficient malware detection at the end host

1. or: How I Learned to Stop Worrying and Love the Malware-infested Internet C. Kolbitsch, P. M. Comparetti, C. Kreugel, E. Kirda, X. Zhou and X. Wang Effective and efficient malware detection at the end host Presentation by Clark Wachsmuth

2. THE PROBLEM • Malware! • Ineffective (and/or inefficient) detection models • Can be evaded by fairly simple means by malware authors, such as using polymorphism, obfuscation or system call reordering • Resource-heavy detectors may be effective, but not efficient enough for average consumer computer

3. PAST IMPLEMENTATIONS • Network-based detection • Pros: • Useful for detecting some network malware • Modern malware is heavily network-bound • Cons: • For network-based malware only • Content sniffers thwarted by encrypted data • Blending attacks make malicious data match normal data signatures

4. PAST IMPLEMENTATIONS • Host-based detection • Pros: • Has the resources to see complete work of malware programs – not limited to specific host resource • Some pre-emptive strategies • Cons: • Code obfuscation and polymorphism can easily bypass methods such as byte signatures while keeping the same functionality • System call sequence based detection, again, easily bypassed by reordering calls or making unused calls

5. PAST IMPLEMENTATIONS • Static analysis-based detection • Pros: • More effective due to focus on malware behavior, thus less stymied by obfuscation and polymorphism • Cons: • Method itself is difficult to employ • Has its own vulnerabilities such as detecting metamorphic code and runtime packaging • Takes a heavy toll on system resources making it unreasonable for home computer systems

6. PAST IMPLEMENTATIONS • Dynamic analysis-based detection • Pros: • Less rigid focus on malware behavior allowing for a more general and broad way of detecting malware • Cons: • Can require special hardware for detection (data tainting) • Large associated overhead making it unusable in the home computer realm

7. THE SOLUTION • Effective and Efficient malware detection, duh! • But how? • Effective: • Can’t be duped by simple order-changing, rearranging schemes • Doesn’t rely only on known quantities; can detect unknown running programs • No false positives • Efficient: • Not incurring a very significant chunk of system resource overhead

8. THE PLAN • In a sandboxed environment, observe different malware and develop fine-grained models • Efficiently match these models up with the run-time behavior of an unknown program • If a match is found, terminate and eliminate

9. BUT HOW?! • By creating a behavior graph where each node is an “interesting” system call • The nodes store a symbolic expression (simple node) or a program “slice” (complex node) that can calculate the output of the system call • These expressions/slices used to detect if output is the argument of another interesting system call during runtime • If found, an edge is created between the two nodes

10. THE CONTROLLED ENVIRONMENT • Uses Anubis (Analyzing Unknown Binaries) • Disassembles instructions (including system calls) and keeps an instruction log • Keeps memory log for instructions that read from memory, where (in memory) the instruction reads and writes • Each bite tainted to detect data dependencies between system calls • Any labels within a branch operation are labeled with the taint of the controlling instruction for control dependencies

11. THE INITIAL BEHAVIOR GRAPH • With all the instructions labeled, an initial graph is creating placing on it all system calls (as nodes) • Edges are created when a dependency is found • Using the logs, a recursive backwards trace of system call arguments is made to determine how the argument’s bits were created • These instructions are gathered into a program slice until either an instruction that can’t be traced further (from the outside) or a value produced by an immediate operand from an instruction or coming from the initialized data segment

12. PROGRAM SLICES  FUNCTIONS • With the slice, we know how and who created the argument of the sys call • It’s not necessarily the direct program code, though (unrolled loops won’t match with different sizes) • Each line in binary that appears at least once in slice is marked and appropriate code copied to function. Non-marked lines become nops. • Stack needs fixing because stack creating code often not part of slice (uses instruction log)

13. SIMPLIFYING FUNCTIONS • Yay! We have a function that gives an expected output for a given input • Some functions can be quite long and fairly basic • We can optimize it to a smaller symbolic expression • This optimization can have huge overhead reduction at the end host • Other functions aren’t so basic, so we retain the program code of the function rather than have a symbolic reduction

14. SCANNING END HOST • Scanner monitors running program for sys calls • Has admin privileges running is user-mode • Assume programs can’t get to kernel • All nodes inactive in initial behavior graph • When a system call is made, the scanner checks graph for inactive nodes of the same type and sees if parent nodes are active • If found, checks all arguments from sys calls for simple functions; defers complex functions for later but allows complex function to hold • If all simple function arguments hold, node becomes active

15. SCANNING END HOST • When do we check the complex functions? • When we reach an interesting node • Interesting if it is a security-relevant system call (writing to file system, network or registry, starting new processes) • Also interesting if node has no outgoing edges • If complex function holds, the interesting node is confirmed • Otherwise, the node with the complex function becomes inactive and any subgraph rooted under it becomes inactive as well as the edge being formed

16. MATCHING MALWARE • If an interesting node is confirmed, then the program is matched as malware • However, if there is no complex function dependency, then the graph created is not used to help detect future malware programs • The subgraph created with the interesting node is also a behavior graph that denotes a trait of the particular malware running

17. DETECTION EFFECTIVENESS • Generated behavior graphs for six popular malware families (Table 1) • 100 samples of each family were Selected from the database and the non-interesting samples were tossed out • 50 random samples chosen from remaining bunch to create behavior graphs and train dataset • Not all samples could be detected due to non-interesting behavior and complex function crashes

18. TESTING DATA

19. IS IT EFFECTIVE? • Some were effective and some weren’t so much • AV software notoriously bad at classifying malware • Confirmed by manual inspection, especially for Agent • Restricting samples to 155 known variants yielded 92% effectiveness • Also restricted data samples to 108 unknown variants and still achieved 23% effectiveness, indicating that this method can even detect some unknown variants • This behavior-based method is more general than an AV scanner, therefore requires less graphs than signatures

20. WHAT ABOUT FALSE POSITIVES? • Tested on WinXP using IE, Firefox, Thunderbird, putty and Notepad • Yielded no false positives • When complex functions were unchecked and allowed to hold, all of the above yielded false positives • Therefore, system call dependencies are at the root of this method’s success

21. OK, BUT IS IT EFFICIENT? • System setup for testing: • WinXP, single-core 1.8Ghz P4 with 1GB RAM • Tested using 7-Zip, IE, Visual Studio

22. UMM, DID THAT SAY 40%? • CPU / I/O-Bound tests showed low overhead • Compiling seems quite high at 40% • System calls in compiling 5000/sec compared to 7-zip’s 700/sec • Compilation is worst-case scenario • Improved symbolic execution engine could possibly reduce high complex function evaluation of 16.7% • Still performed well for common tasks

23. LIMITATIONS • Authors could use time-triggered behavior or command and control mechanisms to prevent malware behavior during test • A reactive method that only works on running malware • But, new graphs can be employed quickly and it can detect some unknown variants • Authors could change algorithms rendering program slices unusable • Changing algorithms is a lot of work and this method still raises the bar considerably higher for malware authors

24. TECHNICAL CONTRIBUTIONS • Developed effective models with detailed semantic information about the malware family • Created a scanner that efficiently matches the behavior of an unknown, running program against the models by tracking system call dependencies • Experimental evidence that approach is feasible and usable in practice

25. CONCLUSION • Effective? Check • With correctly labeled, known variants, a 92% effectiveness was obtained with no false positives • Efficient? Check • While compiling was a worst-case scenario, tasks common to the average end user incurred only a low overhead