Malware Detection Based on Malicious Behaviors Using Artificial Neural Network

1. Malware Detection Based on Malicious Behaviors Using Artificial Neural Network Student: Hsun-Yi Tsai Advisor: Dr. Kuo-Chen Wang 2012/05/28

2. Outline • Introduction • Problem Statement • Related Work • Design Approach • Sandboxes • Behaviors • Proposed Algorithm • Weight Training • Malicious Degree • Evaluation • Conclusion and Future Works • References

3. Introduction • In recent years, malware has been severe threats to the cyber security • Virus, Worms, Trojan horse, Botnet … • Traditional signature-based malware detection algorithms [15] [17] • Drawbacks of signature-based malware detection algorithms • Need human and time to approve • Need to update the malicious digest frequently • Easily bypassed by obfuscation methods • Can not detect zero day malware • Increase false negative rate

4. Introduction (Cont.) • To conquer the shortcomings of the signature-based malware detection algorithms, behavior-based malware detection algorithms were proposed • Behavior-based malware detection algorithms [14] [19] • Detect the unknown malware or the variations of known malware • Decrease false negative rate (FNR) • Increase false positive rate (FPR) • To decrease the FPR, we proposed a behavioral neural network-based malware detection algorithm

5. Problem Statement • Given • Several sandboxes • l known malware Mi= {M1,M2, …, Ml} for training • mknown malware Sj= {S1, S2, …, Sm} for testing • Objective • n behaviors Bk= {B1,B2, …, Bn} • n weights Wk= {W1,W2, …, Wn} • MD (Malicious degree)

6. Related Work • MBF [14] • File, process, network, and registry actions • 16 malicious behavior feature (MBF) • Three malicious degree: high, warning, and low • RADUX [19] • Reverse Analysis for Detecting Unsafe eXecution (RADUX) • Collected 9 common malicious behaviors • Bayes’ theorem

7. Related Work (Cont.)

8. Background - Sandboxes Dynamic analysis system Isolated environment Interact with malware Record runtime behaviors

9. Background - Sandboxes (Cont.) • Web-basedsandboxes • GFI Sandbox [1] • Norman Sandbox [2] • Anubis Sandbox [3] • PC-based sandboxes • Avast Sandbox [4] • Buster Sandbox Analyzer [5]

10. Design Approach-Behaviors • Malware Host Behaviors • Creates Mutex • Creates Hidden File • Starts EXE in System • Checks for Debugger • Starts EXE in Documents • Windows/Run Registry Key Set • Hooks Keyboard • Modifies Files in System • Deletes Original Sample • More than 5 Processes • Opens Physical Memory • Deletes Files in System • Auto Start • Malware Network Behaviors • Makes Network Connections • DNS Query • HTTP Connection • File Download

11. Design Approach-Behaviors (Cont.)

12. Design Approach-Behaviors (Cont.) Ulrich Bayer et al. [10]

13. Design Approach-Proposed Algorithm

14. Design Approach – Weight Training Using Artificial Neural Network (ANN) to train weights

15. Design Approach – Weight Training (Cont.) • Neuron for ANN hidden layer

16. Design Approach – Weight Training (Cont.) • Neuron for ANN output layer

17. Design Approach – Weight Training (Cont.) d: expected target value Mean square error: Weight set: : learning factor; x: input value , Delta learning process

18. Design Approach-Malicious Degree • Malicious Degree • Malicious behaviors: • Weights: • Bias: • Transfer function:

19. Evaluation MD Threshold Benign Ambiguous Malicious Try to find the optimal MD value to make FPR and FNR approximate to 0.

20. Evaluation (Cont.) Matlab 7.11.0 Initial weights and bias: random by function initnw Transfer function: tangent-sigmoid function Architecture of ANN (Matlab7.11.0):

21. Evaluation (Cont.) Malicious sample source: Blast’s Security [6] and VX Heaven [7] websites Benign sample source: Portable execution files under windows XP SP2 Training data and testing data

22. Evaluation (Cont.) Range of threshold Mean square error: 0.19 Execution time: 2 seconds MD threshold (according to training data)

23. Evaluation (Cont.) Choose threshold

24. Evaluation (Cont.) Experiment results

25. Evaluation (Cont.)

26. Evaluation (Cont.)

27. Conclusion and Future Work • Conclusion • Collect several common behaviors of malwares • Compose Malicious Degree (MD) formula • The false positive rate and false negative rate is approximated to 0 • Detect unknown malware • Future work • Automate the system • Implement PC-based sandboxes • Add more malware network behaviors • Classify malwares according to their typical behaviors

28. References [1] GFI Sandbox. http://www.gfi.com/malware-analysis-tool [2] Norman Sandbox. http://www.norman.com/security_center/security_tools [3] Anubis Sandbox. http://anubis.iseclab.org/ [4] AvastSandbox. http://www.avast.com/zh-cn/index [5] Buster Sandbox Analyzer (BSA). http://bsa.isoftware.nl/ [6] Blast's Security. http://www.sacour.cn [7] VX heaven. http://vx.netlux.org/vl.php [8] Neural Network Toolbox. http://dali.feld.cvut.cz/ucebna/matlab/toolbox/nnet/initnw.html [9] “A malware tool chain: active collection, detection, and analysis,” NBL, National Chiao Tung University. [10] U. Bayer, I. Habibi, D. Balzarotti, E. Krida, and C. Kruege, “A view on current malware behaviors,” Proceedings of the 2nd USENIX Workshop on Large-Scale Exploits and Emergent Threats : botnets, spyware, worms, and more, pp. 1 - 11, Apr. 22-24, 2009. [11] U. Bayer, C. Kruegel, and E. Kirda, “TTAnalyze: a tool for analyzing malware,” Proceedings of 15th European Institute for Computer Antivirus Research, Apr. 2006. [12] M. Egele, C. Kruegel, E. Kirda, H. Yin, and D. Song, “Dynamic spyware analysis,” Proceedings of USENIX Annual Technical Conference, pp. 233 - 246, Jun. 2007. [13] H. J. Li, C. W. Tien, C. W. Tien, C. H. Lin, H. M. Lee, and A. B. Jeng, "AOS: An optimized sandbox method used in behavior-based malware detection," Proceedings of Machine Learning and Cybernetics (ICMLC), Vol. 1, pp. 404-409, Jul. 10-13, 2011.

29. References (Cont.) [14] W. Liu, P. Ren, K. Liu, and H. X. Duan, “Behavior-based malware analysis and detection,” Proceedings of Complexity and Data Mining (IWCDM), pp. 39 - 42, Sep. 24-28, 2011. [15] C. Mihai and J. Somesh, “Static analysis of executables to detect malicious patterns,” Proceedings of the 12th conference on USENIX Security Symposium, Vol. 12, pp. 169 - 186, Dec. 10-12, 2006. [16] A. Moser, C. Kruegel, and E. Kirda, “Exploring multiple execution paths for malware analysis,” Proceedings of 2007 IEEE Symposium on Security and Privacy, pp. 231 - 245, May 20-23, 2007. [17] J. Rabek, R. Khazan, S. Lewandowskia, and R. Cunningham, “Detection of injected, dynamically generated, and ob-fuscated malicious code,” Proceedings of the 2003 ACM workshop on Rapid malcode, pp. 76 - 82, Oct. 27-30, 2003. [18] K. Rieck, T. Holz, C. Willems, P. Dussel, and P. Laskov, “Learning and Classification of Malware Behavior,” in Detection of Intrusions and Malware, and Vulnerability Assessment, Vol. 5137, pp. 108-125, Oct. 9, 2008. [19] C. Wang, J. Pang, R. Zhao, W. Fu, and X. Liu, “Malware detection based on suspicious behavior identification,” Proceedings of Education Technology and Computer Science, Vol. 2, pp. 198 - 202, Mar. 7-8, 2009. [20] C. Willems, T. Holz, and F. Freiling. “Toward automated dynamic malware analysis using CWSandbox,” IEEE Security and Privacy, Vol. 5, No. 2, pp. 32 - 39, May. 20-23, 2007. [21] Y. Zhang, J. Pang, R. Zhao, and Z. Guo,"Artificial neural network for decision of software maliciousness," Proceedings of Intelligent Computing and Intelligent Systems (ICIS), Vol. 2, pp. 622 - 625, Oct. 29-31, 2010.