Identification of Bot Commands By Run-time Execution Monitoring

1. Identification of Bot Commands By Run-time Execution Monitoring Younghee Park, Douglas S. Reeves North Carolina State University ACSAC 2009

2. OUTLINE • INTRODUCTION • THE PROPOSED METHOD • EXPERIMENTAL EVALUATION • DISCUSSION • CONCLUSION

3. OUTLINE • INTRODUCTION • THE PROPOSED METHOD • EXPERIMENTAL EVALUATION • DISCUSSION • CONCLUSION

4. About Botnets • A major source of network threats • DDoS, spam, identity theft, click frauds • A variety of protocols • IRC, HTTP, peer-to-peer • Botnets is estimated to be in the millions of hosts

5. BotTee • Monitoring and analyzing bot execution to identify the bot commands that are being executed. • Bot commands with the same purpose that is highly correlated, across all types of bots. • Bot commands can be accurately identified during execution.

6. OUTLINE • INTRODUCTION • THE PROPOSED METHOD • EXPERIMENTAL EVALUATION • DISCUSSION • CONCLUSION

7. System architecture for BotTee

8. Bot behavior classification through bot commands

9. Hooking API calls • These bots invoke Windows functions through the API provided to applications. • When each API call is intercepted, the time is also recorded. • To hook only a limited set of Windows API calls. • Approximately 300 commonly-used API functions from 50 real bot instances. • 153 APIs were in file kernel32.dll; the rest were found in user32.dll, advapi32.dll, ws2_32.dll (Wsock32.dll), etc.

10. Bot Command Identifier • What sequence of system calls may correspond to a bot command? recv and send • Repeated consecutive occurrences of the same API call in a trace are eliminated. • γ = 2 • AAABCCAAAADDDA → AABCCAADDA • Semantic unit ‘synflood’ • socket, TLSGetValue, InterlockedDecrement, ioctlsocket, connect, WaitForSingleObject, etc.

11. Correlation Engine • This engine is used to create command templates, and to match captured system call traces to these templates. • Longest common subsequence algorithm (LCS) , and statistical correlation • Define θ1 as P(ρi,j > δ) | H1)

12. Common API Call Trace • The CACTs for each command include important APIs for identifying the execution of the bot command. • These are termed the featured APIs. • CACT of ‘dns’ with the length 30. • recv, TlsGetValue, GetLocalTime, GetUserDefaultLCID, WideCharToMultiByte, GetTimeFormatA, GetConsoleMode, WriteConsoleA, WriteFile, inet_addr, ..., GetTickCount, InterlockedExchange, CloseHandle, gethostbynam, inet_ntoa, send,

13. A Real-time Semantic Behavior Matcher • Semantic unit is compared to all of the templates of bot commands. • A candidate template must be identified. • Computing the correlation of Semantic unit’s timing vector with each timing vector in the template. • Additional information can be recorded about the arguments of API calls that are hooked.

14. OUTLINE • INTRODUCTION • THE PROPOSED METHOD • EXPERIMENTAL EVALUATION • DISCUSSION • CONCLUSION

15. Implementation and Experiments • Prototype of BotTee • Used the Deviare API for intercepting Windows API calls on the fly. • A botnet in a private network was deployed. • Among 167 available bot source codes, there were 103 variants • Agobot, Spybot, Sdbot, and Jrbot

16. Performance Overhead of Hooking

17. Correlation Results

18. Identification of Specific Bot Commands

19. False Identification • If CACTs are not distinctive enough to differentiate bots from non-bot programs.

20. Detection Rate with API Call Injection Attack • Injection for obfuscation purposes may be intended to obfuscate timing analysis and correlation as well.

21. OUTLINE • INTRODUCTION • THE PROPOSED METHOD • EXPERIMENTAL EVALUATION • DISCUSSION • CONCLUSION

22. DISCUSSION • The more accurately that botnet-driven network threats can be identified. • BotTee can specify victims targeted by active botnets and infer the overall behaviors of the active botnets. • The hooking technique allows potentially malicious bot commands to be replaced by more benign actions, or to be thwarted.

23. OUTLINE • INTRODUCTION • THE PROPOSED METHOD • EXPERIMENTAL EVALUATION • DISCUSSION • CONCLUSION

24. CONCLUSION • A method for identifying the high-level commands being executed by a bot, in real time. • Comparison of the resulting traces with a previously-captured set of bot command templates. • This held true even for commands executed by bots from other bot families.