INTELLIGENT INTRUSION DETECTION RESEARCH

1. INTELLIGENT INTRUSION DETECTION RESEARCH ATTACKING A HIGH PERFORMANCE COMPUTER CLUSTER by Miguel Torres, Rayford B. Vaughn, German Florez, Zhen Liu, Susan M. Bridges <mt98,vaughn,gf24,zliu,bridges>@cs.msstate.edu May 2003

2. Outline • Introduction • IIDS • Daemon Attacks • Interposition Library Attacks • Device Driver Attack • Conclusions

3. Introduction • Linux High Performance Clusters • Supports the research in the intrusion detection area which captures function or system calls to detect anomalous behavior • The implemented attacks will be presented here

4. Why This Work Is Needed… • Creates anomalous data sets for testing our IIDS AI techniques • Assist us in providing an established data set with known attacks for community research

5. Ethernet Cable Giganet Cable Computing Node Head Node: Dual 450MHz Pentium III512 MB RAM6 GB Local Hark Disk Microcosm1 Microcosm2 Microcosm3 Microcosm4 CS Network MPI Messages TCP/IP packets 100Mbps Ethernet Switch Giganet Switch Data Bro/Snort & IIDS Data Repository MPI Messages Data Microcosm0 Microcosm5 Microcosm6 Microcosm7 Microcosm8 Bro/Snort & IIDS Data Repository Structure of the Cluster: Microcosm

6. Network Traffic or Audit Data (1) Network Traffic or Audit Data (2) Network Traffic or Audit Data (m) …... Machine Learning Component (Mining fuzzy association rules and fuzzy frequent episodes ) Background Unit Core Component Intrusion Detection Module 1 Intrusion Detection Module n+1 Intrusion Detection Module n’ … . . . . . . Other methods Security Administrator Decision Engine Intrusion Detection Module n File Repository Communication Module Server Host or Network Device (e.g., router) Host or Network Device (e.g., router) Host or Network Device (e.g., router) …... Clients The Architecture of the IIDS

7. Daemon Attacks • daemon process: A Non-interactive process running in the background that cannot maintain an association with a terminal • can not write error messages on standard error • can write files and do any other operation allowed to a normal process

8. Daemon Attacks (Cont.) • Trojan horse implemented in a trusted application, that can execute the daemon process at any moment • Can be implemented in a C and/or MPI environment • Example: DoS (buffer overflow, resource attack)

9. Child Process Grandchild Process Daemon Attacks (Cont.) Steps to Follow: Trusted Process • fork() Trusted process • setid() Make the new process a group leader, session leader, and lose control of tty • signal(SIGHUP, SIG_IGN) Ignore SIGHUP that will be sent to a child of the process • umask(0) lose file creation mask inherited from parent • chdir(“/”) Change to working dir • fork() Child process (grandchild) • signal(SIGPIPE, SIG_IGN) Ignore for reading and writing to non-opened pipes.

10. Interposition Library Attacks • Used for tuning performance, collecting runtime statistics, function/parameter information or debugging applications. • Is a library that is loaded into the system to intercept any function call that an application makes to any selected shared library (libc, MPI PRO, etc) • Once the function call is intercepted, the behavior of the function can be altered or changed to perform an attack on the system. • Linux LD_PRELOAD environment variable

11. Interposition Library Interposition Library Attacks (Cont.) Steps to Follow: MPI or C Executable Program • Create library • Function headers • Mount the library • LD_PRELOAD Libc libmpipro System Calls Linux Kernel

12. Interposition Library Attacks (Cont.) • Linux Dynamically Loaded (DL) Libraries Package provides specific functions that assist in the implementation of the interposition library • dlsym: provides the address of a function symbol. • dlopen: opens a library • dlclose: closes a library

13. Interposition Library Attacks (Cont.) FILE *fopen(const char *filename, const char *mode) { // data referent to the real function typedef FILE*(*function_type) (const char *filename, const char *mode); static function_type function=NULL; static char* function_name="fopen"; FILE *retvalue; // Search for the next function that fits the specification of function_name if (!function){ function = (function_type) dlsym(RTLD_NEXT,function_name); } // If the profiling is active if (DoProfile){ // Malicious code here!!! } else //do not profile, only execute retval = ((*function)(filename,mode)); return (retval); }

14. Interposition Library Attacks (Cont.) • Examples: LIBC • fopen (file open function libc): the requested file can be copied in another directory with the user permissions changed so it can be accessed later by the attacker. • fread, fwrite (read/write operation on files libc): can corrupt the information before the real function is called. • malloc, memcpy (libc): memory management.

15. Interposition Library Attacks (Cont.) • Examples: MPI • MPI_INIT, MPI_FINALIZE: randomly generates a daemon attack on each processor that is running the distributed application to slow down the system. • MPI_Recv, MPI_Send : corrupting the transmitted data between processes. • The LIBC attacks can be applied here too.

16. Device Driver Attack • Implemented for a Linux environment • Trojan horse inserted in a device driver, installed on the system • Is loaded into the system to intercept any system call that an application makes to the kernel of the OS (sys_open, sys_close, etc) • Change the current pointers of the system calls of the kernel, for other functions • Can be used for: DoS, stealing information or hijacking the system.

17. Device Driver Attack (Cont.) MPI or C Executable Program System Calls Interposed Calls Linux Kernel

18. Detection of Cluster Attacks • Longer goal: design and building of a prototype Intelligent Intrusion Detection System (IIDS) that demonstrates the effectiveness o Artificial Intelligence approaches in a HPC. • MPI (Message Passing Interface) • Problem: Large amount of data to present and analyze.

19. Detection of Cluster Attacks (Cont.) Current mechanisms • Detection with system calls • Neural Networks • Detection with function calls • Hidden Markov Models Creation of an attack database

20. Conclusions • The reported attacks were tested in a Linux OS and MPI environment • We have shown some vulnerabilities in a cluster environment using MPI • We created anomalous data sets where none previously existed