HyperSpector: Virtual Distributed Monitoring Environments for Secure Intrusion Detection

1. HyperSpector: Virtual Distributed Monitoring Environments forSecure Intrusion Detection Kenichi Kourai Shigeru Chiba Tokyo Institute of Technology

2. Distributed intrusiondetection system (DIDS) • Useful to achieve self-monitoring of distributed systems • Towards self-protection • Consists of multiple IDSes • Including • Host-based IDS (HIDS) • Network-based IDS (NIDS) • IDSes cooperate with each otheror with an analyzer analyzer IDS server distributed system

3. Threats against the DIDS • Active attacks • Directly take actions against IDSes by • Sending malicious packets to network ports used by IDSes • modifying IDS policy files or terminating IDS processes • Passive attacks • Wait until IDSes read data including malicious code by • Sending malicious packets to monitored servers • Changing attributes of monitored files active attack IDS monitor server passive attack

4. mirroring port Traditional approach:Isolated monitoring DIDS • Isolates NIDSes from servers physically • Using NIDS hosts and a back-end switch • NIDS hosts monitor packets by port mirroring • NIDS hosts are connected to mirroring ports in a front-end switch • The front-end switch duplicates and forwards packets back-end switch NIDS host server host front-end switch Internet

5. Security ofisolated monitoring DIDS • Prevents active attacks • The attacker cannot attack NIDS hosts using mirroring ports • Mirroring ports are only for monitoring • Confines the impact of passive attacks to within the DIDS • The attacker cannot access the outside of the DIDS • Important because preventing passive attacks is difficult back-end switch NIDS host server host front-end switch mirroring port Internet

6. Problems inisolated monitoring • Need additional hardware • Lots of machines for NIDSes • A back-end switch • A front-end switch with port mirroring • Support only NIDSes • Legacy HIDSes do not support monitoring of remote server hosts • Achieving secure monitoring of remote server hosts from HIDS hosts is difficult

7. Our approach: HyperSpector • Virtual distributed monitoring environment • IDS VM and server VM • Isolate each other withoutadditional hardware • The IDS VM can monitorthe server VM • A virtual network • Connects the IDS VMs • Isolated from a network usedby servers DIDS virtual network IDS VM IDS VM server VM server VM

8. Inter-VMmonitoring mechanisms • Requirements • Interfaces to legacy IDSes • Secure monitoring between VMs • HyperSpector provides three mechanisms • Software port mirroring (for packet capturing) • Inter-VM disk mounting (for file system checking) • Inter-VM process mapping (for process checking)

9. mirroring port Software port mirroring • Virtual switch • Achieves port mirroring by software • Connects its mirroring port to the IDS VM • Using a virtual network interface (VNI) • Duplicates and forwards packets to the IDS VM server VM IDS VM NIDS BPF device VNI virtual switch VMM outside

10. read shadow file system Inter-VM disk mounting • Inter-VM disk mounter • Mounts the file system of the server VM on the IDS VM • As a shadow file system • Forwards requests to a shadow file system to the server VM • Using VMM interfaces server VM IDS VM file system HIDS VMM interface inter-VMdisk mounter VMM

11. shadow process ptrace wakeup Inter-VM process mapping • Inter-VM process mapper • Maps the processes in the server VM to the IDS VM • As shadow processes • Forwards • Requests to shadow processes to the server VM • Notifications from the server VM to HIDSes • Using VMM interfaces server VM IDS VM server process HIDS VMM interface inter-VMprocess mapper VMM

12. Prevents active attacks From the server VMs From hosts outside the DIDS Confines the impact of passive attacks The IDS VM cannot attack the server VM The IDS VM cannot attack hosts outside the DIDS Security of HyperSpector DIDS virtual network IDS VM IDS VM IDS VM IDS VM server VM server VM server VM server VM

13. outside hosts monitor request modify Security of the inter-VM monitoring mechanisms • Secure, because • The server VM cannot use inter-VM monitoring mechanisms • The IDS VM cannot interfere with the server VM • Inter-VM monitoring mechanisms are only for monitoring • The IDS VM cannot send monitored information outside the DIDS • Although it can view secret information of servers... serverVM IDS VM VMM

14. Implementation • We have implemented HyperSpector in the FreeBSD kernel • IDS VM and server VM • Based on our portspace • The portspace virtualizes onlya network system, file system,and processes • Secure enough • We assume the kernel and thebase system are not exploitable base system serverVM IDS VM net net fs fs fs VMM kernel

15. Implementation of the VMM • Implemented efficiently in the kernel • Virtual switch • Maps a network interface of the server VM to the IDS VM in a read-only manner • Inter-VM disk mounter • Mounts the file system of the server VM on the IDS VM read-only, using the modified union file system • Inter-VM process mapper • Makes the IDS VM share the processes of the server VM in a read-only manner

16. Experiments • We measured overhead of HyperSpector • Experimental setup • Snort, Tripwire, or truss in the IDS VM • thttpd in the server VM • ApacheBench in the client host • Hardware • 2 PCs (3.0 GHz Pentium 4,1 GB of memory, Intel Pro/100+) • 100Base-T network switch IDS VM client host server VM server host

17. Snort • Monitors packets fromApacheBench to thttpd • We measured thethroughput of thttpd • For comparison • The base system • Isolated monitoring • Maximum overhead • 7.5% slower than the base system • 7% slower than isolated monitoring (over 2 KB file size) • 30% in 0 KB file size

18. Tripwire • Checks the integrity ofthe whole file system • 54,885 objects • We measured the timeof the integrity check • altering the file changerate • For comparison • The base system • Overhead • 17 to 26% slower than the base system

19. Truss • Traces system callsissued by thttpd • We measured thethroughput of thttpd • Using ApacheBench • For comparison • The base system • Overhead • 0.8 to 7.3% slower than the base system

20. Related work • ReVirt [Dunlap’02], Livewire [Garfinkel’03] • Enable IDSes to monitor servers running in a VM • The VM protects IDSes from active attacks via servers • Do not consider other attacks against IDSes • Backdoors [Bohra’04] • Enables isolated monitoring for HIDSes • Using programmable NICs to monitor server state • Needs much hardware • Insecure because HIDS hosts are network-reachable These need to develop specialized IDSes

21. Conclusion • We proposed HyperSpector, which • Isolates IDSes from servers without additional hardware • Using IDS VMs, server VMs, and a virtual network • Provides secure Inter-VM monitoring mechanisms: • Software port mirroring, inter-VM disk mounting, and inter-VM process mapping • Prevents active attacks and confines the impact of passive attacks to within the DIDS

22. Future work • Support for active monitoring • Needs a mechanism to securely send probe messages to servers • Support for DoS attacks • Needs to allocate sufficient resources to the IDS VM even under overload • Automatic detection of compromised HyperSpector • Monitoring resource usage may help