Adaptive Intrusion Detection and Mitigation Systems for WiMAX Networks

1. Adaptive Intrusion Detection and Mitigation Systems for WiMAX Networks Yan Chen Northwestern Lab for Internet and Security Technology (LIST) Dept. of Computer Science Northwestern University http://list.cs.northwestern.edu Motorola Liaisons Gregory W. Cox, Z. Judy Fu, Philip R. Roberts Motorola Labs

2. The Spread of Sapphire/Slammer Worms

3. Current Intrusion Detection Systems (IDS) • Mostly host-based • not scalable to high-speed networks • Mostly simple signature-based • Can’t deal with unknown attacks, polymorphic worms • Statistical detection • Unscalable for flow-level detection • Overall traffic based: inaccurate, high false positives • Cannot differentiate malicious events with unintentional anomalies

4. Adaptive Intrusion Detection and Mitigation for WiMAX Networks (WAIDM) • Attached to a switch connecting BS as a black box • Enable the early detection and mitigation of global scale attacks • Could be differentiator for Motorola’s 802.16 products Users Internet Users WAIDM system Internet 802.16 scan port 802.16 BS BS Switch/ Switch/ BS controller BS controller 802.16 802.16 BS BS Users Users (a) (b) WAIDM deployed Original configuration

5. Adaptive Intrusion Detection and Mitigation for WiMAX Networks (WAIDM) • Scalability • Online traffic recording • Reversible sketch for data streaming computation • Record millions of flows (GB traffic) in a few hundred KB • Online sketch-based flow-level anomaly detection • Adaptively learn the traffic pattern changes • Accuracy Integrated approach for false positive reduction • Automatic Polymorphic Worm signature generation (Hamsa) • Network element fault Diagnostics with Operational Determinism (ODD)

6. WAIDM Architecture Remote aggregated sketch records Sent out for aggregation Part I Sketch-based monitoring & detection Reversible sketch monitoring Normal flows Sketch based statistical anomaly detection (SSAD) Local sketch records Streaming packet data Keys of suspicious flows Filtering Keys of normal flows Polymorphic worm detection (Hamsa) Signature-based detection Per-flow monitoring Suspicious flows Part II Per-flow monitoring & detection Network fault diagnosis (ODD) Intrusion or anomaly alarms Modules on the critical path Modules on the non-critical path Data path Control path

7. Accomplishments • Motorola Interactions • The first two components of WAIDM are ready for field test on Motorola WiMAX networks or testbed • Product teams interested to use as differentiator (Networks security service director: Randall Martin) • Close collaboration/interaction with Motorola Labs (Judy Fu, Phil Roberts, Steve Gilbert) • Patents being filed through Motorola • Reverse Hashing for High-speed Network Monitoring: Algorithms, Evaluation, and Applications. • Hamsa: Fast Signature Generation for Zero-day Polymorphic Worms with Provable Attack Resilience. • Students involved • Three Ph.D. students: Yan Gao, Zhichun Li, & Yao Zhao • One M.S. student: Prasad Narayana

8. Accomplishments on Publications • Four conference papers and one journal papers (with another four under submission) • A DoS Resilient Flow-level Intrusion Detection Approach for High-speed Networks, to appear in IEEE International Conference on Distributed Computing Systems (ICDCS), 2006 (14%). • Hamsa: Fast Signature Generation for Zero-day Polymorphic Worms with Provable Attack Resilience, to appear in IEEE Symposium on Security and Privacy, 2006 (about 8%). • Reverse Hashing for High-speed Network Monitoring: Algorithms, Evaluation, and Applications, Proc. of IEEE INFOCOM, 2006 (18%). • IDGraphs: Intrusion Detection and Analysis Using Stream Compositing, to appear in IEEE Computer Graphics & Applications, special issue on Visualization for Cyber Security, 2006. • An earlier version also in Proc. of the IEEE Workshop on Visualization for Computer Security (VizSEC), 2005

9. Hamsa: Fast Signature Generation for Zero-day Polymorphic Wormswith Provable Attack Resilience

10. Desired Requirements for Polymorphic Worm Signature Generation • Network based, no host-level info • Noise tolerant • Most network flow classifiers suffer false positives. • Even host based IDSes, such as honeynets, can be injected with noise. • Attack resilience • Attackers always try to evade the IDS • Efficient signature matching for high-speed links No existing work satisfies these requirements !

11. Hamsa Architecture

12. Choice of Signatures Two classes of signatures • Content based • Invariant content • Protocol Frame • Control Data: leading to control flow hijacking • Worm Executable Payload • Token: a substring with reasonable coverage to the suspicious traffic • Signatures: conjunction and/or sequence of tokens • Behavior based Our choice: content based • Fast signature matching • ASIC based approach can achieve 6 ~ 8Gbps • Generic, not depend upon any protocol or server

13. Hamsa Design • Key idea: model the uniqueness of worm invariants • Greedy algorithm for finding token conjunction signatures • Highly accurate while much faster • Both analytically and experimentally • Compared with the latest work, polygraph • Suffix array based token extraction • Provable attack resilience guarantee • Propose an adversary model • Noise tolerant

14. Hamsa Signature Generator • Core part: Model-based Greedy Signature Generation • Iterative approach for multiple worms • Signature refinement for better specificity • False positive is worse than false negative

15. Experiment Methodology • Experiential setup: • Suspicious pool: • Three pseudo polymorphic worm based on real exploits (Code-Red II, Apache-Knacker and ATPhttpd), • Two polymorphic engine from Internet (CLET and TAPiON). • Normal pool: 2 hour departmental http trace (326MB) • Signature evaluation: • False negative: 5000 generated worm samples per worm • False positive: • 4-day departmental http trace (12.6 GB) • 3.7GB web crawling including .mp3, .rm, .ppt, .pdf, .swf etc. • /usr/bin of Linux Fedora Core 4

16. Results on Signature Quality • Single worm with noise • Suspicious pool size: 100 and 200 samples • Noise ratio: 0%, 10%, 30%, 50% • Noise samples randomly picked from the normal pool • Always get above signature and accuracy • Multiple worms with similar results

17. Speed and Attack Resilience Results • Implementation with hybrid of C++/Python • 500 samples with 20% noise, 326MB normal traffic pool, 15 seconds on an XEON 2.8Ghz • Provable attack resilience • We propose a new attack, token-fit • It fails the existing state-of-the-art, Polygraph • BUT We still can generate correct signature!

18. Ongoing Work • Semantics Aided Signature Generation for Zero-day Polymorphic Worms • Some worms do not have any content invariant • Incorporate semantic information for more accurate detection • Vulnerability Analysis for 802.16 WiMAX Network Protocols • Use formal verification methods to automatically search for vulnerabilities in 802.16 specs. • Completeness and correctness