1 / 27

Benchmarking Anomaly-based Detection Systems

Benchmarking Anomaly-based Detection Systems. Ashish Gupta Network Security May 2004. Overview. The Motivation for this paper Waldo example The approach Structure in data Generating the data and anomalies Injecting anomalies Results Training and Testing: the method Scoring

gknott
Download Presentation

Benchmarking Anomaly-based Detection Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Benchmarking Anomaly-based Detection Systems Ashish Gupta Network Security May 2004

  2. Overview • The Motivation for this paper • Waldo example • The approach • Structure in data • Generating the data and anomalies • Injecting anomalies • Results • Training and Testing: the method • Scoring • Presentation • The ROC curves: somewhat obvious

  3. Motivation Does anomaly detection depend on regularity/randomness of data ?

  4. Where’s Waldo !

  5. Where’s Waldo !

  6. Where’s Waldo !

  7. The aim • Hypothesis: • Differences in data regularity affect anomaly detection • Different environments  different regularity • Regularity • Highly redundant or random ? • Example of environment’s affect 010101010101010101010101 Or 0100011000101000100100101

  8. Consequences One IDS : Different False Alarm Rates Need custom system/training for each environment ? Temporal affects: Regularity may vary over time ?

  9. Structure in data Measuring randomness

  10. 010101010101010101010101 Or 0100011000101000100100101 Measuring Randomness + Relative Entropy Sequential Dependence Conditional Relative Entropy

  11. The benchmark datasets • Three types: • Training data ( the background data) • Anomalies • Testing data ( background + anomalies ) • Generating the sequences • 5 sets, each set  11 files ( for increasing regularity) • Each set  different alphabet size • Alphabet size  decides complexity

  12. Anomaly Generation • What’s a surprise ? • Different from the expected probability • Types: • Juxta-positional : different arrangements of data • 001001001001001001111 • Temporal • Unexpected periodicities • Other types ?

  13. Types in this paper • Foreign symbol • AAABABBBABABCBBABABBA • Foreign n-gram • AAABABAABAABAAABBBBA • Rare n-gram • AABBBABBBABBBABBBABBBABBAA

  14. Injecting anomalies • Make sure not more than 0.24 %

  15. The experiments The Hypothesis is true

  16. The hypothesis: • Nature of “normal” background noise affects signal detection • The anomaly detector • To detect anomalous subsequences • Learning phase  n-gram probability table • Unexpected event  anomaly ! • Anomaly threshold decides level of surprise

  17. Example of anomaly detection AAC  ANOMALY !

  18. Scoring • Event outcomes • Hits • Misses • False alarms • Threshold • Decides level of surprise • 0  completely unsurprising, 1  astonishing • Need to calibrate

  19. Presentation of results • Presents two aspects: • % correct detections • % false detections • Detector operates through a range of sensitivities • Higher sensitivity  ? • Need the right sensitivity

  20. Interpretation • Nothing overlaps  regularity affects detection !

  21. What does this mean ? • Detection metrics are data dependent • Cannot say: • My XYZ product will flag down 75% percent anomalies with 10% false hit rate ! • Sir, are you sure ?

  22. Real world data • Regularity index for system calls for different users

  23. Is this surprising ? • What about network traffic ?

  24. Conclusions Anomaly Detection Effectiveness Data Structure Evaluation is data dependent

  25. Conclusions Different system Or Change the parameters Change in regularity

  26. Quirks ? • Assumes rather naïve detection systems • “Simple retraining will not suffice” • An intelligent detection can take this into account. • What is really an anomaly ? • If data is highly irregular, won’t randomness produce some anomalies by itself • Anomaly is a relative term • Here anomalies are generated independently

More Related