1 / 49

Bitmap Algorithms for Counting Active Flows on High Speed Links

Learn about bitmap algorithms for counting active flows on high-speed links and their applications in network operations.

garygordon
Download Presentation

Bitmap Algorithms for Counting Active Flows on High Speed Links

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bitmap Algorithms for Counting Active Flows on HighSpeed Links Cristian Estan, George Varghese, Mike Fisk Computer Science and Engineering Department, University of California, San Diego Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  2. Why count flows? • Detect port/IP scans • Identify DoS attacks • Estimate spreading rate of a worm • Packet scheduling Dave Plonka’s FlowScan Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  3. Network bandwidth Memory size & bandwidth Memory Existing flow counting solutions Router Traffic reports Server NetFlow data Analysis Fast link Network Network Operations Center Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  4. Motivating question • Can we count flows at line speeds at the router? • Wrong solution – counters • Naïve solution – use hash tables (like NetFlow) • Our approach – use bitmaps Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  5. Bitmap counting algorithms • A family of algorithms that can be used as building blocks in various systems • Algorithms can be adapted to application • Low memory and per packet processing • Generalize flows to distinct header patterns • Count flows or source addresses to detect attack • Count destination address+port pairs to detect scan Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  6. Talk structure • Per packet processing for bitmap algorithms • Computing flow count estimates from bitmaps • Variance analysis of estimates • Derived algorithms • Related work • Measurements • Conclusions Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  7. Bitmap counting – direct bitmap Set bits in the bitmap using hash of the flow ID of incoming packets HASH(green)=10001001 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  8. Bitmap counting – direct bitmap Different flows have different hash values HASH(blue)=00100100 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  9. Bitmap counting – direct bitmap Packets from the same flow always hash to the same bit HASH(green)=10001001 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  10. Bitmap counting – direct bitmap Collisions OK, estimates compensate for them HASH(violet)=10010101 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  11. Bitmap counting – direct bitmap HASH(orange)=11110011 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  12. Bitmap counting – direct bitmap HASH(pink)=11100000 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  13. Bitmap counting – direct bitmap As the bitmap fills up, estimates get inaccurate HASH(yellow)=01100011 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  14. Bitmap counting – direct bitmap Solution: use more bits HASH(green)=10001001 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  15. Bitmap counting – direct bitmap Solution: use more bits Problem: memory scales with the number of flows HASH(blue)=00100100 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  16. Bitmap counting – virtual bitmap Solution: a) store only a portion of the bitmap b) multiply estimate by scaling factor Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  17. Bitmap counting – virtual bitmap HASH(pink)=11100000 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  18. Bitmap counting – virtual bitmap Problem: estimate inaccurate when few flows active HASH(yellow)=01100011 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  19. Bitmap counting – multiple bmps Solution: use many bitmaps, each accurate for a different range Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  20. Bitmap counting – multiple bmps HASH(pink)=11100000 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  21. Bitmap counting – multiple bmps HASH(yellow)=01100011 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  22. Bitmap counting – multiple bmps Use this bitmap to estimate number of flows Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  23. Bitmap counting – multiple bmps Use this bitmap to estimate number of flows Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  24. OR OR Bitmap counting – multires. bmp Problem: must update up to three bitmaps per packet Solution: combine bitmaps into one Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  25. Bitmap counting – multires. bmp HASH(pink)=11100000 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  26. Bitmap counting – multires. bmp HASH(yellow)=01100011 Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  27. Talk structure • Per packet processing for bitmap algorithms • Computing flow count estimates from bitmaps • Variance analysis of estimates • Derived algorithms • Related work • Measurements • Conclusions Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  28. Basic estimates Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  29. Find most accurate component Estimate number of flows hashing to it Apply scaling factor Multiresolution bitmap estimate Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  30. Talk structure • Per packet processing for bitmap algorithms • Computing flow count estimates from bitmaps • Variance analysis of estimates • Derived algorithms • Related work • Measurements • Conclusions Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  31. Relative error in estimates Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  32. Error of virtual bitmap Average (relative) error Flow density (flows/bit) Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  33. Memory requirements Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  34. 100 million flows, error 1% Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  35. Talk structure • Per packet processing for bitmap algorithms • Computing flow count estimates from bitmaps • Variance analysis of estimates • Derived algorithms • Related work • Measurements • Conclusions Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  36. Triggered bitmap • Need multiple instances of counting algorithm (e.g. port scan detection) • Many instances count few flows • Triggered bitmap • Allocate small direct bitmap to new sources • If number of bits set exceeds trigger value, allocate large multiresolution bitmap Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  37. Talk structure • Per packet processing for bitmap algorithms • Computing flow count estimates from bitmaps • Variance analysis of estimates • Derived algorithms • Related work • Measurements • Conclusions Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  38. Related work • Flajolet, Martin (1985) probabilistic counting • Memory use similar to multiresolution bitmap • Whang et al (1990) introduce direct bitmap • You, Chang (1996) use virtual bitmap • Chauduri, Motwani, Narasayya (1998) • Counting flows without bias impossible from sampled data • Duffield, Lund, Thorup (2002) • Accurate solutions based on counting TCP SYN flags Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  39. Talk structure • Per packet processing for bitmap algorithms • Computing flow count estimates from bitmaps • Variance analysis of estimates • Derived algorithms • Related work • Measurements • Conclusions Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  40. Multires. bmp. vs. prob. counting Average (relative) error Number of flows (log scale) Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  41. Scan detection memory usage Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  42. Talk structure • Per packet processing for bitmap algorithms • Computing flow count estimates from bitmaps • Variance analysis of estimates • Derived algorithms • Related work • Measurements • Conclusions Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  43. A family of counting algorithms Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  44. Bitmap counting algorithms • A family of algorithms that can be used as building blocks in various systems • Algorithms can be adapted to application • Low memory and per packet processing • With 2Kbytes error around 1% Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  45. The end Bitmap algorithms will be available at: http://ial.ucsd.edu/bitmaps/ Any questions? Acknowledgements: Vern Paxson, David Moore, Philippe Flajolet, Marianne Durand, Alex Snoeren, K Claffy, Stefan Savage, Florin Baboescu, NIST,NSF Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  46. Adaptive bitmap • Virtual bitmap measures accurately number of flows if range known in advance • Often number of flows does not change rapidly • Measurement repeated • Can use previous measurement to tune virtual bitmap • Combine a large virtual bitmap with a small multiresolution bitmap used for tuning Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  47. Adaptive bitmap accuracy Average (relative) error Number of flows (log scale) Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  48. With 2 kilobytes of memory Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

  49. Increment-decrement algorithms • Active flow defined as flow with packets in queue • Must support additions and deletions • Replace bits of bitmap with counters • Increment when packet arrives • Decrement when packet leaves • Estimate number of flows based on zero counters Bitmap algorithms for flow counting – Internet Measurement Conference, October 2003

More Related