1 / 22

An Information-theoretic Approach to Network Measurement and Monitoring

An Information-theoretic Approach to Network Measurement and Monitoring. Yong Liu, Don Towsley, Tao Ye, Jean Bolot. Outline. motivation background flow-based network model full packet trace compression marginal/joint coarser granularity netflow and SNMP future work. Motivation.

tanika
Download Presentation

An Information-theoretic Approach to Network Measurement and Monitoring

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Information-theoretic Approach to Network Measurement and Monitoring Yong Liu, Don Towsley, Tao Ye, Jean Bolot

  2. Outline • motivation • background • flow-based network model • full packet trace compression • marginal/joint • coarser granularity • netflow and SNMP • future work

  3. Motivation • network monitoring: sensing a network • traffic engineering, anomaly detection, … • single point v.s. distributed • different granularities • full traffic trace: packet headers • flow level record: timing, volume • summary statistics: byte/packet counts • challenges • growing scales: high speed link, large topology • constrained resources: processing, storage, transmission • 30G headers/hour at UMass gateway • solutions • sampling: temporal/spatial • compression: marginal/distributed

  4. Questions • how much can we compress monitoring traces? • how much information is captured by different monitoring granularity? • packet trace/NetFlow/SNMP • how much joint information is there in multiple monitors? • joint compression • trace aggregation • monitor placement

  5. Our Contribution • flow-based network models • explore temporal/spatial correlation in network traces • projection to different granularity • information theoretic framework • entropy: bound/guideline on trace compression • quantitative approach for more general problems • validation against measurement from operational network

  6. Entropy & Compression • Shannon entropy of discrete r.v. • compression of i.i.d. symbols (length M) by coding • coding: • expected code length: • info. theoretic bound on compression ratio: • Shannon/Huffman coding • assign short codeword to frequent outcome • achieve the H(X) bound

  7. Entropy & Correlation • joint entropy • entropy rate of stochastic process • exploit temporal correlation • Lempel-Ziv Coding: (LZ77, gzip, winzip) asymptotically achieve the bound for stationary process • joint entropy rate of correlated processes • exploit spatial correlation • Slepian-Wolf Coding: (distributed compression) encode each process individually, achieve jointentropy rate in limit

  8. Network Trace Compression • naïve way: treat as byte stream, compress by generic tools • gzip compress UMass traces by a factor of 2 • network traces are highly structured data • multiple fields per packet • diversity in information richness • correlation among fields • multiple packets per flow • packets within a flow share information • temporal correlation • multiple monitors traversed by a flow • most fields unchanged within the network • spatial correlation • network models • explore correlation structure • quantify information content of network traces • serves as lower bounds/guidelines for compression algorithms

  9. Packet Header Trace 0 16 31 time stamp (sec.) Timing time stamp (sub-sec.) vers. HLen ToS total length IPID flags fragment offset TTL protocol header checksum IP Header source IP address destination IP address source port destination port data sequence number acknowledgment number TCP Header Hlen TCP flags window size checksum urgent pointer

  10. Header Field Entropy 0 16 31 time stamp (sec.) Timing time time stamp (sub-sec.) vers. HLen ToS total length IPID flags fragment offset TTL protocol header checksum IP Header source IP address flow id destination IP address source port destination port data sequence number acknowledgment number TCP Header Hlen TCP flags window size checksum urgent pointer

  11. packet inter-arrival: # bits per packet: • flow based trace: T0 F0 T3 F0 Tm F0 • flow record: F0 K T0 flow ID flow size arrival time packet inter-arrival Single Point Packet Trace T0 F0 T1 F1 T3 F0 Tm F0 Tn Fn • temporal correlation introduced by flows • packets from same flow closely spaced in time • they share header information

  12. Network Models • flow-based model • flow arrivals follow Poisson with rate • flows are classified to independent flow classes according to routing (the set of routers traversed) • flow i is described by: • flow inter-arrival time: • flow ID: • flow length: • packet inter-arrival time within the flow: • packet arrival stochastic process:

  13. Entropy in Flow Record • # bits per flow: • # bits per second: • marginal compression ratio • determined by flow length (pkts.) and variability in pkt. inter-arrival.

  14. Single Point Compression: Results • Compression ratio lower bound calculated by entropy much lower than real compression algorithm • Real compression algorithm difference • Records IPID, packet size, TCP/UDP fields • Fixed packet buffer for each flow => many flow records for long flows

  15. Distributed Network Monitoring • single flow recorded by multiple monitors • spatial correlation: traces collected at distributed monitors are correlated • marginal node view:#bits/sec to represent flows seen by one node, bound on single point compression • network system view:#bits/sec to represent flows cross the network, bound on joint compression • joint compression ratio: quantify gain of joint compression

  16. Baseline Joint Entropy Model • “perfect” network • fixed routes/constant link delay/no packet loss • flow classes based on routes • flows arrive with rate: • # of monitors traversed: • #bits per flow record: • info. rate at node v: • network view info. rate: • joint compression ratio: • dependence on # of monitors travered

  17. Joint Compression: Results

  18. Coarser Granularity Models • NetFlow model • similar to flow model: • joint compression result similar to full trace • SNMP model • any link SNMP rate process is sum of rate processes of all flow classes passing through that link • traffic rates of flow classes are independent Gaussian • entropy can be calculated by covariance of these processes • information loss due to summation • small joint information between monitors • difficult to recover rates of flow classes from SNMP data

  19. Joint Compression Ratio of Different Granularity

  20. Conclusion • information theoretic bound on marginal compression ratio -- ~ 20% (time+flow id, even lower if include other low entropy fields) • marginal compression ratio high (not very compressible) in SNMP, lower in NetFlow, and the lowest in full trace • joint coding is much more useful/nessassary in full trace case than in SNMP • “More entropy for your buck”

  21. Future Work • network impairments • how many more bits for delay/loss/route change • model netflow with sampling • distributed compression algorithms • lossless v.s. lossy compression • entropy based monitor placement • maximize information under constraints

  22. Thanks!

More Related