1 / 10

To the Terabyte and Beyond! Leveraging Pilot and Wireshark to Analyze Truly Massive Packet Traces

To the Terabyte and Beyond! Leveraging Pilot and Wireshark to Analyze Truly Massive Packet Traces June 17, 2010 Loris Degioanni CTO | CACE Technologies SHARK FEST ‘10 Stanford University June 14-17, 2010. Packet Aquisition. Capture Card. Dedicated card is essential

signa
Download Presentation

To the Terabyte and Beyond! Leveraging Pilot and Wireshark to Analyze Truly Massive Packet Traces

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. To the Terabyte and Beyond! Leveraging Pilot and Wireshark to Analyze Truly Massive Packet Traces June 17, 2010 Loris Degioanni CTO | CACE Technologies SHARKFEST‘10 Stanford University June 14-17, 2010

  2. Packet Aquisition

  3. Capture Card • Dedicated card is essential • No network stack overhead • Minimizes copies • Optimizes locality • Filtering capability in the card normally not really useful • Unless in some unusual conditions, the application wants to see everything • PCI bus is the only resource that card filtering optimizes • Any tap nowadays can do basic filtering • Small packets is the worst condition • CACE Turbocap • Hybrid between home-built and off the shelf • No unnecessary features (who needs filtering?) • Affordable price

  4. CPU • Bottlenecks • CPU clock (expensive) • Number of CPUS (cheap) • Multi-threading hard to leverage when capturing and processing network packets • Network monitoring is intrinsically sequential • Locking is evil • Doing things more than once is better than locking • At 10Gbps, cache coherency is a big deal • Small packets is the worst condition

  5. Disk • Bottlenecks • Single disk write speed • Number of spindles • Raid Controller • Big packets is the worst condition • Solid State? Not a good idea yet • Single disk performance is not really the bottleneck • Cost is an important factor when you build a system with tens of disks • Reliability not as proven as the old magnetic disks

  6. Disk write speed based on position

  7. I can capture a lot of packets. Now what? • Read of packets must be non-disruptive! • Even if I stop the capture process, since I was writing at full speed, reading the data is going to take around the same time of writing it • Read needs to be localized • I need high level visibility to reach the point I need Indexing

  8. Standalone card vs. kit A network card nowadays is not enough to build a functional packet capture system.

  9. Indexing • While capturing, on a Shark Appliance capture job • On a trace file, after the fact • Summary of the network traffic • Volume, talkers and protocol information • Coordinated with the packet store • “Netflow on steroids” • Designed to be extremely efficient in terms of disk usage • Coordinated with the packet store

  10. Indexing Index file Time intervals Index entry Time index File Positions pcap file Packet

More Related