1 / 44

Stefan Burschka

Tranalyzer Feel the packets, be the packets. Stefan Burschka. What we do:. Network Troubleshooting, Security: TRANALYZER(T2/3): High Speed and Volume Traffic Analyzer TRAVIZ: Graphical Toolset for Tranalyzer Complete Tool Sets for Traffic Mining (TM) , Forensics Artificial Intelligence

schulter
Download Presentation

Stefan Burschka

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tranalyzer Feel the packets, be the packets Stefan Burschka

  2. What we do: • Network Troubleshooting, Security: • TRANALYZER(T2/3): High Speed and Volume Traffic Analyzer • TRAVIZ: Graphical Toolset for Tranalyzer • Complete Tool Sets for Traffic Mining (TM), Forensics • Artificial Intelligence Research: TM & Visualisation Brain support 4 multi-dim datasets Encrypted Traffic Mining Operational Picture Malware and covert channel detection Nifty stuff 2

  3. “It's the network – go fix it!” • 3

  4. The Network is slow, The Network is insecure; NO, it's not Microsoft, shut up, It wasn't me ... Production (poor Techie) Knows, Always warned, Always his fault: FUBAR License to get fired Finance (MBA) Knows basic calculus License to Excel Manager (MBA) Always right, DoR License to Powerpoint We didn't find the problem in 4 months, can you do the job in 2 weeks? (We supply 20TB data)

  5. Troubleshooting, Security Traffic Mining: Change your perspective 5

  6. What is wrong here? 6

  7. See the disaster now? Now you have context! 7

  8. Traffic Mining(TM): Hidden Knowledge: Listen | See, Understand, Invariants, Model • Application in • Troubleshooting, Security (Classification, Encrypted TM ) • Netzwerk usage(VoiP, P2P traffic shaping, application/user profiling) • Profiling & Marketing (usage performance- & market- index) • Law enforcement and Legal Interception (Indication/Evidence) 8

  9. Basic Need: Versatile Flow Compression A B Definition: (6-Tuple) Vlan(s), srcIP, srcpPort, dstIP, dstPort, L4Protocol Or why not a bit more context and meaning ? srcWho, dstWho srcNetwork, dstNetwork Bad, Good Internal / External 9

  10. Closed source loud Tools • Netflow (Sometimes not so loud, comes with routers) • Pro: Good hands-on tool, flow statistics, header parameters, standard • Cons: Not all statistics we need, no developer support GigaStor (Horrible loud and exceptional expensive HW) Pro: heuristic expert system, Graphics, reports, whatever is in the DB Cons: What we needed is not in the DB, no developer support DPI (Elacoya, Sandvine,..) (Terrible loud and expensive HW) Pro: good protocol resolution, nice reports Cons: Its a DPI not a verstile flow engine with developer support 10

  11. Open source silent SW • Wireshark, T-Shark (packet, flow statistics) • Pro: Hands-on tool, protocol db, GUI, command line, filtering • Cons: Limited flow statistics and file size, post processing difficult Silk (flow based) Cons:Not even close to Netflow, 5 tuple, esoteric config Netmate Pro: Flow, packet based, nice features, Cons: Config , handling, 5 tuple, that is, ... University NTOP(ng) Pro: Monitoring, flow statistics, config, GUI, Graphics Cons: not really flow based as we need it, protocol encapsulation? IDS (SNORT, BRO) Pro: Alarming, regex, flexible Cons: Alarming, no Flows, BRO: memory leaks, university stuff 11

  12. Need an Allrounder, script friendly between Wireshark, Netflow and 2006: Somebody has to develop me !!

  13. Tranalyzer2(T2), C99, (Geek/Dev/Prof) • High Volume Traffic Preprocessing and Troubleshooting Open Source • Speed and Memory optimized by *.h“, config and ./autogen.sh -n • Command line based, full pcap, eth and dag cards • Post processing : HEX, ‘text \t’; Bash, AWK, Perl, … friendly • C Plugin based, Linux, Mac, (Windoof) • Subnet labeling (Who, Where, What) • BPF • Hands-on: Anomaly and security related flags • Researchers: Full Statistical and Packet Signal Analysis support • Interfaces: Matlab, GnuPlot, SPSS, Excel, oocacl, soon Netflow tools • The “-s” option: The command line AWK, Perl friendly packet mode • GUI: Traviz (http://sourceforge.net/projects/traviz) • Easy to use but, You have to know your shit • 13

  14. T3, C99, (Geek/Normalo NonDev/Prof) High Speed and Volume Troubleshooting, Security, Monitoring • Complete new Concept and Design • Full IPv4/6, more protocols as T2 • Basic Features from T2 + new nifty Plugins • Full Subnet labeling and flexible flow aggregation • Multi Threading and Interface: High performance • GUI Support via professional Tool Set: Unlimited flows and files • ipSOM: AI Tool Set to answer ANY question • Core functions into DSP and FPGA in future for the 40Gig+ • More non geek/dev user friendly but, • You still have to know your shit • 14

  15. Report T2 • /tranalyzer -r ~/wurst/data/weichwurst.dmp -w ~/wurst/results/hartwurst • ================================================================================ • Tranalyzer 0.5.8 (Anteater), beta. PID: 6123 • ================================================================================ • Active plugins: • 00: protocolStatistics, version 0.5.8 --> _protocols.txt, ports.txt • 01: basicFlowOutput, version 0.5.8 --> _flow.txt / bin subnet.txt • 02: macRecorder, version 0.5.0 --> _flow.txt / bin • 03: portBasedClassifier, version 0.5.8 --> _flow.txt / bin, portmap.txt • 04: basicLayer4CalcStatistics, version 0.5.6 --> _flow.txt / bin • 05: tcpFlags, version 0.5.8 --> _flow.txt / bin • 06: tcpStates, version 0.5.6 --> _flow.txt / bin • 07: icmpDecode, version 0.5.8 --> _flow.txt / bin, _icmpStats.txt • 08: connectionCounter, version 0.5.5 --> _flow.txt / bin • 09: descriptiveStatistics, version 0.5.6 --> _flow.txt / bin • 10: nFirstPacketsStats, version 0.5.8 --> _flow.txt / bin • 11: packetSizeInterArrivalTimeHisto, version 0.5.8 --> _flow.txt / bin • 12: standardFileSink, version 0.5.0 --> creates text output _flow.txt • 13: textFileSink, version 0.5.8 --> creates binary output _flow.bin • Start processing file: /home/wurst//data/weichwurst.dmp • BPF: (null) • Dump start: 1351794649.186547 sec : Wed 01 Nov 2012 18:30:49.186547 • Shutting down Tranalyzer 0.5.8... • Dump stop: 1351837376.118852 sec : Thu 02 Nov 2012 06:22:42.118852 • Total dump duration: 42712.932305 sec • Number of processed packets: 6497970 • Number of processed traffic bytes: 1749617780 • Number of ARP packets: 1603 • Number of RARP packets: 5 • Number of IPv4 fragmented packets: 299 • Number of IPv6 packets: 0 • Number of IPv4 flows: 3395325 • Average snapped Bandwidth: 327.634 KBit/s • Average full IP Bandwidth: 326.386 Kbit/s • Warning: IPv4 Fragmentation header packet missing • 15

  16. T2 Protocol File • Total packets captured: 42278 • L4 Protocol # Packets Relative Frequency[%] Protocol description • 1 21 0.049671 Internet Control Message Protocol • 2 6 0.014192 Internet Group Management Protocol • 6 41698 98.628128 Transmission Control Protocol • 17 250 0.591324 User Datagram Protocol • 103 28 0.066228 Protocol Independent Multicast • Total TCP packets: 41698 • Port # Packets Relative Frequency[%] • 80 41519 99.570723 World Wide Web HTTP • 445 8 0.019186 Win2k+ Server Message Block • 5557 147 0.352535 • Total UDP packets: 250 • Port # Packets Relative Frequency[%] • 53 2 0.800000 Domain Name Server • 137 50 20.000000 NETBIOS, [trojan] Msinit • 138 21 8.400000 NETBIOS Datagram Service • 1900 18 7.200000 SSDP • 1908 2 0.800000 Dawn • 1985 156 62.400000 Hot Standby Router Protocol • 16

  17. T2 ICMP Stats File • Total # of ICMP messages: 22258 • ICMP / Total traffic percentage[%]: 0.343 • Echo reply / request ratio: 0.892 • Type Code # of Messages Relative Frequency [%] • ICMP_ECHOREQUEST - 111 0.499 • ICMP_ECHOREPLY - 99 0.445 • ICMP_SOURCE_QUENCH - 15 0.067 • ICMP_TRACEROUTE - 0 0.000 • ICMP_DEST_UNREACH ICMP_NET_UNREACH 60 0.270 • ICMP_DEST_UNREACH ICMP_HOST_UNREACH 15674 70.420 • ICMP_DEST_UNREACH ICMP_PROT_UNREACH 0 0.000 • ICMP_DEST_UNREACH ICMP_PORT_UNREACH 3100 13.928 • ICMP_DEST_UNREACH ICMP_FRAG_NEEDED 0 0.000 • ICMP_DEST_UNREACH ICMP_SR_FAILED 0 0.000 • ICMP_DEST_UNREACH ICMP_NET_UNKNOWN 0 0.000 • ICMP_DEST_UNREACH ICMP_HOST_UNKNOWN 0 0.000 • ICMP_DEST_UNREACH ICMP_HOST_ISOLATED 0 0.000 • ICMP_DEST_UNREACH ICMP_NET_ANO 8 0.036 • ICMP_DEST_UNREACH ICMP_HOST_ANO 600 2.696 • ICMP_DEST_UNREACH ICMP_NET_UNR_TOS 0 0.000 • ICMP_DEST_UNREACH ICMP_HOST_UNR_TOS 0 0.000 • ICMP_DEST_UNREACH ICMP_PKT_FILTERED 776 3.486 • ICMP_DEST_UNREACH ICMP_PREC_VIOLATION 0 0.000 • ICMP_DEST_UNREACH ICMP_PREC_CUTOFF 0 0.000 • ICMP_REDIRECT ICMP_REDIR_NET 1125 5.054 • ICMP_REDIRECT ICMP_REDIR_HOST 589 2.646 • ICMP_REDIRECT ICMP_REDIR_NETTOS 0 0.000 • ICMP_REDIRECT ICMP_REDIR_HOSTTOS 0 0.000 • ICMP_TIME_EXCEEDED ICMP_EXC_TTL 95 0.427 • ICMP_TIME_EXCEEDED ICMP_EXC_FRAGTIME 0 0.000 • ICMP_TRACEROUTE - 0 0.000 • 17

  18. T2 Flow Header File: Hands-On 20 ..... 21 8:NR Minimum layer3 packet size 22 8:NR Maximum layer3 packet size 23 19:NR Average packet load ratio 24 19:NR Send packets per second 25 19:NR Send bytes per second 26 19:NR Packet stream asymmetry 27 19:NR Byte stream asymmetry 28 8:NR IP Minimum delta IP ID 29 8:NR IP Maximum delta IP ID 30 7:NR IP Minimum TTL 31 7:NR IP Maximum TTL 32 7:NR IP TTL Change count 33 13:NR IP Type of Service 34 14:NR IP aggregated flags 35 8:NR IP options count 36 13,15:NR IP aggregated options • 18

  19. T2 Flow Header View: Hands-On 37 8:NR TCP packet seq count 38 10:NR TCP sent seq diff bytes 39 8:NR TCP sequence number fault count 40 8:NR TCP packet ack count 41 10:NR TCP flawless ack received bytes 42 8:NR TCP ack number fault count 43 8:NR TCP initial window size 44 19:NR TCP average window size 45 8:NR TCP minimum window size 46 8:NR TCP maximum window size 47 8:NR TCP window size change down count 48 8:NR TCP window size change up count 49 8:NR TCP window size direction change count 50 13:NR TCP aggregated protocol flags (cwr, ecn, urgent, ack, push, reset, syn, fin) 51 14:NR TCP aggregated header anomaly flags 52 8:NR TCP options Packet count 53 8:NR TCP options count 54 15:NR TCP aggregated options 55 8:NR TCP Maximum Segment Length 56 7:NR TCP Window Scale 57 19:NR TCP Trip Time Syn, Syn-Ack | Syn-Ack, Ack 58 19:NR TCP Round Trip Time Syn, Syn-Ack, Ack | TCP Ack-Ack RTT 59 19:NR TCP Ack Trip Min 60 19:NR TCP Ack Trip Max 61 19:NR TCP Ack Trip Average 62 13:NR TCP aggregated protocol state flags 63 15,14:NR ICMP Aggregated type & code bit field 64 19:NR ICMP Echo reply/request success ratio 65 9:NR Number of connections from source IP to different hosts 66 9:NR Number of connections from destination IP to different hosts 67 9:NR Number of connections between source IP and destination IP Yes I know, I should do something special for the TimeStamp option • 19

  20. T2 Flow Header View: TM geeks 68 19:NR Minimum packet length 69 19:NR Maximum packet length 70 19:NR Mean packet length 71 19:NR Lower quartile of packet lengths 72 19:NR Median of packet lengths 73 19:NR Upper quartile of packet lengths 74 19:NR Inter quartile distance of packet lengths 75 19:NR Mode of packet lengths 76 19:NR Range of packet lengths 77 19:NR Standard deviation of packet lengths 78 19:NR Robust standard deviation of packet lengths 79 19:NR Skewness of packet lengths 80 19:NR Excess of packet lengths 81 19:NR Minimum inter arrival time 82 19:NR Maximum inter arrival time 83 19:NR Mean inter arrival time 84 19:NR Lower quartile of inter arrival times 85 19:NR Median inter arrival times 86 19:NR Upper quartile of inter arrival times 87 19:NR Inter quartile distance of inter arrival times 88 19:NR Mode of inter arrival times 89 19:NR Range of inter arrival times 90 19:NR Standard deviation of inter arrival times 91 19:NR Robust standard deviation of inter arrival times 92 19:NR Skewness of inter arrival times 93 19:NR Excess of inter arrival times 94 8,25:R L2L3/L4/Payload( s. PACKETLENGTH in packetCapture.h) length and inter-arrival times for the N first packets 95 8,9,9,9,9:R Packetsize Inter Arrival Time histogram bins All you never wanted to know about statistics in a flow L2/3/4/7 configurable Packet Statistics • 20

  21. HOW TO find the needle in the flow stack? Have a break have a HEX & ¦ scripting!

  22. T2 Text Flow File: Basic plugins • A 1196278772.439355 1196279184.642073 412.202718 0x9B42 22 192.168.1.10 0x00000001 2119 68.3.4.5 0x800806034 80 6 00:0f:1f:cf:7c:45_00:00:0c:07:ac:0a_6387 http 6387 8272 464 5437587 0 4 15.494803 1.125660 -0.128590 -0.999829 1 87 128 128 0x00 0x42 0x0000 116 464 6231 4116 5437724 2253 63754 64831.988281 62501 65535 3342 2904 5713 0x18 0xF900 0x0000 0x03 0x00000000 0x0000 -1.0 1 1 1 ... B 1196278772.409312 1196279184.642073 412.232761 0x9B43 22 192.168.1.10 0x00000001 80 68.3.4.5 0x80080634 2119 6 00:d0:00:64:d0:00_00:0f:1f:cf:7c:45_8272 http 8272 6387 5437587 464 0 1380 20.066333 13190.574633 0.128590 0.999829 1 3 63 63 0x00 0x42 0x0000 8146 5440245 109 116 464 8104 5840 5840.000000 65535 0 0 0 0 0x18 0x1B00 0x0000 0x03 0x00000000 0x0000 -1.0 1 1 1 ... • 22

  23. T2 Binary Coding Status: • 2^0 0x0001 Flow Warning Flag: If A flow: Invert Flow, NOT client flow • 2^1 0x0002 Dump/flow: L3 Snaplength too short • 2^2 0x0004 Dump/flow: L2 header length too short • 2^3 0x0008 Dump/flow: L3 header length too short • 2^4 0x0010 Dump: Warning: IP Fragmentation Detected • 2^5 0x0020 Flow: ERROR: Severe Fragmentation Error • 2^6 0x0040 Flow: ERROR: Fragmentation Header Sequence Error • 2^7 0x0080 Flow ERROR: Fragmentation Pending at end of flow • 2^8 0x0100 Flow/Dump: Warning: VLAN(s) detected • 2^9 0x0200 Flow/Dump: Warning: MPLS unicast detected • 2^10 0x0400 Flow/Dump: Warning: MPLS multicast detected • 2^11 0x0800 Flow/Dump: Warning: L2TP detected • 2^12 0x1000 Flow/Dump: Warning: PPP detected • 2^13 0x2000 Flow/Dump: 0/1: IPv4/IPv6 detected • 2^14 0x4000 Flow/Dump: Warning: Land Attack detected • 2^15 0x8000 Flow/Dump: Warning: Time Jump So what is: 0x9B43 • 23

  24. T2 Flow Binary Coding: ipFlags • 2^0 0x0001 IP Options present, s. IP Options Type Bit field • 2^1 0x0002 IPID out of order • 2^2 0x0004 IPID rollover • 2^3 0x0008 Fragmentation: Below expected RFC minimum fragment size: 576 • 2^4 0x0010 Fragmentation: Fragments out of range (Possible tear drop attack) • 2^5 0x0020 Fragmentation: MF Flag • 2^6 0x0040 Fragmentation: DF Flag • 2^7 0x0080 Fragmentation: x Reserved flag bit from IP Header • 2^8 0x0100 Fragmentation: Unexpected position of fragment (distance) • 2^9 0x0200 Fragmentation: Unexpected sequence of fragment • 2^10 0x0400 L3 Checksum Error • 2^11 0x0800 L4 Checksum Error • 2^12 0x1000 SnapLength Warning: IP Packet truncated, L4 Checksums invalid • 2^13 0x2000 Packet Interdistance == 0 • 2^14 0x4000 Packet Interdistance < 0 • 2^15 0x8000 Internal State Bit for Interdistance assessment So what is: 0x1C21 • 24

  25. T2 Flow Binary Coding: tcpFlags 2^0 0x01 FIN No more data, finish connection 2^1 0x02 SYN Synchronize sequence numbers 2^2 0x04 RST Reset connection 2^3 0x08 PSH Push data 2^4 0x10 ACK Acknowledgement field value valid 2^5 0x20 URG Urgent pointer valid 2^6 0x40 ECE ECN-Echo 2^7 0x80 CWR Congestion Window Reduced flag is set 2^0 0x0001 Fin-Ack Flag 2^1 0x0002 Syn-Ack Flag 2^2 0x0004 Rst-Ack Flag 2^3 0x0008 Syn-Fin Flag, Scan or malicious packet 2^4 0x0010 Syn-Fin-Rst Flag, potential malicious scan packet or malicious channel 2^4 0x0020 Fin-Rst Flag, abnormal flow termination 2^5 0x0040 Null Flag, potential NULL scan packet, or malicious channel 2^6 0x0080 XMas Flag, potential Xmas scan packet, or malicious channel 2^8 0x0100 Due to packet loss, Sequence Number Retry, retransmit 2^9 0x0200 Sequence Number out of order 2^10 0x0400 Sequence mess in flow order due to pcappkt loss 2^11 0x0800 Warning: L4 Option field corrupt or not acquired 2^12 0x1000 Syn retransmission 2^13 0x2000 Ack number out of order 2^14 0x4000 Ack Packet loss, probably on the sniffing interface 2^15 0x8000 Internal state: TCP Window Size Machine So what is: 0x1B 0xC403 • 25

  26. T2 Flow Binary Coding: icmpFlags • Aggregated ICMP Type & Code bit Field • So what is: 0x00000100_0x0001 • 26

  27. T2 Packet Signal: Encrypted VoIP Mining PacketLength_Packet-Interdistance; … 1023_0.000000;758_0.030043;1380_0.110201;80_0.00000;369_0.000010;230_0.020029;1380_0.070101;80_0.000000;50_0.060086;1380_0.070101;80_0.090130; … Packet Length Post processing scripts: /tranalyzer/trunk/scripts time 27

  28. T2 Statistical Application / User profiling Packet length-Interdistance Statistics: Fingerprint PktLen_Packet-IAT_cnt_cntPktLen_cntIAT; … • 0_0_2322_6271_2396;0_2_82_6271_90;0_4_114_6271_114;0_6_138_6271_140;0_8_162_6271_164;0_10_157_6271_160;0_12_220_6271_224;0_14_217_6271_222;0_16_325_6271_325;0_18_373_6271_376;0_20_493_6271_498;0_22_340_6271_343;0_24_238_6271_238;0_26_283_6271_284;0_28_143_6271_143;0_30_114_6271_114;0_32_139_6271_140;0_34_175_6271_176;0_36_72_6271_73;0_38_25_6271_25;0_40_20_6271_20;0_41_12_6271_13;0_42_8_6271_8;0_43_6_6271_6;0_44_6_6271_6;0_45_4_6271_4;0_46_5_6271_5;0_47_9_6271_10;0_48_9_6271_9;0_49_6_6271_6;0_50_4_6271_4;0_51_4_6271_4;0_52_5_6271_5;0_53_3_6271_3;0_54_9_6271_9;0_55_7_6271_8;0_56_1_6271_1;0_57_4_6271_4;0_58_1_6271_1;0_59_3_6271_3;0_60_4_6271_4;0_61_4_6271_4;0_62_2_6271_2;0_63_1_6271_1;0_64_1_6271_1;0_65_1_6271_1;4_0_74_116_2396;4_2_8_116_90;4_6_2_116_140;4_8_2_116_164;4_10_3_116_160;4_12_4_116_224;4_14_5_116_222;4_18_3_116_376;4_20_5_116_498;4_22_3_116_343;4_26_1_116_284;4_32_1_116_140;4_34_1_116_176;4_36_1_116_73;4_41_1_116_13;4_47_1_116_10;4_55_1_116_8 ….. Post processing scripts: /tranalyzer/trunk/scripts Skype: Vulnerable against TM Attack • 28

  29. Some T3 Plugins • L7 Protocols: Mail, HTTP, etc • Routing: OSPF • DNS / DHCP • Full PCRE Regex • Signal Processing • Artificial Intelligence (RNN, Bayes, ESOM), nifty entropy shit • Connection Matrix, Centrality • IP Statistics: Host • Database • 29

  30. So what? Some Examples

  31. The one way TCP Flow problem • Symptom: on and off access problems • TCP flows established, unidirectional • T2 proofed: Reverse connection exists, not through firewall • Not communicated online mis-configuration of firewall Trampel OSPF

  32. FFT of some Packet Signals • Packet Length • time • 32

  33. Traffic Mining: Encrypted Content Guessing • SSH Command Guessing • IP Tunnel Content Profiling • Pitch based Classification • Encrypted Voip Guessing: CCC 2011 33

  34. TM Your OWN: Packet Length Signal See the features? Codec training SN Ping min l =3 Burschka (Fischkopp) Linux Dominic (Student) Windows

  35. Connection plugin: Social Behaviour • 35

  36. What is the Unknown? • 36

  37. HOW TO find Bad Guys? Day: 0.7% of all users 42% bandwidth, WTF? P2P Traffic P2P Traffic Average Users ??? Percentil User Percentil User Normal Traffic Normal Traffic 37

  38. HOW TO find Bad Guys? Night: Same guys @ night 3am, ... P2P Traffic Average Users Machines of WAREZ guys Percentil User Normal Traffic 38

  39. Layer3/4/whatever Visualization Graphviz --> Operational Picture in Bootcamp _flow.txt Your AWK script Graphviz: dotty • 39

  40. Layer3/4 Visualization Graphviz --> simple forensic Picture

  41. Network Classification Centrality Connection Matrix PCA Largest Eigenvector Plot / t • 41

  42. Network / Host Classification Centrality

  43. ipSOM Operational Picture: 13 Dim statistical T2 Flow parameters Now conceivable by human brain Bot Scanner DNS Zone Transfer 43

  44. Questions / Comments RFM and try me Join the development force Who wants Bootcamp? http://sourceforge.net/projects/tranalyzer/ http://tranalyzer.com http://sourceforge.net/projects/traviz Google: Dataming for Hackers stefan.burschka@ruag.com 44

More Related