Advanced Scripting and Command Line Usage with tshark and Related Utilities March 31 st , 2008

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Advanced Scripting and Command Line Usage with tshark and Related Utilities March 31st, 2008 Sake Blok Research & Development Engineer @ ion-ip sake@euronet.nl / sake.blok@ionip.com SHARKFEST '08 Foothill College March 31 - April 2, 2008

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Agenda • Part I : Why and how to use scripting? • GUI versus CLI • Collect, Filter, Transform, Present, Validate • How? • Part II : Useful CLI Tools • Wireshark CLI tools • Other CLI tools • Part III : Examples • CLI command piping • Scripting • Contest !?! • Questions?

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 GUI versus CLI • graphical USER interface vs COMMAND line interface • GUI -> powerful “fixed” functionality • CLI combined with other tools -> flexibility • CLI -> automate repetitive tasks • CLI not only when GUI is unavailable

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 5 steps of network analysis • Collect raw data • Filter raw data • Transform data into information • Present Information • Validate information

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 How? • What information do I need? • visualize your output • What (raw) data sources do I have? • Know the output formats of your data sources • What tools are available? • What can they do, browse through manpages for unknown options • Practice & Experiment & be creative 

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Part II : Tools • Wireshark CLI tools • tshark • dumpcap • editcap • mergecap • capinfos • Other CLI tools • |, >, for … do … done, `<command>` • cut, sort, uniq, tr • sed, awk • scripting (sh/perl/python/…)

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 tshark (1) • CLI version of wireshark • Similar to tcpdump, but statefull • uses dumpcap as capture engine • standard options: -D, -i, -c, -n, -l, -f, -R, -s, -w, -r • name resolving (-n) • time stamps (-t<format>) • decode as (-d tcp.port==8080,http) • preferences (-o <pref>:<value>)

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 tshark (2) • output formats (-V -T <format>) • summary, use of column prefs • Verbose (-V), hex dump (-x) • PDML (-T pdml) • fields (-T fields -E <sep> -e <field1> -e <field2> …) • statistics (-z …) • protocol hierarchy (-qz io,phs) • conversations (-qz conv,eth , -qz conv,tcp) • i/o statistics (-qz io,stat,10,ip,icmp,udp,tcp)

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 dumpcap • used by (wire|t)shark • can be used separately • options similar to tshark • stateless so traces can run forever • fast! only network->disk • ring buffers extremely useful • dumpcap -i 5 -s0 -b filesize:16384 -files:1024 -w ring.cap

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 capinfos • display summary of a tracefile • all info vs specific info • $ capinfos sharkfest-1.cap • File name: sharkfest-1.cap • File type: Wireshark/tcpdump/... - libpcap • File encapsulation: Ethernet • Number of packets: 3973 • File size: 1431813 bytes • Data size: 1368221 bytes • Capture duration: 1299.436650 seconds • Start time: Thu Jan 17 11:37:16 2008 • End time: Thu Jan 17 11:58:55 2008 • Data rate: 1052.93 bytes/s • Data rate: 8423.47 bits/s • Average packet size: 344.38 bytes • $ capinfos -ae sharkfest-*.cap • File name: sharkfest-1.cap • Start time: Thu Jan 17 11:37:16 2008 • End time: Thu Jan 17 11:58:55 2008 • File name: sharkfest-2.cap • Start time: Thu Jan 17 11:39:27 2008 • End time: Thu Jan 17 12:02:52 2008

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 editcap use ‘editcap –h’ to see all options • used to manipulate a capture file • select frame ranges or time ranges • editcap -r sharkfest-1.cap tmp.cap 1-1000 2001-3000 • editcap -A "2008-01-17 11:40:00" -B "2008-01-17 11:49:59" sharkfest-1.cap tmp.cap • split file in chunks • editcap -c 1000 sharkfest-1.cap tmp.cap • change snaplen, time • editcap -s 96 sharkfest-1.cap tmp.cap • editcap -t -3600 sharkfest-1.cap tmp.cap

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 mergecap use ‘mergecap –h’ to see all options • used to merge captures • based on timestamps • mergecap -w out.cap in-1.cap in-2.cap • or just append each file • mergecap -a -w out.cap in-1.cap in-2.cap

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Part II : Tools • Wireshark CLI tools • tshark • dumpcap • editcap • mergecap • capinfos • Other CLI tools • |, >, for … do … done, `<command>` • cut, sort, uniq, tr • sed, awk • scripting (sh/perl/python/…)

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 |, >, for … do … done, `<command>` use ‘man bash’ for more info • Command piping with | • ls -1t | head • Output redirection with > • ls -1t | head > newfiles.txt • Looping with for … do … done • for word in ‘one’ ‘two’ ‘three’; do echo $word; done • Command evaluation with backtics (``) • for file in `ls -1t | head`; do echo $file; head -1 $file;echo "";done > firstlines.txt • Variable assignments • newfile=`echo ${file}.bak`

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 cut, sort, uniq, tr • cut • -c (by position) cut -c1-10 /etc/passwd • -f [-d ‘<delimiter>’] (by field) cut -d ':‘ -f1 /etc/passwd • sort • with no option sort names.txt • -r (reverse sorting) sort -r names.txt • -n (numerical sort) du -ks * | sort -rn | head • uniq • with no option sort names.txt | uniq • -d (only ‘doubles’) sort names.txt | uniq -d • -c (show count) sort names.txt | uniq -c • tr • “ ” “_” echo “one two” | tr “ “ “_” • -d “\015” cat dosfile.txt | tr –d “\015” > unixfile.txt

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 sed, awk • both powerful scripting languages • sed - stream editor for filtering and transforming text • -e 's/<deleteme>//' • -e 's/<replaceme>/<withthis>/' • -e 's/^.*$<keepme>$.*$<andme>$.*$/\1 \2/‘ • awk - pattern scanning and processing language • netstat -an | awk '$1=="TCP" {print $4}' | sort | uniq –c • … | awk '{printf(“ %s tcp.port==%s",sep,$1);sep="||"}'

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 scripting • parsing output when command piping is not enough • automate execution of tshark/dumpcap/mergecap etc • anything you can think of  • use your own favorite language (sh/perl/python/etc)

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Part III : Examples • Using command piping • Counting http response codes • Top 10 URL's • All sessions with session-cookie XXXX • Using scripting • All sessions for user XXXX (shell script) • Synchonize time between trace files (perl script)

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Test setup used to create capture files

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 1:counting http response codes (1) • Problem I need an overview of http response codes • Output table with http response codes & counts • 10 200 • 4 302 • 5 404 • Input Existing capture file with http traffic • Steps print response codes • make table

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 1:counting http response codes (2) • Command: tshark -o "tcp.desegment_tcp_streams:TRUE" -r sharkfest-1.cap -R "http.response"-T fields –e http.response.code |sort | uniq –c • New tricks learned: • -o <pref>:<value> • -T fields –e <field> • | sort | uniq -c

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 2:Top 10 requested URL's (1) • Problem I need a list of all URL’s that have been visited • Output Sorted list with requested URL’s and count • 13 http://www.shop.com/index.html • 10 http://www.shop.com/images/logo.gif • Input Existing capture file with http traffic • Steps Print http.host and http.request.uri • Strip everything after “?” • Combine host + uri and format into normal URL • count url’s • make top 10

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 3:All sessions with cookie XXXX (1) • Problem I know in which “session” a problem exists, but I need all data from that session to work it out • Output New capture file with whole tcp sessions that contain cookie PHPSESSID=c0bb9d04cebbc765bc9bc366f663fcaf • Input Existing capture file with http traffic • Steps extract port numbers based on cookie • create new filter based on port numbers • filter tcp sessions to a new capture file

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 3:All sessions with cookie XXXX (2) • Command: tshark -r sharkfest-1.cap -w cookie.cap -R `tshark -r sharkfest-1.cap -T fields -e tcp.srcport -R "http.request and http.cookie contains \"PHPSESSID=c0bb9d04cebbc765bc9bc366f663fcaf\"" | awk '{printf(“ %s tcp.port==%s",sep,$1);sep="||"}‘ ` • New tricks learned: • tshark -R `<other command that generated filter>` • awk '{printf(“ %s tcp.port==%s",sep,$1);sep="||"}'

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 4:All sessions for user XXXX (1) • Problem A particular user has multiple sessions and I need to see all sessions from that user • Output New capture file with all data for user xxxx • Input Existing capture file with http data • Steps link session cookies to user • extract session cookies • create new capture file per session cookie • merge files

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 4:All sessions for user XXXX (2) • New tricks learned: • for … do … done • <var>=ècho …| …` • cut -d <FS> -f <x> • tr –d “\015” • mergecap -w <outfile> <infile1> <infile2> … • `ls -1 tmp_*.cap` • Script: • $ cat example6.sh • #!/usr/bin/bash • file=$1 • user=$2 • for cookie in `tshark -r $file -R "http.request and http contains $user" -T fields -e http.cookie | tr " " "_"` • do • sid=ècho $cookie | cut -d '_' -f 2 | tr -d "\015"` • tmpfile="tmp_ècho $sid | cut -d '=' -f 2`.cap" • echo "Processing session cookie $sid to $tmpfile" • tshark -r $file -w $tmpfile -R `tshark -r $file "http.request and http.cookie contains \"$sid\"" -T fields -e tcp.srcport | awk '{printf("%stcp.port==%s",sep,$1);sep="||"}'` • done • mergecap -w $user.cap `ls -1 tmp_*.cap` • rm `ls -1 tmp_*.cap`

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 5:Synchronize time between trace files (1) • Problem In order to analyse a problem, two traces at different locations need to be compared, but the timestamps are different • Output New file with “synchronized” timestamps • Input Two capture files (with icmp packets) • Steps Make sure to ping between the capture host • Match icmp packets in both files • Calculate min and max difference in time • Create new file with corrected timestamps

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Example 5:Synchronize time between trace files (2) • Script: • $ cat example7.pl • #!/usr/bin/perl • $file1 = shift || die; • $file2 = shift || die; • $file3 = shift; • $tshark = '/cygdrive/c/Program\ Files/Wireshark/tshark.exe'; • $editcap = '/cygdrive/c/Program\ Files/Wireshark/editcap.exe'; • $options = "-T fields -e frame.time -e icmp.type -e icmp.ident -e icmp.seq -R icmp"; • $cmd1 = $tshark . " -r $file1 " . $options . " 2>/dev/null |"; • $cmd2 = $tshark . " -r $file2 " . $options . " 2>/dev/null |"; • $t1 = \%times1; • $t2 = \%times2; • sub readline { • my ($p,$line) = @_; • chomp $line; • $line =~ s/\015//; • ($time,$type,$ident,$seq) = split '\t',$line; • $id = $ident . "-" . sprintf "%05d",$seq; • $seq{$id} = 1; • if( $type == 8 ) { • $p->{'req'}{$id} = &conv_time($time); • } else { • $p->{'resp'}{$id} = &conv_time($time); • } • } • sub conv_time { • my ($str) = @_; • ($dummy,$h,$m,$s) = ($str =~ /^(.*) (\d\d):(\d\d):(\d\d\.\d+)$/); • return 3600 * $h + 60 * $m + $s; • } • open(F, $cmd1); • while(<F>) { • &readline($t1, $_); • } • close(F); • open(F, $cmd2); • while(<F>) { • &readline($t2, $_); • } • close(F); • foreach $seq (sort keys %seq) { • next unless defined $times1{'req'}{$seq}; • next unless defined $times1{'resp'}{$seq}; • next unless defined $times2{'req'}{$seq}; • next unless defined $times2{'resp'}{$seq}; • $reqdiff= $times2{'req'}{$seq} - $times1{'req'}{$seq}; • $respdiff= $times2{'resp'}{$seq} - $times1{'resp'}{$seq}; • printf "%s : %9.3f %9.3f %9.3f %9.3f -> %15.9f %15.9f\n", $seq, • $times1{'req'}{$seq}, • $times2{'req'}{$seq}, • $times2{'resp'}{$seq}, • $times1{'resp'}{$seq}, • $reqdiff, $respdiff; • if( !defined $mindiff ) { • $mindiff = $reqdiff; • $maxdiff = $respdiff; • } else { • $mindiff = $reqdiff unless $mindiff > $reqdiff; • $maxdiff = $respdiff unless $maxdiff < $respdiff; • } • } • $diff = -int( 500000*($mindiff + $maxdiff) ) / 1000000; • printf "\n"; • printf "Minimum difference found : %15.9f\n", $mindiff; • printf "Maximum difference found : %15.9f\n", $maxdiff; • printf "\n"; • if( abs($diff)<0.000001 ) { • printf "File \"%s\" and \"%s\" are already time synchronized.\n",$file1, $file2; • } else { • printf "File \"%s\" needs to be adjusted %f seconds to match file \"%s\"\n", • $file2, $diff, $file1; • if( $file3 eq "" ) { • printf "\nExecute \"editcap -t %f %s <outfile>\" to create a synchronized\nversion of file \"%s\"\n", $diff, $file2, $file2; • } else { • $cmd = $editcap . " -t $diff " . $file2 . " $file3"; • system $cmd; • printf "\nExecuted \"editcap -t %f %s %s\" to create a synchronized\nversion of file \"%s\"\n", $diff, $file2, $file3, $file2; • } • }

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Summary • tshark, dumpcap, capinfos, editcap, mergecap • tshark+scripting can complement GUI • use little building blocks and combine them • Hopefully you are triggered to experiment  See http://www.euronet.nl/~sake/sharkfest08/ for the presentation, used trace files, example-scripts and notes of this session.

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Contest • Create a one-liner that creates one capture file for each pop3-user with the name <username>.capThese files should contain all pop3 traffic for the specific user and nothing else. • Use file sharkfest-2.cap (available at http://www.euronet.nl/~sake/sharkfest08/) • Send your one-liner to me at sake@euronet.nl today before 11:59 PM. I will draw 2 winners from the one-liners that work as stated above. I will e-mail back my solution and the list of winners on Tuesday. • Prices can be collected by the winners on Wednesday, please look me up at lunch 

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Questions? • ?

Advanced Scripting and Command Line Usage with tshark and Related Utilities March 31 st , 2008