Load Testing Using Perl

National Aeronautics & Space Administration Load Testing Using Perl • Homer Hummel • Jet Propulsion Laboratory, • California Institute of Technology • Homer.Hummel@jpl.nasa.gov • 2008-07-25

Presentation Overview • UnixLoadGen • Load Test Configuration, RSA Auth and Unix fork • Typical LoadGen Run Scenario • LoadGen directory structure • File formats • Perl scripts • Real-world experiences • General comparison with COTS • References

UnixLoadGen • A general framework for load testing application servers. • Written in Perl. • Uses “fork” feature of Unix. • Uses “RSA Authentication” feature of SSH. • UnixLoadGen distributed on website [TBD] • UnixLoadGen – hereinafter referred to as LoadGen.

Typical Load Test Configuration Generator 1 Test Server Generator 2 Control Workstation Generator X

Use of RSA Authentication Generator 1 tester Public Key Control Workstation Generator 2 tester tester Private Key Public Key Public Key Generator X tester Public Key

Control Workstation and Alternate Control Workstn Generator 1 tester tester Public Key Public Key Private Key Public Key Generator 2 tester Public Key Public Key Control Workstn (alt) tester Private Key Public Key Generator X tester Public Key Public Key

Use of Unix “fork” Generator Test Server startN.pl Y load_script load_script (1) load_script (2) load_script (Y)

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server load_script (1) load_script (2) Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: load_script (Y) Generator 2 load_script (1) load_script (2) load_script (Y) Gen1 cmd Gen2 cmd GenX cmd Generator X load_script (1) load_script (2) load_script (Y)

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: Generator 2 Gen1 cmd Gen2 cmd GenX cmd Generator X

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server scripts data files Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: Generator 2 Gen1 cmd Gen2 cmd GenX cmd Generator X

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server scripts data files Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: Generator 2 scripts data files Gen1 cmd Gen2 cmd GenX cmd Generator X

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server scripts data files Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: Generator 2 scripts data files Gen1 cmd Gen2 cmd GenX cmd Generator X scripts data files

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server load_script (1) load_script (2) Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: load_script (Y) Generator 2 Gen1 cmd Gen2 cmd GenX cmd Generator X

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server load_script (1) load_script (2) Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: load_script (Y) Generator 2 load_script (1) load_script (2) load_script (Y) Gen1 cmd Gen2 cmd GenX cmd Generator X

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server raw_results (1) raw_results (2) Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: raw_results (Y) Generator 2 raw_results (1) raw_results (2) raw_results (Y) Gen1 cmd Gen2 cmd GenX cmd Generator X raw_results (1) raw_results (2) raw_results (Y)

Typical LoadGen Run Scenario Generator 1 Control Workstn Test Server Edit scripts & data push_files.pl loadgen.pl process_results.pl Control file: Generator 2 Summary Test Report Gen1 cmd Gen2 cmd GenX cmd Generator X

LoadGen Directory Structure Home directory of local user account ‘tester’ control (on control workstation only) data test_type1 (e.g., dir) test_type2 (e.g., pop3) [etc.] raw_results test_type1 test_type2 [etc.] results (on control workstation only) scripts

Unique Input Data Lists names000 (file): names001 (file): … names199 (file): testuser1 testuser51 testuser9951 testuser2 testuser52 testuser9952 … … … testuser50 testuser100 testuser10000 list_of_names_params_5.list00 (unique param list example): -names data/dir/names000 -names data/dir/names001 -names data/dir/names002 -names data/dir/names003 -names data/dir/names004 list_of_names_params_5.list00 (one file per generator) list_of_names_params_5.list01 … list_of_names_params_5.list09

Control File Format(General) Generator_1 command_string Generator_2 command_string … Generator_X command_string Example: file jabber.ctl: gen-int-000 'export TERM=vt100; cd /home/tester/jabber/jabsimul/jab_simul; ./jab_simul > /tmp/jabber_output_gen-int-000.txt' gen-int-001 'export TERM=vt100; cd /home/tester/jabber/jabsimul/jab_simul; ./jab_simul > /tmp/jabber_output_gen-int-001.txt‘ … gen-int-019 'export TERM=vt100; cd /home/tester/jabber/jabsimul/jab_simul; ./jab_simul > /tmp/jabber_output_gen-int-019.txt'

Control File Format(Y-instances of cmd_str) Generator_1 startN.pl Y command_string Generator_2 startN.pl Y command_string … Generator_X startN.pl Y command_string

Control File Format(y-instances with unique params) Generator_1 startNupl.pl Y list_of_names_params_5.list00 cmd_string Generator_2 startNupl.pl Y list_of_names_params_5.list01 cmd_string … Generator_X startNupl.pl Y list_of_names_params_5.listxx cmd_string Example: file ldir_auth_mix_edg.ctl: gen-int-001 scripts/startNupl.pl 10 data/dir/list_of_names_params_10.list00 scripts/lmodattr_lnx.pl -h ldaptest –b dirtest -w xxx -a jplalias -v tester1 -c dirtest -r -iterations 200 gen-int-002 scripts/startNupl.pl 10 data/dir/list_of_names_params_10.list01 scripts/lmodattr_lnx.pl -h ldaptest -b dirtest -w xxx -a jplalias -v tester1 -c dirtest -r -iterations 200 … gen-int-010 scripts/startNupl.pl 10 data/dir/list_of_names_params_10.list09 scripts/lsgetentry_lnx.pl -h ldaptest -b dirtest -w xxx -p -c uid=celachi -iterations 200

Raw Results File Naming & Format Filename: test_run_id~servername~generator~log.pid File contents: start_time,stop_time start_time,stop_time … (one line for each iteration of load script transaction) Example: File: mix1~ldaptest~gen-int-005~log.12403 1212105348.215921,1212105349.307710 1212105349.307803,1212105351.421984 1212105351.422039,1212105353.107550 1212105353.107605,1212105356.026773 …

Summary Report File Example Server: ldaptest Load test: lsgetentry Test run: mix1 Date/time of beginning of test: 2008-05-29 16:55:50 Date/time of end of test: 2008-05-29 17:05:57 Test duration: Hours: 0 Minutes: 10 Seconds: 7 Values for minimum, average, maximum and standard deviation are in seconds. Min = 0.256 Ave = 2.894 Max = 6.293 Std Dev = 1.354 Total number of transactions = 5000 Concurrent Test Clients Users Iterations Errors gen-int-003 5 1000 0 gen-int-006 5 1000 0 gen-int-007 5 1000 0 gen-int-012 5 1000 0 gen-int-018 5 1000 0

push_files.pl snippets my $ctrl_c = 0; # initialize $ctrl_c flag $SIG{INT} = sub { $ctrl_c = 1 }; # set the $ctrl_c flag on interrupt open CONTROL, "<../control/$control_file" or die "Can't open $control_file: $!"; while (<CONTROL>) { next if ($_ =~ /^#/); # skip comment lines (# in column 1) ($mach, $remaining) = split /\s+/, $_, 2; push @machines, $mach; push @cmd_strings, $remaining; } close (CONTROL);

push_files.pl snippet my $localhome = $ENV{HOME}; foreach my $mach (@machines) { my $remotehome = `ssh $mach 'echo \$HOME'`; chomp $remotehome; if ($scripts_flag == 1) { print "Pushing scripts directory...\n"; `scp -r $localhome/scripts $mach:$remotehome`; } if ($data_flag == 1) { print "Pushing data directory...\n"; `scp -r $localhome/data/$test_types[$i] $mach:$remotehome/data`; } print "Push files completed on $mach.\n"; last if ($ctrl_c == 1); }

loadgen.pl snippet my $errors = 0; my @pid; foreach my $i (0..$#machines) { FORK: { if ($pid[$i] = fork) { # Parent process, child's pid in the array next } elsif (defined $pid[$i]) { # Child process print "Starting command string on $machines[$i]\n"; exec "ssh $machines[$i] $cmd_strings[$i] -test_run_id $test_run_id"; die "Exec failed after fork\n"; } elsif ($! =~ /EAGAIN/) { # EAGAIN is the supposedly recoverable fork error sleep 5; redo FORK; } else { die "Can't fork: $!\n"; } } sleep $ramp_up_delay; }

process_results.pl snippet my $rawdir = "$localhome/raw_results/$test_type"; # get all raw_results files for test_type if (-e $rawdir) { my @all_raw_results_files = `ls -1 $rawdir`; # -1 (one) option to # 'ls' cmd gives just file names } else { die "No raw results of test type: $test_type\n"; } # collect relevant raw results file names by test run identifier foreach my $file (@all_raw_results_files) { chomp $file; my @file_name_portions = split /~/, $file; if ($file_name_portions[0] =~ /$test_run_id/) { push @raw_results_files, $file; } }

load_script_template.pl (part 1) #!/usr/bin/perl use strict; use warnings; use Getopt::Long; use Sys::Hostname; use Time::HiRes; my $ctrl_c = 0; # initialize $ctrl_c flag $SIG{INT} = sub { $ctrl_c = 1 }; # set the $ctrl_c flag on interrupt my $iterations = 3; my $delay_iteration = 0; # delay between iterations in seconds my $test_run_id; # test run identifier string e.g. run01, aurnn14, or xyz my $servername; # servername (unix hostname) my $outdir = "raw_results/"; my @start_times = (); # start timestamps in floating point seconds my @stop_times = (); # stop timestamps in floating point seconds

load_script_template.pl (part 2) &GetOptions( "iterations:i" => \$iterations, "delay_iteration:i" => \$delay_iteration, # delay 'tween iterations "test_run_id:s" => \$test_run_id, # test run identifier "servername:s" => \$servername, # servername ); # if test_type subdir of raw_results does not exist, create it. # substitute "test_type" with the real one e.g. "dir" mkdir ($outdir . "test_type") unless -e ($outdir . "test_type"); $outdir .= "test_type/"; my $logfile = $outdir . $test_run_id . "~$servername~" . hostname() . "~log.$$"; # $logfile example: raw_results/dir/run01~devldap~gen-01~log.3257

load_script_template.pl (part 3) # main processing loop foreach my $iter (1..$iterations) { push @start_times, Time::HiRes::time; # save start timestamp # Perl code for client-server transaction goes here push @stop_times, Time::HiRes::time; # save stop timestamp last if ($ctrl_c eq 1); # exit loop on interrupt sleep $delay_iteration; # delay in seconds before starting next iter } # output raw timing results open LOG, ">$logfile" or die "Can't create $logfile: $!"; for my $i (0..$#start_times) { printf LOG ("%.6f,%.6f\n", $start_times[$i], $stop_times[$i]); } close (LOG); exit;

loadad.pl (part 1) #!/usr/bin/perl # load Active Directory server use strict; use warnings; use Getopt::Long; use Sys::Hostname; use Time::HiRes; use Authen::Simple::ActiveDirectory; my $ctrl_c = 0; # initialize $ctrl_c flag $SIG{INT} = sub { $ctrl_c = 1 }; # set the $ctrl_c flag on interrupt my $iterations = 3; my $delay_iteration = 0; # delay between iterations in seconds my $test_run; # test run identifier string e.g. run01, aurnn14, or xyz my $servername; # servername (unix hostname) my $domain; my $names;

loadad.pl (part 2) my $username; my $password = "xxxxxxxx"; my $outdir = "raw_results/"; my @start_times = (); # start timestamps in floating point seconds my @stop_times = (); # stop timestamps in floating point seconds &GetOptions( "iterations:i" => \$iterations, "delay_iteration:i" => \$delay_iteration, # delay 'tween iterations "test_run:s" => \$test_run, # test run identifier "servername:s" => \$servername, # servername "names:s" => \$names, # pathname of file of user names "domain:s" => \$domain, # Active Directory domain ); # if test_type subdir does not exist, create it. # substitute "test_type" with the real one e.g. "dir" mkdir ($outdir . "loadad") unless -e ($outdir . "loadad"); $outdir .= "loadad/"; my $logfile = $outdir . $test_run . "~$servername~" . hostname() . "~log.$$"; # $logfile example: raw_results/dir/run01~devldap~client01~log.3257 open NAMES, "<$names" or die "Can't open $names: $!";

loadad.pl (part 3) # main processing loop foreach my $iter (1..$iterations) { $username = <NAMES>; # read a line from file of user names if (!($username)) { # if end-of-file seek NAMES,0,0; # point to beginnning of file $username = <NAMES>; # and read the first line } chomp $username; # discard newline push @start_times, Time::HiRes::time; # save start timestamp my $ad = Authen::Simple::ActiveDirectory->new( host => $servername, principal => $domain, ); if ( $ad->authenticate( $username, $password ) ) { # print "Good authentication!\n"; # for debugging single instance } else { print "Bad authentication!\n"; } push @stop_times, Time::HiRes::time; # save stop timestamp last if ($ctrl_c eq 1); # exit loop on interrupt sleep $delay_iteration; # delay in seconds before starting next iter }

Some utility scripts in the distribution stop.pl psignal.pl check_generators.pl makelists.pl makelistoflists.pl servermon.pl swingbench.pl

Real-world Experience • Ability to integrate load scripts written in other languages. • World’s first load test of Kerberos authentication servers. • Driver for other simulators. • LoadGen used for LDAP Directory, Active Directory, pop3, imap4, Remedy, Kerberos and Jabber servers.

General Comparison with COTS

Some Relevant References UnixLoadGen User’s Guide, by Hummel (included in distribution) SSH The Secure Shell by Barrett & Silverman, publisher O’Reilly Linux System Programming by Love, publisher O’Reilly Programming Perl 3rd Ed. by Wall, Christiansen & Orwant, publisher O’Reilly

Acknowledgements CSC Management Perl Advisors Mike Gross Peter Scott Todd Lucas Patrick Ward Eric Gerritsen Thomas Berry Virinder Dhillon JPL Management Kevin Klenk cpan.org sponsors & Sarala Rajeshuni authors, specifically Henry Dillard Christian Hansen Luke Dahl

National Aeronautics & Space Administration Load Testing Using Perl • Homer Hummel • Jet Propulsion Laboratory, • California Institute of Technology • Homer.Hummel@jpl.nasa.gov • 2008-07-25

Load Testing Using Perl