530 likes | 692 Views
Test - Strategy and Implementation. By jonasbn <jonasbn@io.dk>. What this presentation will cover?. An outline of test methods How to design a test strategy from test methods How to implement it a test strategy using test methods Exemplified by all the mistakes humanly possible.
E N D
Test - Strategy and Implementation By jonasbn <jonasbn@io.dk>
What this presentation will cover? • An outline of test methods • How to design a test strategy from test methods • How to implement it a test strategy using test methods • Exemplified by all the mistakes humanly possible
What this presentation will not give you? • A solution to all your problems • A neat Solution::Simple for download from CPAN
What this presentation try to give you? • A new mindset when thinking about testing • A tool to make better tests • A tool to make the right tests • A tool to improve your development
Are YOU a Tester? “Hello my name is jonasbn – I’m a tester. In my daily work I work as a developer and that’s what it says on the paper, but I’m really a tester. Every time my colleagues turn the other way I write one and every time my manager gives me a task - all I think about is tests and how to test – often I work late so I can finish my tests without using valuable development time – Yes I’m a tester and I cannot help it.” - jonasbn, developer (tester)
What does test mean? • How do you define test?
My test definition • Tests can assure me that what I am coding works correctly • Tests are aiming at being automated • Test is a tool to help my development • Tests can be difficult to develop correctly
What kind of test methods do you know? • What test methods do you currently use? • Or have discarded? • What test methods have you used?
My list of Test Methods • Coverage Test • Smoke Test • Integration Test • Acceptance Test • System Execution Test • Regression Test • Stress Test • Performance Test • Unit Test
Coverage Test • Simple Philosophy • All lines should be run at least once • Testing all possible branches of the code • Time consuming • Large number of tests • Difficult way of testing • Tools to assist coverage test exist
Smoke Test • From the world of electronics • Power it up - does it break? • Not much detail • Crude way of testing
Integration Test • When things need to be assembled • Known from large projects • For dispersed development • When integration with 3rd. Party components
Acceptance Test • When the Client/Customer are to approve • Should not replace Performance Test • Should not replace automated testing • Should not be automated - this is a manual process
System Execution Test • From phase based development • When test is a phase • Focus is changed • Many aspects covered in other methods • Has serious flaws • Can be cut due to time constraints
Regression Test • When old things are revisited • Used to ensure backwards compability
Stress Test • When trying to break the system • Resembles smoke test • Also known as monkey test • Changing environments • Garbage Feeding
Performance Test • When speed is the need… • For benchmarking • Can be used to identify bottlenecks • History is important
Unit Test • The foundation of all testing • Can be used to implement almost all test methods • Simplicity • SMOP • Can be automated • Programmatic approach
Test Methods Summarized • Automatic and manual test methods • Black box and White box angles
Nothing is known apart from what is public Everything is known Introspections are possible Black/White Box Testing
Test Strategy Melchett: You look surprised, Blackadder. Edmund: I certainly am, sir. I didn't realise we had any battle plans. Melchett: Well, of course we have! How else do you think the battles are directed? Edmund: Our battles are directed, sir? Melchett: Well, of course they are, Blackadder -- directed according to the Grand Plan. Edmund: Would that be the plan to continue with total slaughter until everyone's dead except Field Marshal Haig, Lady Haig and their tortoise, Alan? Melchett: Great Scott! Even you know it! Guard! Guard! Bolt all the doors; hammer large pieces of crooked wood against all the windows! This security leak is far worse than we'd imagined!
Project Goals • We need to examine the project’s goals to make the right test strategy • The test strategy should support the project’s goals not work against them
Defining a strategy • We apply a selection the listed test methods to form a strategy • Which ones should we pick?
An example - “eXtreme Programming” • Unit Test during development • Integration Test when applicable • Acceptance Test and the end of each cycle
An example - “Waterfall/Phase-divided” • Unit Test during development • Smoke Test when building • Integration Test when partially done • System Execution Test when done • Acceptance Test
Other Strategies? • What strategy do YOU use?
Select the right strategy • Is your application a GUI? • Is your application performance critical? • Is quality important? • Is data correctness essential? • Is data processing essential? • Is backwards compability important?
Basic Unit Tests use strict; use Test::More tests => 6; use lib qw(lib); BEGIN { use_ok( 'Geneva::Product' ); } require_ok( 'Geneva::Product' ); my $p = Geneva::Product->new(); ok(! (ref $p), 'failing constructor test'); $p = Geneva::Product->new( p_customerRef => 'argle', ); ok((ref $p) eq 'Geneva::Product', 'succeeding constructor test'); ok($p->isa('Geneva'), 'Testing isa of Geneva'); ok($p->can('new'), 'Testing can Geneva::Product::new');
Unit Test Tactics • Should I go for black box or white box? • Can I trust the units
Advanced Unit Tests 1 sub move2done ($) { my $file = shift ; if ( -e "$mail_processed/$file") { unlink ("$mail_processed/$file"); } my $rv = rename("$mail_spool/$file","$mail_processed/$file") || warn "Unable to move file: $mail_spool/$file to $mail_processed/$file - $!\n"; return $rv; }
Advanced Unit Test 2 sub DSL0281ReceiveTDCRejected ($$) { my ($dbh, $filename) = @_; my ($dslid, $order_copper_id, $reason) = RejectedHandler($filename); if ($dslid and $order_copper_id) { eval { my $status = getstatus($dbh, $order_copper_id); if ($status ne DB_STATUS_RECEIVED) { die ("Status of order_copper_id $order_copper_id is $status, expected ".DB_STATUS_RECEIVED) ; } updateOrderCopper($dbh, $order_copper_id, DB_STATUS_REJECTED, undef, undef, \$reason); addtolog($dbh, DB_STATUS_REJECTED, $dslid, $order_copper_id); } ; if ($@) { warn $@ ; return "Error: $@" ; } else { return "OK" ; } } else { return "Error: DSL_ID or Order_Copper_Id could not be extracted from mail" ; } }
Acceptance Test • Acceptance Test is a manual process and should therefore not be automated • Unit Test reports can add some color to the delivery though underlining the fact that the product/system has been tested
Coverage Test • Devel::Coverage • Devel::Cover
Smoke Test • Perl smoke testing worth a study
Integration Test • You can easily test this if you take a Smoke Test approach
System Execution Test • This Test method is merely a phase, but it should be implemented using the test methods/strategy mentioned in this presentation
Regression Test • Regression Test will be necessary at some point • Save and use old tests, don’t throw anything away
Stress Test • Just bombard your application with calls from your test suite • Your test suite will cover “all” aspects of your application • Do it structured • Examine the output
Performance Test 1 • Benchmark is your friend • Save reports, historic data is essential • Examine output • And compare output
Performance Test 2 use SOAP::Lite; use Benchmark; my $t0 = new Benchmark; my $proxy = "http://somemachine/tempuri/Service1.asmx"; my $uri = "http://service.domain.dk/"; my $soap = SOAP::Lite ->uri($uri) ->proxy($proxy) ->on_action(sub {join '', @_}); my $method = SOAP::Data->name('getcustomer')->attr({xmlns => 'http://service.domain.dk/'}); my @params = (); push(@params, SOAP::Data->name('dslid' => 'dsl89305')->attr({'xsi:type' => 'xsd:string'})); my $rc = $soap->call($method => @params)->valueof("//getcustomerResponse"); my $t1 = new Benchmark; my $td = timediff($t1, $t0); print "the code took:",timestr($td),"\n";
Databases • Database encapsulations can be tricky to test • Data is often volatile and a baseline is difficult to optain
Solution 1: “The Sims” • The volatile aspect disappears • Data is known and controlled • The data can be misleading
Solution 2: “The Others” • Real-life data
“The Others Example” use strict; use Test::More tests => 2; use LWP::UserAgent; my $phoneno = '32545697'; my $uri = 'http://hostname.org/cgi-bin/proxy1.cgi'; $uri .= "?phoneno=$phoneno"; my $ua = LWP::UserAgent->new(env_proxy => 1, keep_alive => 1, timeout => 30, ); my $request = HTTP::Request->new('GET', $uri); my $response = $ua->request($request); my $content = $response->content; is($response->message, "OK", "Testing connection to URL: $uri"); like($content, qr/Amagerbro/i, "Testing content of response");
The Frame-work • Too many tests • Redundant code all over the place • Programmatic Practice • SMOP
The Frame-work 2 my %data_map = ( message => { type => 'xsd:string', value => ’kødpålæg', }, ); my $test_target = 'addtolog'; my $proxy = "http://webservice/Service1.asmx"; my $uri = "http://webservice.host.org/"; my @api_params = qw(message); my $t = Test::Accept::SOAP->new(); $t->test( 'proxy' => $proxy, 'uri' => $uri, 'test_target' => $test_target, 'api_params' => \@api_params, 'data_map' => \%data_map, );
The Frame-work 3 sub test ($%) { my ($self, %args) = @_; my $proxy = $args{'proxy'}; my $uri = $args{'uri'}; my $test_target = $args{'test_target'}; my $api_params = $args{'api_params'}; my $data_map = $args{'data_map'}; my $soap = $args{'soap'}; unless ($proxy and $uri and $test_target and $api_params and $data_map) { warn "Unable to perform test, insufficient parameters\n"; return; } my $method = $self->methodize($uri, $test_target); my @params = (); foreach my $p (@{$api_params}) { $self->parameterize(\@params,$data_map, $p); } my $rc = $self->exercize($soap, $test_target, $method, \@params, $debug); ok($rc, 'Checking we got something back from the webservice'); my $test_result = $test_target.'Result'; ok($rc->{$test_result}, 'Testing the returned result (shallow)'); return 1; }
Test Related Perl Modules • Test • Test::Simple • Test:More • Devel::Cover • Devel::Coverage