QuickCheck@Neilfest (or what I learned from Neil about software testing)

QuickCheck@Neilfest(or what I learned from Neil about software testing) John Hughes Chalmers University/Quviq AB

Company founded May 2006 • Tools for testing software • Profitable in the first year! • Customersinclude… Soon TBA

QuickCheck • Tests software automaticallyagainst a formal specification prop_reverse() -> ?FORALL({Xs,Ys}, {list(int()),list(int())}, reverse(Xs++Ys) == reverse(Xs)++reverse(Ys)).

Testing… 19> eqc:quickcheck(ex:prop_reverse()). .........Failed! After 10 tests. {[2],[3,2]} Shrinking....(4 times) {[0],[1]} false

QuickCheck • Tests software automaticallyagainst a formal specification prop_reverse() -> ?FORALL({Xs,Ys}, {list(int()),list(int())}, reverse(Xs++Ys) == reverse(Xs)++reverse(Ys)).

QuickCheck in Brief Specification Test case Test outcome Generate Execute Simplify Minimal counter-example e.g. {[0],[1]}

A MoreRealistic Test Case • A test case for the Erlang process registry test_register() -> Pid = spawn(), register(name,Pid), Pid2 = whereis(name), assert(Pid==Pid2), unregister(name). Spawn a new process (set up test data) Register it—a side-effect Inspect the results Did the test pass? Restore the state, ready for next test

Industrial Test Cases • A sequence of • Calls to an API under test • Checks on the results • Not much like {[0],[1]}… • Howcanweconvert this kind of test caseinto a logicalproperty?

QuickCheck in Industry Specification Test case: a program Test outcome Generate Execute Simplify Minimal program provoking a fault

Programs as Data Objects • Why? • So failing test casescan be printed • So failing test casescan be repeated • So failing test casescan be simplified

Can wemix generation and execution? • Randomlygenerateand performeachcall • Save a test case as a list of functions and actual parameters? • NO!!! • Whenwerepeat the test, Pid has a different value • Wemustuse a symbolic variable! test_register() -> Pid = spawn(), register(name,Pid),…

Test Case Language • Do weneedconditionals? • Tests should be deterministic • Check the right branch, don’t test • Usersneedsimplicity—straight-linecode is simple, and suffices X = foo(…), casep(X) of true -> baz(X); false -> bar(X) end X = foo(…), assert not(p(X)), bar(X)

Generating Test Cases • Howdoweknow • Whichcommands are valid at eachpoint? • What test data is available at eachpoint? • Whatresults are expected? • Track a test casestate • Check preconditionsbeforegeneratingeachcommand • Store available data in the state • Check postconditionswrt test casestate

Process Registry State Available process ids typestate() = record pids::list(pid()), regs::list({atom(),pid()}) end. Currentlyregisteredprocesses

Process Registry State • Two-level programs! • Static = test case generation time • Dynamic = test execution time Available process ids typestate() = record pids::list(symbolic(pid())), regs::list({atom(),symbolic(pid())}) end. Currentlyregisteredprocesses

Two-level State Machines • Preconditions • Checkedduring generation, purelystatic • Postconditions • Checkedduringexecution, purelydynamic • Nextstatefunction • Used at bothtimes, two-level • Command generator • Usedduring generation, two-level

Process Registry • spawn() • Adds a (dynamic) pid to the state • unregister(Name) • Pre:Name must be registered • RemovesName from the state • register(Name,Pid) • Adds {Name,Pid} to the state • Post:exceptionifName or Pidalreadyregistered

A Failing Test Case [{set,{var,2},{call,…,spawn,[]}}, {set,{var,3},{call,…,register,[a,{var,2}]}}, {set,{var,6},{call,…,register,[b,{var,2}]}}, {set,{var,8},{call,…,spawn,[]}}, {set,{var,9},{call,…,register,[b,{var,8}]}}] V2=spawn(), register(a,V2), register(b,V2), V8=spawn(), register(b,V8) Code Spec Inconsistency!

A BuggySpec • spawn() • Adds a (dynamic) pid to the state • unregister(Name) • Pre: must be registered • RemovesName from the state • register(Name,Pid) • Adds {Name,Pid} to the state • Post:exceptionifName or Pidalreadyregistered …providedthere is no exception!

”The Trick” • Whatifweknow part of the structure? • Check the known part in the postcondition • Add the known part statically to the state • ”The Trick” is as useful as ever! Added to the state Purelydynamic [{set,{var,2},{call,…,spawn,[]}}, …]

Test Case Generator Two-levellanguages are helping to findthornybugs in industrial systems! RandomizedGenerating Extension =

Best Bug! • In Ericsson’s Media Proxy (Multimedia IP-telephonyproduct, SIP) • A seriousbug • Obtained from a simple specification, by simplifying 160-command sequence! Add Add Sub Add Sub Add Sub

QuickCheck@Neilfest (or what I learned from Neil about software testing)