Feedback-Based Specification, Coding and Testing… …with JWalk

1. Feedback-Based Specification, Codingand Testing… …with JWalk Anthony J H Simons, Neil Griffiths and Christopher D Thomson

2. Overview • Lazy Systematic Unit Testing • JWalk testing tool (Simons) • The JWalkEditor tool • Integrated Java editor and JWalk (Griffiths) • Feedback-based methodology • Prototype, validate, specify, test • Evaluation of cost-effectivness • Testing effort, time, coverage (Thomson) http://www.dcs.shef.ac.uk/~ajhs/jwalk/

3. Motivation • State of the art in agile testing • Test-driven development is good, but… • …no specification to inform the selection of tests • …manual test-sets are fallible (missing, redundant cases) • …reusing saved tests for conformance testing is fallible –state partitions hide paths, faults (Simons, 2005) • Lazy systematic testing method: the insight • Complete testing requires a specification (even in XP!) • Infer an up-to-date specification from a code prototype • Let tools handle systematic test generation and coverage • Let the programmer focus on novel/unpredicted results

4. Lazy Systematic Unit Testing • Lazy Specification • late inference of a specification from evolving code • semi-automatic, by static and dynamic analysis of codewith limited user interaction • specification evolves in step with modified code • Systematic Testing • bounded exhaustive testing, up to the specification • emphasis on completeness, conformance, correctnessproperties after testing, repeatable test quality http://en.wikipedia.org/wiki/Lazy_systematic_unit_testing

5. JWalk Testing Tool • Lazy systematic unit testing for Java • static analysis - extracts the public API of a compiled Java class • protocol walking (code exploration) - executes all interleaved methods to a given path depth • algebraic testing (memory states) - validates all observations on all mutator-method sequences • state-based testing (high-level states) - validates all state-transitions (n-switch coverage) for inferred high-level states http://www.dcs.shef.ac.uk/~ajhs/jwalk/

6. JWalkEditor • Integration of JWalk with a Java-editor • Editing features • Java-sensitive • Integrated with JDK compiler tools – track exceptions to source • Testing features • Invokes JWalk (6 validation/testing modes) • Confirm/reject key results via dialogs • Browse test result-sets via tabbed panes

7. Snapshot – Editing a Stack Full build cycle, multiple sources Syntax sensitive text highlights Set JWalk test parameters

8. Snapshot –Testing a Stack Tabbed pane with test results for the Empty state, for all paths of depth 2 Colour-coded test sequences Colour-coded test outcomes

9. Dynamic Analysis of States • Memory states • generate all interleaved method paths to depth 1..n • prune sequences ending in observers from the active edges, preserving mutator sequences • distinguish observer/mutator methods by empirical low-level state comparison (extracted by reflection) • High-level states • generate all mutator-method paths (as above) • evaluate all state predicates, eg: isEmpty(), isFull() • seek states corresponding to the product of boolean outcomes: viz: {Default, Empty, Full, Full&Empty}

10. Test Result Prediction • Strong prediction • From known results, guarantee further outcomes in the same equivalence class • eg: test sequences containing observers in the prefix map onto a shorter sequence • target.push(e).size().top() == target.push(e).top() • Weak prediction • From known facts, guess further outcomes; an incorrect guess will be revealed in the next cycle • eg: methods with void type usually return no result, but may raise an exception • target.pop() predicted to have no result • target.pop().size() == -1 reveals an error

11. Feedback-based Methodology • Coding • The programmer prototypes a Java class in the editor • Validation • JWalk systematically explores method paths, providing useful instant feedback to the programmer • Specification • JWalk infers a specification, building a test oracle based on key test results confirmed by the programmer • Testing • JWalk tests the class to bounded exhaustive depths, based on confirmed and predicted test outcomes • JWalk uses state-based test generation algorithms

12. Example – Library Book public class LibraryBook { private String borrower; public LibraryBook(); public void issue(String); public void discharge(); public String getBorrower(); public Boolean isOnLoan();} • Validation • surprise: target.issue(“a”).issue(“b”).getBorrower() == “b” • violates business rules: fix code to raise an exception • Testing • all observations on chains of issue(), discharge() • n-switch cover on states {Default, OnLoan}

13. Extension – Reservable Book public class ReservableBook extends LibraryBook { private String requester; public ReservableBook(); public void reserve(String); public void cancel(); public String getRequester(); public Boolean isReserved();} • Validation • surprise: target.reserve(“a”).issue(“b”).getBorrower() == “b” • violates business rules: override issue() to refuse “b” here. • Testing • all obs. on chains of issue(), discharge(), reserve(), cancel() • n-switch cover on states {Default, OnLoan, Reserved, Reserved&OnLoan}

14. Evaluation • User Acceptance • programmers find JWalk habitable • they can concentrate on creative aspects (coding) while JWalk handles systematic aspects (validation, testing) • Cost of Confirmations • not so burdensome, since amortized over many test cycles • metric: measure amortized confirmations per test cycle • Comparison with JUnit • propose a common testing objective for manual and lazy systematic testing; evaluate coverage and testing effort • Eclipse+JUnit vs. JWalkEditor: given the task of testing the “transition cover + all equivalence partitions of inputs”

15. Test class a1 a2 a3 s1 s2 s3 LibBk con 3 5 7 0 0 5 LibBk pre 2 8 18 18 38 133 ResBk con 3 14 56 0 11 83 ResBk pre 6 27 89 36 241 1649 Amortized Interaction Costs • number of new confirmations, amortized over 6 test cycles • con = manual confirmations, > 25 test cases/minute • pre = JWalk’s predictions, eventually > 90% of test cases eg: state-test to depth 2, 241 predicted results eg: algebra-test to depth 2, 14 new confirmations

16. Comparison with • JUnit manual testing method • Manual test creation takes skill, time and effort (eg: ~20 min to develop manual cases for ReservableBook) • The programmer missed certain corner-cases • eg: target.discharge().discharge() - a nullop? • The programmer redundantly tested some properties • eg: assertTrue(target != null) - multiple times • The state coverage for LibraryBook was incomplete, due to the programmer missing hard-to-see cases • The saved tests were not reusable for ReservableBook, for which all-new tests were written to test new interleavings

17. Advantages of JWalk • JWalk lazy systematic testing • JWalk automates test case selection -relieves the programmer of the burdenof thinking up the right test cases! • Each test case is guaranteed to test a unique property • Interactive test result confirmation is very fast (eg: ~80 sec in total for 36 unique test cases in ReservableBook) • All states and transitions covered, including nullops, to the chosen depth • The test oracle created for LibraryBook formed the basis for the new oracle for ReservableBook, but… • JWalkpresented only those sequences involving new methods, and all interleavings with inherited methods

18. Test class T TE TR Adeq time min.sec LibBk manual 31 9 22 90% 11.00 ResBk manual 104 21 83 53% 20.00 LibBk jwalk 10 10 0 100% 0.30 ResBk jwalk 36 36 0 90% 0.46 Speed and Adequacy of Testing • Test goal: transition cover + equiv. partitions of inputs • manual testing expensive, redundant and incomplete • JWalk testing very efficient, close to complete eg: JWalk achieved 100% test coverage eg: wrote 104 tests, 21 were effective and 83 not!

19. Conclusion • Performance of JWalk testing • clearly outperformed manual testing • coverage based on all states and transitions • input equivalence partitions are not yet handled • Performance of JWalkEditor • unexpected gain: automatic validation of prototype code • c.f. Alloy’s model checking from a partial specification • Moral for testing • just automatically executing saved tests is not so great • need systematic test generation tools to get coverage • automate the parts that humans get wrong!

20. Any Questions? http://www.dcs.shef.ac.uk/~ajhs/jwalk/

21. Bibliography • A J H Simons, JWalk: a tool for lazy systematic testing of Java classes by introspection and user interaction, Automated Software Engineering, 14 (4), December, ed. B. Nuseibeh, (Springer, USA, 2007), 369-418. SpringerLink: DOI 10.1007/s10515-007-0015-3, 8 September, 2007. Final draft version also deposited with White Rose Research Online. • A J H Simons and C D Thomson, Lazy systematic unit testing: JWalk versus JUnit, Proc 2nd. Testing in Academia and Industry Conference - Practice and Research Techniques, 22-24 September, eds. P McMinn and M Harman, (Cumberland Lodge, Windsor Great Park: IEEE, 2007), 138. See also the A1 poster presenting this result. • A J H Simons and C D Thomson, Benchmarking effectiveness for object-oriented unit testing, Proc 1st. Software Testing Benchmark Workshop, 9-11 April, eds. M Roper and W M L Holcombe, (Lillehammer: ICST/IEEE, 2008). • A J H Simons, N Griffiths and C D Thomson, Feedback-based specification, coding and testing with JWalk, Proc 3rd. Testing in Academia and Industry Conference - Practice and Research Techniques, 29-31 August, eds. L. Bottacci and G. M. Kapfhammer and M. Roper, (Cumberland Lodge, Windsor Great Park: IEEE, 2008), to appear. • A J H Simons, A theory of regression testing for behaviourally compatible object types, rev. and ext., Software Testing, Verification and Reliability, 16 (3), UKTest 2005 Special Issue, September, eds. M Woodward, P. McMinn, M. Holcombe, R. Hierons (London: John Wiley, 2006), 133-156. • A J H Simons, Testing with guarantees and the failure of regression testing in eXtreme Programming, Proc. 6th Int. Conf. on eXtreme Programming and Agile Processes in Software Engineering (XP 2005), eds. H Baumeister et al., Lecture Notes in Computer Science, 3556, (Berlin: Springer Verlag, 2005), 118-126. • Wikipedia entry for JWalk • Wikipedia entry for Lazy Systematic Unit Testing