1 / 21

Feedback-Based Specification, Coding and Testing… …with JWalk

Feedback-Based Specification, Coding and Testing… …with JWalk. Anthony J H Simons, Neil Griffiths and Christopher D Thomson. Overview. Lazy Systematic Unit Testing JWalk testing tool (Simons) The JWalkEditor tool Integrated Java editor and JWalk (Griffiths)

val
Download Presentation

Feedback-Based Specification, Coding and Testing… …with JWalk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Feedback-Based Specification, Codingand Testing… …with JWalk Anthony J H Simons, Neil Griffiths and Christopher D Thomson

  2. Overview • Lazy Systematic Unit Testing • JWalk testing tool (Simons) • The JWalkEditor tool • Integrated Java editor and JWalk (Griffiths) • Feedback-based methodology • Prototype, validate, specify, test • Evaluation of cost-effectivness • Testing effort, time, coverage (Thomson) http://www.dcs.shef.ac.uk/~ajhs/jwalk/

  3. Motivation • State of the art in agile testing • Test-driven development is good, but… • …no specification to inform the selection of tests • …manual test-sets are fallible (missing, redundant cases) • …reusing saved tests for conformance testing is fallible –state partitions hide paths, faults (Simons, 2005) • Lazy systematic testing method: the insight • Complete testing requires a specification (even in XP!) • Infer an up-to-date specification from a code prototype • Let tools handle systematic test generation and coverage • Let the programmer focus on novel/unpredicted results

  4. Lazy Systematic Unit Testing • Lazy Specification • late inference of a specification from evolving code • semi-automatic, by static and dynamic analysis of codewith limited user interaction • specification evolves in step with modified code • Systematic Testing • bounded exhaustive testing, up to the specification • emphasis on completeness, conformance, correctnessproperties after testing, repeatable test quality http://en.wikipedia.org/wiki/Lazy_systematic_unit_testing

  5. JWalk Testing Tool • Lazy systematic unit testing for Java • static analysis - extracts the public API of a compiled Java class • protocol walking (code exploration) - executes all interleaved methods to a given path depth • algebraic testing (memory states) - validates all observations on all mutator-method sequences • state-based testing (high-level states) - validates all state-transitions (n-switch coverage) for inferred high-level states http://www.dcs.shef.ac.uk/~ajhs/jwalk/

  6. JWalkEditor • Integration of JWalk with a Java-editor • Editing features • Java-sensitive • Integrated with JDK compiler tools – track exceptions to source • Testing features • Invokes JWalk (6 validation/testing modes) • Confirm/reject key results via dialogs • Browse test result-sets via tabbed panes

  7. Snapshot – Editing a Stack Full build cycle, multiple sources Syntax sensitive text highlights Set JWalk test parameters

  8. Snapshot –Testing a Stack Tabbed pane with test results for the Empty state, for all paths of depth 2 Colour-coded test sequences Colour-coded test outcomes

  9. Dynamic Analysis of States • Memory states • generate all interleaved method paths to depth 1..n • prune sequences ending in observers from the active edges, preserving mutator sequences • distinguish observer/mutator methods by empirical low-level state comparison (extracted by reflection) • High-level states • generate all mutator-method paths (as above) • evaluate all state predicates, eg: isEmpty(), isFull() • seek states corresponding to the product of boolean outcomes: viz: {Default, Empty, Full, Full&Empty}

  10. Test Result Prediction • Strong prediction • From known results, guarantee further outcomes in the same equivalence class • eg: test sequences containing observers in the prefix map onto a shorter sequence • target.push(e).size().top() == target.push(e).top() • Weak prediction • From known facts, guess further outcomes; an incorrect guess will be revealed in the next cycle • eg: methods with void type usually return no result, but may raise an exception • target.pop() predicted to have no result • target.pop().size() == -1 reveals an error

  11. Feedback-based Methodology • Coding • The programmer prototypes a Java class in the editor • Validation • JWalk systematically explores method paths, providing useful instant feedback to the programmer • Specification • JWalk infers a specification, building a test oracle based on key test results confirmed by the programmer • Testing • JWalk tests the class to bounded exhaustive depths, based on confirmed and predicted test outcomes • JWalk uses state-based test generation algorithms

  12. Example – Library Book public class LibraryBook { private String borrower; public LibraryBook(); public void issue(String); public void discharge(); public String getBorrower(); public Boolean isOnLoan();} • Validation • surprise: target.issue(“a”).issue(“b”).getBorrower() == “b” • violates business rules: fix code to raise an exception • Testing • all observations on chains of issue(), discharge() • n-switch cover on states {Default, OnLoan}

  13. Extension – Reservable Book public class ReservableBook extends LibraryBook { private String requester; public ReservableBook(); public void reserve(String); public void cancel(); public String getRequester(); public Boolean isReserved();} • Validation • surprise: target.reserve(“a”).issue(“b”).getBorrower() == “b” • violates business rules: override issue() to refuse “b” here. • Testing • all obs. on chains of issue(), discharge(), reserve(), cancel() • n-switch cover on states {Default, OnLoan, Reserved, Reserved&OnLoan}

  14. Evaluation • User Acceptance • programmers find JWalk habitable • they can concentrate on creative aspects (coding) while JWalk handles systematic aspects (validation, testing) • Cost of Confirmations • not so burdensome, since amortized over many test cycles • metric: measure amortized confirmations per test cycle • Comparison with JUnit • propose a common testing objective for manual and lazy systematic testing; evaluate coverage and testing effort • Eclipse+JUnit vs. JWalkEditor: given the task of testing the “transition cover + all equivalence partitions of inputs”

  15. Test class a1 a2 a3 s1 s2 s3 LibBk con 3 5 7 0 0 5 LibBk pre 2 8 18 18 38 133 ResBk con 3 14 56 0 11 83 ResBk pre 6 27 89 36 241 1649 Amortized Interaction Costs • number of new confirmations, amortized over 6 test cycles • con = manual confirmations, > 25 test cases/minute • pre = JWalk’s predictions, eventually > 90% of test cases eg: state-test to depth 2, 241 predicted results eg: algebra-test to depth 2, 14 new confirmations

  16. Comparison with • JUnit manual testing method • Manual test creation takes skill, time and effort (eg: ~20 min to develop manual cases for ReservableBook) • The programmer missed certain corner-cases • eg: target.discharge().discharge() - a nullop? • The programmer redundantly tested some properties • eg: assertTrue(target != null) - multiple times • The state coverage for LibraryBook was incomplete, due to the programmer missing hard-to-see cases • The saved tests were not reusable for ReservableBook, for which all-new tests were written to test new interleavings

  17. Advantages of JWalk • JWalk lazy systematic testing • JWalk automates test case selection -relieves the programmer of the burdenof thinking up the right test cases! • Each test case is guaranteed to test a unique property • Interactive test result confirmation is very fast (eg: ~80 sec in total for 36 unique test cases in ReservableBook) • All states and transitions covered, including nullops, to the chosen depth • The test oracle created for LibraryBook formed the basis for the new oracle for ReservableBook, but… • JWalkpresented only those sequences involving new methods, and all interleavings with inherited methods

  18. Test class T TE TR Adeq time min.sec LibBk manual 31 9 22 90% 11.00 ResBk manual 104 21 83 53% 20.00 LibBk jwalk 10 10 0 100% 0.30 ResBk jwalk 36 36 0 90% 0.46 Speed and Adequacy of Testing • Test goal: transition cover + equiv. partitions of inputs • manual testing expensive, redundant and incomplete • JWalk testing very efficient, close to complete eg: JWalk achieved 100% test coverage eg: wrote 104 tests, 21 were effective and 83 not!

  19. Conclusion • Performance of JWalk testing • clearly outperformed manual testing • coverage based on all states and transitions • input equivalence partitions are not yet handled • Performance of JWalkEditor • unexpected gain: automatic validation of prototype code • c.f. Alloy’s model checking from a partial specification • Moral for testing • just automatically executing saved tests is not so great • need systematic test generation tools to get coverage • automate the parts that humans get wrong!

  20. Any Questions? http://www.dcs.shef.ac.uk/~ajhs/jwalk/

  21. Bibliography • A J H Simons, JWalk: a tool for lazy systematic testing of Java classes by introspection and user interaction, Automated Software Engineering, 14 (4), December, ed. B. Nuseibeh, (Springer, USA, 2007), 369-418. SpringerLink: DOI 10.1007/s10515-007-0015-3, 8 September, 2007. Final draft version also deposited with White Rose Research Online. • A J H Simons and C D Thomson, Lazy systematic unit testing: JWalk versus JUnit, Proc 2nd. Testing in Academia and Industry Conference - Practice and Research Techniques, 22-24 September, eds. P McMinn and M Harman, (Cumberland Lodge, Windsor Great Park: IEEE, 2007), 138. See also the A1 poster presenting this result. • A J H Simons and C D Thomson, Benchmarking effectiveness for object-oriented unit testing, Proc 1st. Software Testing Benchmark Workshop, 9-11 April, eds. M Roper and W M L Holcombe, (Lillehammer: ICST/IEEE, 2008). • A J H Simons, N Griffiths and C D Thomson, Feedback-based specification, coding and testing with JWalk, Proc 3rd. Testing in Academia and Industry Conference - Practice and Research Techniques, 29-31 August, eds. L. Bottacci and G. M. Kapfhammer and M. Roper, (Cumberland Lodge, Windsor Great Park: IEEE, 2008), to appear. • A J H Simons, A theory of regression testing for behaviourally compatible object types, rev. and ext., Software Testing, Verification and Reliability, 16 (3), UKTest 2005 Special Issue, September, eds. M Woodward, P. McMinn, M. Holcombe, R. Hierons (London: John Wiley, 2006), 133-156. • A J H Simons, Testing with guarantees and the failure of regression testing in eXtreme Programming, Proc. 6th Int. Conf. on eXtreme Programming and Agile Processes in Software Engineering (XP 2005), eds. H Baumeister et al., Lecture Notes in Computer Science, 3556, (Berlin: Springer Verlag, 2005), 118-126. • Wikipedia entry for JWalk • Wikipedia entry for Lazy Systematic Unit Testing

More Related