1 / 18

BAiT: Adaptive Test Case Execution in Practice

BAiT: Adaptive Test Case Execution in Practice. Jens R. Calamé Centrum voor Wiskunde en Informatica, Amsterdam. Agenda. Introduction The Toolset for BAiT An Exemplary Case: The ATM Structure of BAiT Conclusion. Introduction. BAiT: Behavior Adaptation in Testing

bozica
Download Presentation

BAiT: Adaptive Test Case Execution in Practice

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BAiT: Adaptive Test Case Executionin Practice Jens R. Calamé Centrum voor Wiskunde en Informatica, Amsterdam

  2. Agenda • Introduction • The Toolset for BAiT • An Exemplary Case: The ATM • Structure of BAiT • Conclusion DWFTT 2007, Enschede

  3. Introduction • BAiT: Behavior Adaptation in Testing • Toolset to generate test cases and test oracles as well as execute tests on nondeterministicsystems with data DWFTT 2007, Enschede

  4. Agenda • Introduction • The Toolset for BAiT • An Exemplary Case: The ATM • Structure of BAiT • Conclusion DWFTT 2007, Enschede

  5. BAiT: Test Case & Oracle Generation • Test generation with TGV (enumerative approach) • Chaotic data abstraction for input and output parameters of events • Test oracle generation for Eclipse Prolog constraint solver in order to select test data • Automatic generation of variable names for the test case DWFTT 2007, Enschede

  6. BAiT: Test Proxy Generation • Test Proxy for Java-based SUTs • Instantiation of test object within test proxy • Procedure-basedbidirectional communication (publisher-subscriber pattern) • Mapping of datatypes from specification language to target platform DWFTT 2007, Enschede

  7. BAiT: Test Execution • Test execution of pre-calculated test trace (behavior and data) • Adaptation to the IUT's reactions, rather than immediate assignment of verdict • BAiT framework provides adapters for user-defined • trace selection and • test data selection algorithms DWFTT 2007, Enschede

  8. Agenda • Introduction • The Toolset for BAiT • An Exemplary Case: The ATM • Structure of BAiT • Conclusion DWFTT 2007, Enschede

  9. An Exemplary Case: The ATM ATM specification (partially) ATM test purpose (completely) DWFTT 2007, Enschede

  10. !getPin(1234) ?pinCorrect !initBalance(1000) ?retFifty(2) ?retFifty(1) ?retTwenty(2) ?retTen(1) !initPin(1234) An ATM Test Case !getAmount(100)  ? Photos: Wikipedia and ECB DWFTT 2007, Enschede

  11. An ATM Test Case (2) 1. Precalculation of possible test trace May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Pruning planned test trace. May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.RunTest findTrace FINER: Examining: [init],initPin(Pin),getPin(PinUser),pinCorrect, initBalance(Bal),getAmount(Amt),[tau],retFifty(Fif),emitBankNotes May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl solve FINEST: init(G1),initPin(G1,G2,lparam(nat(Pin))),[...] [Pin in {[-Infinity .. Infinity]} => 1234] May 25, 2007 3:06:21 PM nl.cwi.sen2.bait.steps.TraceImpl solve FINEST: init(G1),initPin(G1,G2,lparam(nat(1234))),[...] [...] May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Trying: [init],initPin(1234),getPin(1234),pinCorrect, initBalance(1000),getAmount(100),[tau],retFifty(2),emitBankNotes May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest FINE: initPin(1234) -> OK [...] May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest FINE: retFifty(2) -> NOT OK May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Expected: retFifty(2), received: retFifty(1) -> Trace failed, trying alternative. 2. Iterative data selection 3. Stepwise execution of the trace… …until the IUT deviates from the precalculated trace. DWFTT 2007, Enschede

  12. An ATM Test Case (3) May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Trace deviated; executed so far: [init],initPin(1234),getPin(1234),pinCorrect,initBalance(1000), getAmount(100),[tau],retFifty(1),retTwenty(2),retTen(1), emitBankNotes May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Pruning planned test trace. May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Adding retFifty(1) to trace stub. [...] May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge FINE: Adding emitBankNotes to trace stub. May 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest INFO: Test finished; executed: [init],initPin(5),getPin(5), pinCorrect,initBalance(1000),getAmount(100),[tau],retFifty(1), retTwenty(2),retTen(1),emitBankNotes May 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest INFO: The test case ended with verdict PASS. 4. Pruning not yet executed part of test trace. 5. Appending received system output to trace. 6. Solve test trace with pending system output; assign verdict or calculate new trace. DWFTT 2007, Enschede

  13. Agenda • Introduction • The Toolset for BAiT • An Exemplary Case: The ATM • Structure of BAiT • Conclusion DWFTT 2007, Enschede

  14. Structure of BAiT: The Tools • BAiT: Behavior Adaptation in Testing • Toolset to generate test cases and test oracles as well as execute tests on nondeterministicsystems with data DWFTT 2007, Enschede

  15. Structure of BAiT: Execution Part • Variable class: Codec of data to constraint solver, data instantiator and IUT • Data instantiation algorithm(s) Test Oracle Data Mapping Constraint Solver Data Mapping Impl. Under Test BAiT Execution Core Impl. Under Test Data Selection Constraint Solver Trace Selection Behavior Mapping Constraint Solver Behavior Mapping Impl. Under Test Test Case(s) • Trace class: trace execution management • Step class: communication with IUT • Trace selection from complete test graph (IOLTS with all possible test cases for a given test purpose) DWFTT 2007, Enschede

  16. Agenda • Introduction • The Toolset for BAiT • An Exemplary Case: The ATM • Structure of BAiT • Conclusion DWFTT 2007, Enschede

  17. Conclusion • Behavior Adaptation in Testing helps finding more reliable verdicts in testing non-deterministic systems • A prototypical framework for BAiT has been developed procedure-based setting with atomic data • Further work: • Extension of frameworkfor other settings • Extension of frameworkto structured datatypes • Case Study:Mozilla FirefoxCSS subsystem DWFTT 2007, Enschede

  18. Further Information Background information • Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Sidorova, Natalia; Data Abstraction and Constraint Solving for Conformance Testing (APSEC 2005). • Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Towards Automatic Generation of Parameterized Test Cases from Abstractions. Technical Report, 2006. The toolset/framework itself • Calamé, Jens R.; Adaptive Test Case Execution in Practice. Technical Report, 2007. • Calamé, Jens R.; The Internals of Adaptive Test Case Execution. Technical Report,to appear. All downloadable from www.calame.de/works,published.bib,date,.html Contact: Jens R. Calamé CWI, Amsterdam jens.calame@cwi.nl DWFTT 2007, Enschede

More Related