1 / 66

Model-Based Testing

Model-Based Testing. Ed Brinksma. University of Twente Dept. of Computer Science Formal Methods & Tools group Enschede The Netherlands. ARTIST2 Summer School Nässlingen. Contents. introduction & background testing pre-orders input/output & quiescence ioco implementation relation

gretel
Download Presentation

Model-Based Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Model-Based Testing Ed Brinksma University of TwenteDept. of Computer ScienceFormal Methods & Tools groupEnschedeThe Netherlands ARTIST2 Summer School Nässlingen

  2. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 2

  3. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 3

  4. Testing is: important much practiced 30% - 50% of project effort expensive time critical not constructive(but sadistic?) But also: ad-hoc, manual, error-prone hardly theory / research no attention in curricula not cool :“if you’re a bad programmer you might be a tester” Practical problems of testing ? Attitude is changing: • more awareness • more professional Improvements possiblewith formal methods ! ARTIST2 Summer School 4

  5. Types of Testing Level system integration unit Accessibility white box black box robustness performance usability reliability functional behaviour Aspect ARTIST2 Summer School 5

  6. Why not generate test automatically?! specification TTCN testcases TTCN pass testtool implementationunder test fail Test Automation Traditional test automation = tools to execute and manage test cases ARTIST2 Summer School 6

  7. Verification : formal manipulation prove properties performed on model Testing : experimentation show error concrete system Verification and Testing formal world concrete world Verification is only as good as the validity of the model on which it is based Testing can only show the presence of errors, not their absence ARTIST2 Summer School 7

  8. Testing with Formal Methods • Testing with respect to a formal specification • Precise, formal definition of correctness :good and unambiguous basis for testing • Formal validation of tests • Algorithmic derivation of tests :tools for automatic test generation • Allows to define measures expressing coverageand quality of testing ARTIST2 Summer School 8

  9. Challenges of Testing Theory • Infinity of testing: • too many possible input combinations -- infinite breadth • too many possible input sequences -- infinite depth • too many invalid and unexpected inputs • Exhaustive testing never possible: • when to stop testing ? • how to invent effective and efficient test cases with high probability of detecting errors ? • Optimization problem of testing yield and invested effort • usually stop when time is over ...... ARTIST2 Summer School 9

  10. i imps specification test generation S i passesTs correctness criterion implementation relation imp test suite TS   exhaustive sound test execution implementation i pass / fail Formal Testing  ARTIST2 Summer School 10

  11. specification S IUT conforms-to s correctness criterion implementation IUT Formal Testing : Conformance s  SPECS SpecificationIUT Implementation under Test IUT is concrete, physical object Model the physical world But IUT is black box ! ? Assume that model iIUT exists ARTIST2 Summer School 11

  12. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 12

  13. implementationi specifications environmente environmente is  e  Env. obs ( e, i )  obs (e, s ) Testing Preorders on Transition Systems For all environments e all observations of an implementation i in e should be explained by observations of the specification s in e. ? ? ? ARTIST2 Summer School 13

  14. implementationi specifications environmente environmente Classical Testing Preorder te i tes   e  E. obs ( e, i )obs (e, s )   LTS(L)Deadlocks(e||s) ARTIST2 Summer School 14

  15. implementationi specifications environmente environmente  FP (i) FP (s) FP (p) ={ ,A | p afer  refuses A} Classical Testing Preorder te ites  e  LTS(L).    L*. e||ideadlocksafter e||sdeadlocksafter ites  e  LTS(L).    L*. {  | e||ideadlocksafter} {  | e||sdeadlocksafter} ARTIST2 Summer School 15

  16. te coin coin coin coin They are testing equivalent! coffee coffee tea tea bang bang bang bang coffee tea tea coffee Quirky Coffee Machine[Langerak] Can we distinguish between these machines? ARTIST2 Summer School 16

  17. implementationi specifications environmente environmente Refusal Preorder rf Deadlocks δ(e||i) = {(L{δ})*| e||ideadlocksafter} i rfs   e  E. obs ( e, i )obs (e, s )   LTS(L{δ}) Deadlocks δ(e||i) e observes with δ deadlock on all alternative actions ARTIST2 Summer School 17

  18. coin coin coin coin coffee coffee tea tea bang bang bang bang rf coin coffee tea tea coffee coffee  bang coffee Quirky Coffee MachineRevisited te δ only enabled if coffee is not tester ARTIST2 Summer School 18

  19. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 19

  20. I/O Transition Systems • testing actions are usually directed, i.e. there are inputs and outputs L=LinLoutwith LinLout= • systems can always accept all inputs (input enabledness) for all states s, for all aLin s  • testers are I/O systems • output (stimulus) is input for the SUT • input (response) is output of the SUT a ARTIST2 Summer School 20

  21. Quiescence • Because of input enabledness S||T deadlocks iff T produces no stimuli and S no responses. This is known as quiescence • Observing quiescence leads to two implementation relations for I/O systems I and S: • I ioteS iff for all I/O testers T: • Deadlocks(I||T)  Deadlocks(S||T) (quiescence) • I iorfS iff for all I/O testers T: • Deadlocksδ(I||T) Deadlocksδ(S||T) (repetitive quiescence) ARTIST2 Summer School 21

  22. coin! quiescent states coffee! coffee?  bang! coffee ! iorf coffee? Input-Output QCM states must be saturated with input loops for input enabledness coin? coin? coffee? tea? tea? coffee? bang? bang? coffee! tea ! coffee? tea? iote coffee! tea ! ARTIST2 Summer School 22

  23. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 23

  24. Implementation Relation ioco δ By adding a transition p  p to every quiescent state of a system we treat quiescence as an observable (synchronizable) action: iiorfs I/O tests T: Deadlocksδ(i||T)Deadlocksδ(s||T)   (L{})*: out ( iafter ) out ( safter) To allow under-specificationwe restrict the set of traces: iiocos  Tracesδ( s ) : out ( iafter ) out ( safter) ARTIST2 Summer School 24

  25. Implementation Relation ioco Correctness expressed by implementation relation ioco: iiocos =defTracesδ( s ) : out (iafter )  out (safter) • Intuition: • i ioco-conforms to s, iff • if i produces output x after trace , then s can produce x after  • if i cannot produce any output after trace , then s cannot produce any output after  (quiescence) ARTIST2 Summer School 25

  26. i d ?kwart ?dub ?dub ?kwart !coffee ?dub ?kwart d Implementation Relation ioco iiocos =defStraces (s) : out (iafter )  out (safter) out ( iaftere )= out ( iafter ?dub ) = out ( iafter ?dub.?dub ) = out ( iafter ?dub.!coffee) = out ( iafter ?kwart ) = out ( iafter !coffee ) = out ( iafter ?dub.!tea ) = out ( iafterd ) = {d} { !coffee } { !coffee } {d} {d }   {d} ARTIST2 Summer School 26

  27. i s ?dub ?dub ?dub !coffee !tea !coffee ?dub Implementation Relation ioco iiocos =defStraces (s) : out (iafter )  out (safter) ioco out (iafter ?dub) = { !coffee } out (safter ?dub) = { !coffee, !tea } ARTIST2 Summer School 27

  28. i s ?dub ?dub ?dub !coffee !tea !coffee ioco ?dub ?dub  out (iafter ?dub) = { !coffee, !tea } out (safter ?dub) = { !coffee} Implementation Relation ioco iiocos =defStraces (s) : out (iafter )  out (safter) ARTIST2 Summer School 28

  29. i s ?dub ?kwart ?dub ?dub ?kwart !coffee !coffee !tea Implementation Relation ioco iiocos =defStraces (s) : out (iafter )  out (safter) ioco out (iafter ?dub) = { !coffee } out (iafter ?kwart) = { !tea } out (safter ?dub) = { !coffee }out (safter ?kwart) =  But ?kwart Tracesδ( s ) ARTIST2 Summer School 29

  30. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 30

  31. i iocos specification ? test generation S i passesTs test suite TS test execution implementation i pass / fail Formal Testing  correctness criterion implementation relation ioco ARTIST2 Summer School 31

  32. coin? coin ?  coffee! tea! fail fail coin? coffee! tea!  pass pass fail Test Cases Test case t  TTS TTS - Test Transition System : • labels in L {} • tree-structured • finite, deterministic • final states pass and fail • from each state pass, fail • either one input i? • or all outputs o! and  ARTIST2 Summer School 32

  33. coin? test case t !coin !coin ; Start timer1 ?tea fail ?timer1 fail ?coffee !coin ; Start timer2 ?tea pass ?timer2 pass ?coffee fail coin ?  coffee! tea! fail fail coin? coffee! tea!  pass pass fail Test Cases ARTIST2 Summer School 33

  34. 1 end test case 3 observe output PASS o! allowed outputs o! 2 supply input  FAIL FAIL forbidden outputs supply i? t(S after o!) t(S after i?) Test Generation Algorithm Algorithm To generate a test case t(S) from a transition system specification with S set of states ( initially S = {s0} ) Apply the following steps recursively, non-deterministically ARTIST2 Summer School 34

  35. specification !9 ? x (x < 0) ?-3 otherwise ?3 ? x (x >= 0) FAIL !4 PASS ! -x ! x ?-2 otherwise ?2 PASS FAIL PASS Test Generation Example Equation solver for y2=x test To cope with non-deterministic behaviour, tests are not linear traces, but trees ARTIST2 Summer School 35

  36. Validity of Test Generation For every test t generated with algorithm : • Soundness :twill never fail with correct implementationiiocos implies i passes t • Exhaustiveness:each incorrect implementation can be detectedwith a generated testtiiocos implies t : i fails t ARTIST2 Summer School 36

  37. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 37

  38. s  LTS der : LTS(TTS) ioco Ts TTS pass t: (traces){fail,pass} iIUT IOTS obs : TTS IOTS (traces) traces fail Formal Testing with Transition Systems ARTIST2 Summer School 38

  39. Test Generation Tools for ioco • TVEDA (CNET - France Telecom) • derives TTCN tests from single process SDL specification • developed from practical experiences • implementation relation R1  ioco • TGV (IRISA - Rennes) • derives tests in TTCN from LOTOS or SDL • uses test purposes to guide test derivation • implementation relation: unfair extension of ioco • TestComposer • Combination of TVEDA and TGV in ObjectGeode • TestGen (Stirling) • Test generation for hardware validation • TorX (Côte de Resyste) ARTIST2 Summer School 39

  40. user: manual automatic next input offer input IUT check output TorX observe output specification pass fail inconclusive A Test Tool : TorX • On-the-fly test generation and test execution • Implementation relation: ioco • Specification languages: LOTOS, Promela, FSP, Automata ARTIST2 Summer School 40

  41. explorer primer driver adapter IUT specificationtext statestransitions abstractactions abstractactions concreteactions IUT TorX specification spec. TorX Tool Architecture ARTIST2 Summer School 41

  42. on the fly explorer primer driver adapter IUT TTCN TTCN testtaal testtaal TTCN TTCN TTCN TTCN TTCN TTCN TTCN TTCN TTCN TTCN TTCN TTCN batch test execution spec. batch test generation On-the-Fly  Batch Testing ARTIST2 Summer School 42

  43. New menu ! x (x < 0) ! x (x >= 0) New menu ! x (x < 0) ! x (x >= 0) New menu ! x (x < 0) ! x (x >= 0) Concrete action ? (timeout) Action ? 3 Abstract action ! 9 Abstract action ? 3 Choice ! 9 Check ? (quiescence) Check ? 3 Choice ! -1 Concrete action ! 00001001 Action ? (quiescence) Abstract action ! -1 Concrete action ? 00000011 Abstract action ? (quiescence) Concrete action ! 11111111 abstract IUT IUT IUT bits states explorer primer driver adapter bytes transitions transition actions specification implementation ? x (x < 0) ? x (x < 0) ? x (x >= 0) ? x (x >= 0) spec ! x ! x ! -x ? x On-the-Fly Testing ARTIST2 Summer School 43

  44. TorX ARTIST2 Summer School 44

  45. Contents • introduction & background • testing pre-orders • input/output & quiescence • ioco implementation relation • test generation • TorX • test case study • real-time testing ARTIST2 Summer School 45

  46. Conference Protocol EasyLink TV-VCR protocol Cell Broadcast Centre component Road Toll Payment Box protocol V5.1 Access Network protocol Easy Mail Melder FTP Client “Oosterschelde” storm surge barrier-control TANGRAM: testing VLSI lithography machine TorX Case Studies academic Philips CMG Interpay Lucent CMG academic CMG ASML ARTIST2 Summer School 46

  47. InterpayHighway Tolling System ARTIST2 Summer School 47

  48. Highway Tolling Protocol Characteristics : • Simple protocol • Parallellism : many cars at the same time • Encryption • System passed traditional testing phase ARTIST2 Summer School 48

  49. Highway Tolling System Payment Box (PB) Road Side Equipment Onboard Unit UDP/IP Wireless ARTIST2 Summer School 49

  50. spec TorX Test Context PaymentBox ObuSim PB + ObuSim + TCP/IP + UDP/IP PCO IAP SUT UDP/IP TCP/IP Highway Tolling: Test Architecture ARTIST2 Summer School 50

More Related