1 / 62

Online Testing of Nondeterministic Systems with the Reactive Planning Tester

Online Testing of Nondeterministic Systems with the Reactive Planning Tester. Jüri Vain Dept. of Computer Science Tallinn University of Technology. Reference. Vain , Jüri; Kääramees, Marko; Markvardt, Maili Online testing of nondeterministic systems with reactive planning tester.

ruby
Download Presentation

Online Testing of Nondeterministic Systems with the Reactive Planning Tester

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Online Testing ofNondeterministic Systems withthe Reactive Planning Tester Jüri Vain Dept. of Computer Science Tallinn University of Technology J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  2. Reference Vain, Jüri; Kääramees, Marko; Markvardt, Maili Online testing of nondeterministic systems with reactive planning tester. In: Petre, L.; Sere, K.; Troubitsyna, E. (Eds.). Dependability and Computer Engineering : Concepts for Software-Intensive Systems. Hershey, PA: IGI Global (2012), pages 113-150. J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  3. Outline • Preliminaries • Model-Based Testing • Online testing • Reactive Planning Tester (RPT) • Synthesis of the RPT • Performance issues • Test execution environment dTron • Demo J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  4. What is testing for? • to check the quality (functionality, reliability, performance, …) of an (software) object-by performing experiments -in a controlled way • In avg. 10-20 errors per 1000 LOC • 30-50 % of development time and cost in software J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  5. Test Cases Output Test Data Oracle What is a Test? Correct result? Software under Test (SUT) J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  6. Model-Based Testing (typically) J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  7. Specifics of testing embedded systems Real environment in the loop! – non-determinism – partial observability –RT constraints – dependability  test coverage issues How to address them in testing? J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  8. Option 1: Offline vs Online testing • Offline testing • Open “control” loop • Test is not adaptive to outputs or timing of SUT • Test planning – result analysis loop is long • Online testing • Flexible, test control is based on SUT feedback • One test covers usually many test cases J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  9. Online testing • Group of test generation and execution algorithms that • compute successivestimuliat runtimedirectedby • the testpurpose and • the observed outputs of the SUT J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  10. Online testing • Advantages: • The state-space explosion problem is reduced because only a limited part of the state-spaceneeds to be kept track of at any point in time. • Drawbacks: • Exhaustive planning is diffcult due to thelimitations of the available computational resources at thetime of test execution. J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  11. Option 2: Test scripting vs model-based generation • Scripting: • does not need SUT modelling + • sensitive to human errors - • inflexible, needs rewriting even with small changes of SUT specs - • hard to achieve test coverage - • Model-based generation: • considerable SUT modelling effort - • correctness of tests is verifiable + • easy to modify and regenerate + • clear characteristics of coverage + J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  12. Model-Based Testing • Given • Model of the SUT specification • System Under Test (SUT), • The test goal (in terms of spec. model elements) • Find • If the SUT conforms to the specification in terms expressed in the test goal. • NEEDED! • Sufficiently rich modelling formalism, • Supporting tool set. J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  13. Model-Based Testing • We assume formal specs as: • UML State Charts • Extended Finite State Machines • MSC • OCL • etc. • UPTA -Timed Automata (timing, parallelism, test data) J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  14. ”Relativized Real-Time i/o conformance” Relation Online MBT architecture: UppAal-TRON Spec = UppAal Timed Automata Network: Env || IUT Inputevent ordering Expectedsystem reaction Timed Trace: i1.2½.o1.3.o2.19.i2.5.i3 [www.uppaal.com] J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  15. Bottleneck of online MBT: planningSpectrum of planning methods • Random walk (RW): select test stimuli in random • inefficient - based on random exploration of the statespace • leads to test cases that are unreasonably long • may leave the test purpose unachieved • RW with reinforcement learning (anti-ant) • the exploration is guided by some reward function • ........ • Exploration withexhaustive planning • MCprovides possibly an optimalwitnesstrace • the sizeof the model is critical in explicit state MC • state explosion in "combination lock" ordeep loop models J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  16. Bottleneck of online MBT: planningSpectrum of planning methods • Random walk (RW): select test stimuli in random • inefficient - based on random exploration of the statespace • leads to test cases that are unreasonably long • may leave the test purpose unachieved • RW with reinforcement learning (anti-ant) • the exploration is guided by reward function • ........ • Exploration withexhaustive planning • MCprovides possibly an optimalwitnesstrace • the sizeof the model is critical in explicit state MC • state explosion in "combination lock" ordeep loop models Zero planning 1 step ahead ??? Full space J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  17. Bottleneck of online MBT: planningSpectrum of planning methods • Random walk (RW): select test stimuli in random • inefficient -random exploration of the statespace • test cases that are unreasonably long • may leave the test purpose unachieved • RW with reinforcement learning (anti-ant) • the exploration is guided by some reward function • ........ • Exploration withexhaustive planning • MCprovides possibly an optimalwitnesstrace • the sizeof the model is critical in explicit state MC • state explosion in "combination lock" ordeep loop models Zero planning 1 step ahead Planning with adaptive horizon! Full space J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  18. Reactive Planning in a Nutshell (1) • Instead of a complete plan, only a set of decision rules is derived • The rules direct the system towards the testgoal. • Based on current situation evaluation just one subsequent input is computed at a time. • Planning horizon is adjusable J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  19. Reactive Planning: Planning cycle • Identify current state of SUT: • Observe the output (or history) of the SUT • Pick the next move: • Select one from unsatisfied test (sub-)goals Gi • Compute the best strategy for Gi: • Gain function guides the exploration of the model (choose the transition with the shortest path to the (sub-)goal Gi) J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  20. Reactive Planning Tester in a Nutshell (2) Offline phase Online phase State model of the IUT SUT Test goal Stimuli / responses Reachability Analysis Reactive Planning Tester (RPT) implementation Reactive Planning Tester (RPT) model RPT autonomously generates stimuli to reach the goal RPT is another state model J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  21. Constructing the RPT model • Test strategy parameters: • planning horizon • timeouts Synthesis of RPT Constructing gain guards (GG) Model of SUT Extraction of the control strcture Reduction of GG Model of RPT Constructing gain functions Test goal J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  22. RPT: Key Assumptions • Testing is guided by the (EFSM) model of the tester and the test goal. • Decision rules of reactive planning are encoded in the guardsof the transitions of the tester model. • The SUT model is presentedas an outputobservable nondeterministic EFSM in which allpaths are feasible1. • 1- A. Y. Duale and M. U. Uyar. A methodenablingfeasible conformance testsequence generation forEFSM models.IEEE Trans. Comput.,53(5):614–627,2004. J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  23. s1 Example: Nondeterministic FSM e1: i0/o1 e3: i3/o3 e0: i0/o0 s2 e2: i2/o2 e5: i5/o5 e6: i6/o6 e7: i7/o7 e4: i3/o4 s3 i0 andi3 are output observable nondeterministic inputs J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  24. Encoding the Test Goalin SUTModel • Trap- a boolean variable assignment attached to the transitions of the IUT model • A trap variable is initiallyset to false. • The trap update functions are executed (set to true) when thetransition is visited. J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  25. Goal directed testing: The power of traps • Several goals can be expressed • all/selected transitionstransition sequences (trapswith • reference to other traps) • advancedgoals using auxiliary variables, • consequent transitions, repeated pass, … • traps with 1st order predicates data variables • Properties not expressible by traps • Assertions/invariants – always/never propertiesThe model specifies only the allowed behaviours Åbo Akademi University, Dec 2011

  26. s1 Add Test Purpose e1: i0/o1, t1=true e3: i3/o3, t3=true e0: i0/o0 t0=true s2 e2: i2/o2, t2=true e5: i5/o5, t5=true e6: i6/o6, t6=true e4: i3/o4, t4=true e7: i7/o7, t7=true s3 bool t0 = false; ... bool t7 = false; J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  27. s1 s1 Example: Adding the Test Goal Test goal is defined by trap variables ti attached to transitions e3: i3/o3, t3=true e1: i0/o1, t1=true e0: i0/o0 t0=true s2 e2: i2/o2, t2=true e5: i5/o5, t5=true e6: i6/o6, t6=true e4: i3/o4, t4=true e7: i7/o7, t7=true Initial values: bool t0 = false; ... bool t7 = false; Trap update functions are executed (set to true) when the transition is visited s3 Trap variables are initially set to false J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  28. Model of the tester • Generated from the SUT model decorated with test purpose • Transition guards encode the rules of online planning • 2 types of tester states: • active – tester controls the next move • passive – SUT controls the next move • 2 types of transitions: • Observable – source state is a passive state (guard  true), • Controllable – source state is an active state (guard pS /\pT where pS– guard of the SUT transition; pT– gain guard) The gain guard(defined on trap variables) must ensure that only the outgoing edges with maximum gain are enabled in the given state. J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  29. e1: i0/o1, t1=true s1 e3: i3/o3, t3=true e0: i0/o0 t0=true s2 e2: i2/o2, t2=true e5: i5/o5, t5=true e6: i6/o6, t6=true e4: i3/o4, t4=true e7: i7/o7, t7=true s3 Construction of the Tester Sceleton s1 sa sb s2 sc States: Transitions: - active - observable - passive - controllable s3 sf sd se J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  30. Add IO and Gain Guards s1 ec01: [pc01(T)] -/i0 eo0: o0/t0=true eo2: o2/t2=true s4 eo1: o1/t1=true eo7: o7/t7=true s5 s2 eo5: o5/t5=true ec2: [pc2(T)] -/i2 ec34: [pc34(T)] -/i3 eo3: o3/t3=true s6 eo6: o6/t6=true ec5: [pc5(T)] -/i5 ec6: [pc6(T)] -/i6 s3 s7 ec7: [pc7(T)] -/i7 s6 s9 J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  31. Constructing the gain guards (GG): intuition • GG must guarantee that • each transition enabled by GG is a prefix of some locally optimal (w.r.t. test purpose) path; • tester should terminate after the test goal is reached or all unvisited traps are unreachable from the current state; • to have a quantitative measure of the gain of executing any transition e we define a gain function ge that returns a distance weighted sum of unsatisfied traps that are reachable along e. J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  32. Constructing Gain Guards: intuition Observed current state Alternative choices tri trj trk Planning cones to be covered for decision making ... ... J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  33. Constructing Gain Guards: intuition Gain functions characterize planning options ... Current state Alternative continuation choices ge(tri,T ) ge(trj,T ) ge(trj,T ) tri trj trk ... J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  34. Constructing the gain guards: properties of gain functions • ge = 0, if it is useless to fire the transition efrom the current state with the current variable bindings; • ge > 0, if fireing the transition e from the current state with the current variable bindings visits or leads closer to at least one unvisited trap; • gei > gejfor transitions ei and ej with the same source state, if taking the transition ei leads to unvisited traps with smaller distance than taking the transition ej; • Having gain function gewith given properties define GG: pT (ge = maxkgek)  ge > 0 J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  35. Constructing the Gain Functions: shortest path trees • Reachability problem of trap labelled transitions can be reduced to single-source shortest path problem1. • Arguments of the gain function geare then • the shortest path tree TRe with root node e • VT – vector of trap variables • To construct TRe we createa dual graph G = (VD,ED) of the testercontrol graph MT where • the vertices VDof G correspond to the transitions of the MT, • the edges ED of G represent the pairs of consequtive transitions sharinga state in MT (2-switches) 1 - Fredman & Tarjan 1987O(E + V log V) J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  36. Constructing the Gain Guards: shortest path tree (example) The dual graph of the tester model The shortest-paths tree (left) and the reduced shortest-paths tree (right) from the transition ec01 J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  37. Constructing the gain guards: gain function (1) • Represent the reduced tree TR(ei,G) as a set of elementarysub-trees each specified by the productioni j{1,..n}j • Rewrite the right-hand sides of the productions as arithmetic terms: (3) • ti - trap variable ti lifted to type N, • c - constant for rescaling the numerical value of the gain function, • d(0, i) the distance between vertices 0 and i, where l - the number of hyper-edges on the path between 0 and i w j – weight of j-th hyperedge J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  38. Constructing the gain guards: gain function (2) • For each symbol i denoting a leaf vertex in TR(e,G) define a production rule (4) • Apply the production rules (3) and (4) starting fromthe root symbol 0of TR(e,G) until all non-terminalsymbols iaresubstituted with the terms that includeonly terminal symbols ti and d(0, i) J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  39. Example: Gain Functions J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  40. Example: Gain Guards J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  41. e1: i0/o1, t1=true s1 e3: i3/o3, t3=true e0: i0/o0 t0=true s2 e2: i2/o2, t2=true e5: i5/o5, t5=true e6: i6/o6, t6=true e7: i7/o7, t7=true e4: i3/o4, t4=true s3 Example: Construction of the Tester Structure eo6: o6/t6=true s1 ec01: [pc01(T)] -/i0 eo0: o0/t0=true eo2: o2/t2=true s4 eo1: o1/t1=true eo7: o7/t7=true s5 s2 eo5: o5/t5=true ec2: [pc2(T)] -/i2 ec34: [pc34(T)] -/i3 The gain guards guarantee that only the outgoing edges with maximum gain are enabled in the given state eo3: o3/t3=true s6 eo4: o4/t4=true ec5: [pc5(T)] -/i5 ec6: [pc6(T)] -/i6 s3 s7 ec7: [pc7(T)] -/i7 J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012 s6 s9

  42. Complexity of constructing and running the tester • The complexity of the synthesis of the reactive planningtester is determined by the complexity of constructingthe gain functions. • For each gain function the cost of findingthe TReby breadth-first-search is O(|VD| + |ED|) [Cormen], where • |VD| = |ET| - number of transitions of MT • |ED| - number of transition pairs of MT(is bounded by |ES|2) • For all controllabletransitions of the MT the upper bound of thecomplexity of the computations of the gain functionsis O(|ES|3). • At runtime each choice by the tester takes O(|ES|2) arithmetic operations to evaluate the gain functions J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  43. Performance of reactive planning • Test Goal: All Transitions • Test Goal: Selected Transition J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  44. Experiments: Case Study: Model of the IUT Model of Feeder Box Controller power management (31 states, 73 transitions) J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  45. Howtoplaninlargemodels? • Adjustable planning horizon • Dependancy between horizon and test length Test goal: all transitions Test goal: singletransition Average lengths ofthe test sequences Horizon saturation point J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  46. Shaping RPT planning cones (i) ... Decision point Gain functions Alternative choices ge(tri,T ) ge(trj,T ) ge(trj,T ) tri trj trk In online-testing decision time must be strictly bounded! ... J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  47. Shaping RPT planning cones (ii) ... Decision point Gain functions Alternative choices ge(tri,T ) ge(trj,T ) ge(trj,T ) tri trj trk Decision time strictly bounded! Prune the cone! Fullyvisible ... Horizon h Different ways for defining h, and ‘visibility’ depending on depth of tree. Partiallyvisible J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  48. Average lengths of test sequences in the experiments J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  49. Average time spent for online planning of the next step J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

  50. How to generate test data? J.Vain “...Reactive Planning Testing...”Abo, Feb 3, 2012

More Related