Synthesis of Test Purpose Directed Reactive Planning Tester for Nondeterministic Systems 1

1. Synthesis of Test Purpose Directed Reactive PlanningTester for Nondeterministic Systems1 Jüri Vain,Kullo Raiend, Andres Kull, Juhan P. Ernits 1 Accepted in ASE 2007, Atlanta, Georgia, November 5-9, 2007 Teooriapäevad, Vanaõue, Sept. 2007

2. On-the-fly testing: what? and why? • The mostappropriate technique for model-based testing of IUT modeled using nondeterministicmodels • Denotes test generation and execution algorithms that compute successivestimuli at runtime, directed by the testpurpose and the observed outputs of the IUT • Advantages: • The state-space explosion problem experienced by manyoffline test generation methods is reduced because only a limited part of the state-spaceneeds to be kept track of at any point in time. • Drawbacks: • Exhaustive planning is diffcult due to thelimitations of the available computational resources at thetime of test execution. Teooriapäevad, Vanaõue, Sept. 2007

3. On-the-fly testing: scale of methods • Random walk (the simplest): select test stimuli in random • inefficient - based on random exploration of the statespace • leads to test cases that are unreasonably long • may leave the test purpose unachieved • RW with additional heuristics: • E.g., anti-ant algorithms are applied forguiding the exploration of the state space [Li, Veanes] • ........? • State space exploration with exhaustive planning - solvingconstraint systems at each step • witnesstrace generated by model checking provides possibly optimalselection of the next test stimulus. • in case of explicit state MC the sizeof the model is critical • leads to the explosion of thestate space specially in models that include "combination lock" ordeep loops [J. Rushby]. Teooriapäevad, Vanaõue, Sept. 2007

4. On-the-fly testing: missing link -Reactive Planning Tester • Reactive planning [C. Williams and P. P. Nayak]: • can cope with highly dynamic and unpredictableenvironments. • instead of a complete plan with branches, a setof decision rulesis produced directing towards the planning goal. • just one subsequent input is computedat every step, based on the current context. • Reactive planning tester [J.Vain et al] • Model “in-the-loop”: testing is guided by an EFSM model • stimulae to IUT: outputs generated by model execution • responses from IUT: inputsto the model • decision rules of reactive planning are encoded in the guards of model transitions • the rules are constructed by offline analysis based on the given IUT model and the testpurpose. Teooriapäevad, Vanaõue, Sept. 2007

5. The model • Key assumption: the IUT model is presentedas an output observable nondeterministic state machine: • FSM • EFSM in which alltransition paths are feasible (see Duale,2004 for algorithms of making EFSM feasible) Teooriapäevad, Vanaõue, Sept. 2007

6. I/O EFSM (I) • Definition 1: (EFSM)M = (S; V; I;O;E), where • S-finiteset of states, • s0S- initial state, • V- finite setof variables with finite value domains, • I - finite setof inputs, • O- finite set of outputs, • E - setof transitions. • Configuration of M is a pair (s; ) where • s S •  mapping from V to values. Teooriapäevad, Vanaõue, Sept. 2007

7. I/O EFSM (II) • Transition e =(s; p; a; o; u; q),where • s - the source state of the transition, • q- target state of the transition (s; q) • p - transitionguard - a logic formula over V, • a - input of M(a I), • o– outputof M (o O), • u- updatefunction over V. • Nondeterministic EFSM: more than oneoutgoing transitions enabled simultaneously Teooriapäevad, Vanaõue, Sept. 2007

8. Example: nondeterministic EFSM Teooriapäevad, Vanaõue, Sept. 2007

9. Test purpose (TP): • TP is a specific property of theIUT that the tester is set out to test. • In MBT the TPsare defined in terms of the IUT model elements • Examples: • "test a state change from state A tostate B in a model", • "test whether some selected states of amodel are visited", • "test whether all transitions are visitedat least once in a model", etc. • All TPs of examples can be specified in terms of structural elements ofthe model that should be traversed during the test execution Teooriapäevad, Vanaõue, Sept. 2007

10. Encoding Test Purpose in the IUT Model • Trap - a boolean variable assignment attached to the transitions of the IUT model • A trap variable is initiallyset to false. • The trap update functions attached to thetransitions are executed (set trap variables to true) when thetransition is visited. Teooriapäevad, Vanaõue, Sept. 2007

11. Example: nondeterministic EFSM extended with trap variables Teooriapäevad, Vanaõue, Sept. 2007

12. Model of the tester • EFSM generated from the IUT modelattributed with test purpose • I/O conformance of IUT and tester models is required • transition guards encode the rules of online planning • 2 types of tester states : • active – tester controls the next move • passive – IUT controls the next move • 2 types of transitions: • Observable – source state is passive state, guard: true, • Controllable – source state is active state, guard: pS  pT, where pS – guard of the dual transition of IUT, pT– gain guard The gain guard (defined on trap variables) must ensure that only those outgoing are enabled having maximum gain Teooriapäevad, Vanaõue, Sept. 2007

13. Tester construction algorithm: a sketch • Construct a control structure structure of the tester model where • the inputs and outputs are inverted. • Split the transitions to pairs of controllable and observable transitions by i/o operation attached. • Extend the tester automaton controllable transitions guards with “place holders” for gain guards • The gain guards are construct with reachability analysis by taking into account • thereachability of all trap-equipped transitions from a givenstate • the length of the paths to them • the currentvalue (visited or not) of each trap. Teooriapäevad, Vanaõue, Sept. 2007

14. Constructing the control structure of the tester Teooriapäevad, Vanaõue, Sept. 2007

15. Constructing the gain guards (GG) (I): intuition • GG must guarantee that • each transition enabled by GG is a prefix of some locally optimal (w.r.t. test purpose) path • tester should terminate after the test goal is reached or all unvisited traps are unreachable from the current stat • to have a quantitative measure of the gain of executing any transition e we define a gain function ge:TRe  B Q+ Teooriapäevad, Vanaõue, Sept. 2007

16. Constructing the gain guards (II): the gain function • ge = 0, if it is useless to fire the transition efrom the current state with the current variable bindings • ge > 0, if fireing transition e from the current state with the current variable bindings visits or leads closer to at least one unvisited trap • gei > gej for transitions ei and ej with the same source state, if taking the transition ei leads to unvisited traps with smaller cost than taking the transition ej. • Having gain function gewith given properties define GG: pT ge = maxekgek  ge > 0 Teooriapäevad, Vanaõue, Sept. 2007

17. Constructing the gain guards (III): shortest path tree • Reachability problem of trap labeled transitions can be reduced to single- source shortest path problem. • Arguments of the gain function ge are • Shortest path tree TRe with root node e • VT – vector of trap variables • To construct TRe we createa graph G = (VD,ED) where • the vertices VDof G correspond to the transitions of the MT, • the edges ED of G represent the pairs of subsequent transitions sharinga state in MT. Teooriapäevad, Vanaõue, Sept. 2007

18. Constructing the gain guards (IV): shortest path tree (example) Figure 4: The shortest-paths tree (left) and the reduced shortest-paths tree (right) from the transition ec01 of the graph shown inFigure 3. Figure 3: The dual graph of the tester model inFigure 2. Teooriapäevad, Vanaõue, Sept. 2007

19. Constructing the gain guards (IV): the gain function • Represent the reduced tree TR(ei,G) as a set of elementarysub-trees each specified by the productioni j{1,..n}j • Rewrite the right-hand sides of the productions as arithmetic terms: (3) • ti - trap variable ti lifted to type N, • c - constant for the scaling of the numerical value of the gain function, • d(0, i) the distance between vertices 0 and i, where l - the number of hyper-edges on the path between 0 and i w j – weight of j-th hyperedge Teooriapäevad, Vanaõue, Sept. 2007

20. Constructing the gain guards (IV): the gain function (continuation) • For each symbol i denoting a leaf vertex in TR(e,G) define a production rule (4) • Apply the production rules (3) and (4) starting fromthe root symbol 0of TR(e,G) until all non-terminalsymbols iaresubstituted with the terms that includeonly terminal symbols ti and d(0, i) Teooriapäevad, Vanaõue, Sept. 2007

21. Constructing the gain guards (IV): the gain function (example) Teooriapäevad, Vanaõue, Sept. 2007

22. Constructing the gain guards (IV): the gain function (example continues) Teooriapäevad, Vanaõue, Sept. 2007

23. Complexity of constructing and running the tester • The complexity of the synthesis of the reactive planningtester is determined by the complexity of constructing gain functions. • For each gain function the cost of findingthe TREby breadth-first-search is O(|VD| + |ED|) [Cormen], where • |VD| = |ET| - number of transitions of MT • |ED| - number of transition pairs of MT(is bounded by |ES|2) • For all controllabletransitions of the MT the upper bound of thecomplexity of the computations of the gain functionsis O(|ES|3). • At runtime each choice by the tester takes O(|ES|2) arithmetic operations to evaluate the gain functions Teooriapäevad, Vanaõue, Sept. 2007

24. Experimental results Teooriapäevad, Vanaõue, Sept. 2007

25. Conclusions • Efficiency of planning: • number of rulesthat need to be evaluated at each step is relatively small (number of outgoing transition of current state) • the execution of decision rules is significantly faster than finding the test path by MC but • neverthelessleads to the test sequence that is lengthwise close tooptimal. Teooriapäevad, Vanaõue, Sept. 2007