Seminar: Software Quality and Safety Learning By Testing

1. Seminar: Software Quality and SafetyLearning By Testing Volker Fortströer

2. Motivation [1] • How improve quality control for reactive systems? • Availability of a specification of system behavior • Rarely precise & reliable because of last minute changes etc. • But a specification of the system is needed to test and to verify the behavior • Automatically generated and maintained reference models can help to get precise system specification Volker Fortströer – Learning by Testing

3. Motivation [2] Volker Fortströer - Learning by Testing • Model generation from source code is possible but not practicable • hardware systems, closed source components • An automated method is needed to generate a model of a (reactive) system

4. Overview Reactive systems Regular languages Finite-state automaton Learning an automaton Angluins Algorithm L* Optimizations to L* Volker Fortströer - Learning by Testing

5. Reactive Systems • Systems which creates output on different input and may change internal states • i.e.: telecommunication systems • Preconditions: • The system responses have to be deterministic • The same input results in the same end state (output) • All input is accepted regardless of the system’s internal state • Conclusion: such a system can be seen as an input/output automaton • This automaton can be learned throughalgorithms by creating a deterministic automaton Volker Fortströer - Learning by Testing

6. Deterministic Finite-State Automaton • A deterministic finite-state automaton (DFA) over Σ is a structure A = (Q, δ, q0, F). • Q: non-empty finite set of states • q0є Q: initial state • F ⊆ Q: set of final states • δ: Q х Σ → Q is the transition function • Run of A: execution of A with input w • w: a1 … an є Σ* • Acceptance of w if A stops in a state q є F • The set of accepted words build a regular language Volker Fortströer - Learning by Testing

7. Regular languages • Let L be a language over the alphabet Σ. • L ⊆ Σ*is called regular iff. either • L is generated by a regular grammar or • L is accepted by a finite automaton or • L can be described trough a regular expression Volker Fortströer - Learning by Testing

8. Learning a Finite Automata [1] • Algorithm L* invented by Angluin (87) • determining an initially unknown regular language L • Two information sources are needed to learn the regular language L: • Membership Oracle (MO) checks if a sequence w is in L • Equivalence Oracle (EO) checks if an acceptor for L is equivalent to the unknown L • Construct a equivalent minimal DFA which accepts L Volker Fortströer - Learning by Testing

9. Learning a Finite Automata [2] • Basic idea behind L*: • Systematically explore feasible strings using the MO • Construct the transition table • maintain during exploration • A set S of state-access strings • A set E of explored states • A function T for mapping (S ∪ S ∙ A) ∙ E → {0,1} • T is kept in the observation table (OT) • A is the input-alphabet Volker Fortströer - Learning by Testing

10. Learning a Finite Automata [3] • Gradually build a DFA M (hypothesis) • If M seems to be stable make an equivalence query • If answer is correct then we have found an equivalent DFA • Otherwise we get a counterexample (which can be used for refining DFA M) Volker Fortströer - Learning by Testing

11. Application of L* [1] Volker Fortströer - Learning by Testing • Assume a reactive system like a telecommunication system • One switch and some phones • Phone actions (input): • on-hook (↑) • off-hook (↓) • perform a call (→) • Switch actions (output): • initiated • cleared • hookswitch (signal received)

12. Application of L* [2] Volker Fortströer - Learning by Testing • different (simple) scenarios: • S1: 1 physical device A AI = {A↑, A↓} AO = {initiatedA, clearedA, [hookswitchA]} • S2: 2 physical device A, B AI = {A↑, A↓, B↑, B↓} AO = {initiatedA,B, clearedA,B, [hookswitchA,B]} • depending on input through devices the switch produces output corresponding to its internal states • ⇒ I/O-Automaton

13. Input / Output Automaton Volker Fortströer - Learning by Testing • An input / output automaton over Σ is a structure S = (Σ, AI, AO, →, s0). • Σ: non-empty finite set of states • AI: finite set of input actions • AO: finite set of output responses • →: transition relation → ⊆ Σ х AIх AO* х Σ • s0є Σ: initial state • An input / output automaton can be transformed in a deterministic finite-state automaton • Divide input words into single symbols • Create auxiliary-states for transitions

14. Adaptations to L* for learning a reactive system [1] • Represent a I/O-automaton as a DFA • Because a large input alphabet results in inefficient runtime of L* • Membership Oracle: • This queries can be answered by the system itself • Problems: • Interface to the system is needed for invoking the input und capturing the response • Expensive in time because of timeouts Volker Fortströer - Learning by Testing

15. Adaptations to L* for learning a reactive system [2] Volker Fortströer - Learning by Testing • Equivalence Oracle: • Learning a black-box system ⇒ one can never be sure that the whole system behavior was learned • Using an approximation: • Checking consistency within a fixed look ahead from all states

16. Optimizing the runtime of L* • Performing an oracle query takes relatively much time • L* has to wait for response of tested system • An obviously improvement is reducing the number of queries • Some queries can be answered with the result of other queries • i.e.: counterexample of EQ • Other queries can be filtered because of the properties of the learned language Volker Fortströer - Learning by Testing

17. Prefix-Closure • If every state is an accepting state the corresponding language is prefix-closed • All prefixes v of every word w in language L are member of L • Filter 1 (positive prefix):∃σ2∈ A*. T(σ1;σ2) = 1 ⇒ MO(σ1) = true • Filter 2 (negative prefix):∃σ2∈ prefix(σ1). T(σ2) = 0 ⇒ MO(σ1) = false Volker Fortströer - Learning by Testing

18. Input Determinism Every input produces always the same output Filter 3 (Input Determinism):∃x ∈ AO, y ∈ A, σ2, σ3∈ A*. σ1 = σ2;x;σ3∧ T(σ2;y;σ3 ) = 1 ∧ x ≠ y ⇒ MO(σ1) = false Filter 4 (Output Completion): ∃a ∈ AI, x ∈ AO. σ2;x ∈ prefix(σ1) ∧ T(σ2;a) = 1 ⇒ MO(σ1) = false Volker Fortströer - Learning by Testing

19. Independence of Events [1] • Reactive systems exhibit often a high degree of parallelism • Sometimes different components of one type are interchangeable • i.e.: a device A behaves like device B • Independence:actions of device A and B can be performed in different order • Symmetry:an action performed by A can also be performed by B Volker Fortströer - Learning by Testing

20. Independence of Events [2] Volker Fortströer - Learning by Testing A membership query for σ1 is true if an equivalent σ2 is already in OT Let Cpo,sym be the set of all equivalent sequences up to the given partial order and symmetry Filter 5 (Partial Order): ∃σ2∈ Cpo,sym(σ1). T(σ2) = 1 ⇒ MO(σ1) = true

21. Independence Pattern Volker Fortströer - Learning by Testing The latter filter depends strongly on the concrete application domain,but there exist an pattern for realization: An expert specifies an independence relation Abstraction by replacing concrete identifiers with generic place holders σ is inspected if it contains independent subparts All re-orderings are computed Generic place holders are replaced again by concrete identifiers

22. Conclusion Volker Fortströer - Learning by Testing remember the two scenarios S1, S2 Number of Membership Queries: relatively great effect other scenario with 80 states has a total factor of 459.5 after applying the filters

23. Thanks for your attention! • References: • HardiHungar, Oliver Niese, Bernhard Steffen: Domain-Specific Optimization in Automata Learning. CAV 2003: 315-327 • Therese Berg, BengtJonsson, Martin Leucker, and MayankSaksena (SVV 2003), Mumbai (India): Insights to Angluin's learning. Electr. Notes Theor. Comput. Sci. 118: 3-18 (2005). Volker Fortströer - Learning by Testing