Conformance Test Suites, Extensionally

1. Conformance Test Suites, Extensionally Arend RensinkUniversity of Twente Dutch Workshop on Formal Testing TechniquesUniversity of Twente13 September 2007

2. Automaton construction Non-deterministicalgorithm Approach Specification • Finite if spec is regular • On-the-fly construction • Completeness criteria • Direct reasoning & manipulation Current SuspensionAutomaton Test Suite Test Automaton New Formal Physical Closer to actual implementation ImplementationUnder Test Conformance Test Suites, Extensionally

3. Background • Specification s given by LTS • Distinguished inputs and outputs • Weakly convergent • Stable, output-free states: quiescent • Made explicit by -labelled self-loop • Suspension language Susp(s): traces with ’s • Implementation i modelled by IOTS • Input enabled • Conformance (ioco) i ioco s,8 w 2 Susp(s): Out(i after w) µ Out(s after w) Conformance Test Suites, Extensionally

4. explicit quiescence   press close press press press  open Example • Self-closing door • Wrong implementation: input: pressoutputs: open close close press press press open input enabled Error:2 Out(i afterpress¢open) Conformance Test Suites, Extensionally

5. press close press press press open close press press open Example (continued) • Alternative specs: non-determinism under-specificationfor press Conformance Test Suites, Extensionally

6. IOCO analyzed • The following are equivalent: • 8 w2Susp(s): Out(i after w) µ Out(s after w) • 8 w2Susp(s), x2U: w¢x 2 Susp(i) ) w¢x 2 Susp(s) • Susp(i) Å (Susp(s)¢U) µ Susp(s) • Susp(i) Å (Susp(s)¢Un Susp(s)) = ; • So the following are equivalent: • i conforms to s • i has no suspension trace accepted by Susp(s)¢Un Susp(s) Conformance Test Suites, Extensionally

7. Intuition • Susp(s): correct behaviour • Susp(s)¢U: Just one more response… • Susp(s)¢Un Susp(s): Forbidden paths Conformance Test Suites, Extensionally

8. openclose  open press close close press open press openclose  press close open close press  press close open open press Example Spec s Construct: Susp(s)¢Un Susp(s) final state close press press open press Example error trace: press¢open¢ Conformance Test Suites, Extensionally

9. -reduction & -saturation • Given a set of traces W µ L* • W- removes all traces from W with multiple successive ’s (of the form v¢¢+¢w) • W+ adds v¢+¢w for all v¢¢w 2 W • Facts: If W is a suspension language • W+ = W • (W-)+ = W • Improvement on previous result: • i conforms to s iff • i has no suspension trace accepted by Susp(s)-¢Un Susp(s) remove -cycles Conformance Test Suites, Extensionally

10. Test automata • Let T be an arbitrary automaton (with ’s) T accepts i,Susp(i) Å Lang(T) = ; • Properties defined for arbitrary T • soundness: i ioco s implies T accepts i • exhaustiveness: T accepts i implies i ioco s • Seen previously: • Susp(s)-¢Un Susp(s) sound & exhaustive for s • General criteria (main result, part 1): • T is sound for sif and only if Lang(T) µ (Susp(s)¢Un Susp(s)) ¢ L* • T is exhaustive for sif Lang(T) ¶ (Susp(s)- ¯Un Susp(s)) No output after  Conformance Test Suites, Extensionally

11. Main result (part 2) • Define: T is full if • v · w 2 Lang(T) implies v 2 Lang(T) • v¢a¢w 2 Lang(T) implies v¢¢a¢w 2 Lang(T) • v¢a 2 Lang(T) implies v 2 Lang(T) • v¢Uµ Lang(T) implies v 2 Lang(T) • For all i: fill(T) accepts i iff T accepts i • Necessary criterion for exhaustiveness: • If T is sound for s, thenT is exhaustive for sonly if Lang(fill(T)) ¶ (Susp(s)- ¯Un Susp(s)) fill(T) can be constructed from T Conformance Test Suites, Extensionally

12. openclose openclose   open open close press press close close close close press press press press open open open press press press Example Susp(s)¢Un Susp(s) Spec s Susp(s)-¯Un Susp(s) Straightforward automata constructions only Conformance Test Suites, Extensionally

13. Composition(Parallel composition,action refinement, …) Composition Compositionality Given: Possible with extensional characterisation sound/complete for Spec s Test T Desired: Spec s’ Test T’ Conformance Test Suites, Extensionally

14. Automaton construction Non-deterministicalgorithm From test automata to test suites Specification SuspensionAutomaton Test Suite Test Automaton Formal Physical ImplementationUnder Test Conformance Test Suites, Extensionally

15. From test automata to test suites • Equivalent testing power: • Test automaton T • Set {Ti}i with i Lang(Ti) = Lang(T) • Tretmans test cases t as test automata: • Cycle-free • One final accepting state (= fail) • One final non-accepting state (= pass) • Non-final states either stimulus or response • Tretmans’ algorithm generates {ti}i s.t.: • Lang(ti) µ Susp(s)¢Un Susp(s) for all ti • 9i: w2 Lang(ti) for all w2 Susp(s)¢Un Susp(s) Conformance Test Suites, Extensionally

16. Automaton construction Non-deterministicalgorithm Direct testing with automata Specification SuspensionAutomaton Test Suite Test Automaton Formal Physical ImplementationUnder Test Conformance Test Suites, Extensionally

17. Direct testing with automata • Given a test automaton T • Deterministic, with explicit quiescence • Corresponds to entire test suite • Synchronisation (product) of IuT and T: • Stimulus specified by T sent to IuT • Response of IuT mimicked in T • Test verdict • Final state of T reached: fail • Response of IuT absent in T: pass • No reachable final state in T: pass • Done testing: pass Conformance Test Suites, Extensionally

18. Conclusion • Test automata fill gap in existing theory • Yield precise conditions for completeness • Enable constructions on test suite • Work modulo Tretmans’ “test criteria”(not covered in this presentation) • Re-opens discussion on • Interaction between test and IuT • Input enabledness of test cases Conformance Test Suites, Extensionally