Formal Conformance Testing of Systems with Refused Inputs and Forbidden Actions

1. Formal Conformance Testing of Systemswith Refused Inputs and Forbidden Actions Igor Burdonov, Alexander Kossatchev, Victor Kuliamin ISP RAS, Moscow

2. Outline • Introduction Formal Testing with LTSes and ioco • Proposed conformance relation • LTS Completion • Conclusion

3. Formal Conformance Testing Real World Model World Specification Requirements !!! conforms ??? conforms formally modeling   System under Test Implementation derivation passes passes formally translation Model Test Suite Test Suite

4. Labeled Transition Systems Model world – LTSes • States • Transitions • Labels • Inputs ?a, ?b • Outputs !x, !y • Internal transitions τ • Initial state ?a !x !y ?b τ

5. ioco Conformance Relation • Testing abilities • Ability to provide inputs • Ability to observe outputs • Ability to observe quiescence • IiocoS for each δ-trace (trace with quiescence) σof S • an output is possible in I after σonly if its is possible in S • quiescence is possible in I after σonly if it is possible in S ?a !x δ ?b

6. Examples S I3 I1 I2 δ δ δ !y !y !y ?a ?a ?a ?b ?a ?b δ !x !y !x τ !x ?b δ δ ?a δ ?a ?a ?b ?b ?b ?b τ δ δ ?a ?a ?a δ I1ioco S I2ioco S I3ioco S

7. More Subtle Abilities I • Testing abilities • Ability to provide inputs • Ability to observe outputs • Ability to observe quiescence • Constraints • Implementation should be input-enabled • Tester has full control over action firing of implementation - synchronous testing( parallel composition semantics ) ?b !y ? !b ?a ?b T ?a τ !x ?a ?b ?b ?a

8. Practical Considerations • Some implementations are notinput-enabled • Launch button • Memory allocation • Testing in almost cases isasynchronous • Especially for distributed systems free(void *ptr) “ptr should be earlier returned by malloc(), calloc(), realloc()” (POSIX)

9. Asynchronous Testing Real World Model World Environment System under Test Implementation I I || C Context C Tester Model Tester T

10. Compositional Conformance • We may try to check that(I||C) ioco (S||C) • But ioco interacts with || in a bad way! • Unexpected conformance (I||C) ioco (S||C) while IiocoS • Unexpected non-conformance (I||C) ioco (S||C) while IiocoS

11. Bad Examples C – input and output queues S I1 I2 ?a !x ?a !x ?a !x !y ?b !y !x ?b ?b !z I1iocoS (I1||C) ioco (S||C) I2iocoS (I2||C) ioco (S||C)

12. Possible Ways out • ioco is asymmetric – I should be input-enabled, while S should not • ioco is preserved when S is input-enabled • Consider input-enabled specs only Too narrow • Replace S with S’ such thatIIiocoS  IiocoS’ ( S ~iocoS’)and S’ is input-enabled • Consider not input-enabled implementations and ioco analogue for them • Consider context-specific conformance relations Queues already considered, but others are also needed

13. Outline • Introduction • Proposed conformance relation Extension of ioco for non-input-enabled implementations • LTS Completion • Conclusion

14. Meaning of Undefined Inputs • ForbiddenShould not be provided • RefusedCan be provided and its refusal can be observed • IgnoredCan be provided, does nothing  should be specified

15. Proposed Model LTS with forbidden actions and refused inputs • Additional elements • Forbidden action γ • Refused inputs {?a}, {?b} • βγδ-traces Contain inputs, outputs, δ, input refusals, γ γcan only be the last symbol ?b {?b} ?a !x ?a,?b {?a} ?b !y τ ?a ?a δ γ ?b ?a,?b {?a,?b}, δ

16. Safe Traces βγδ-traces that cannot cause forbidden action to occur ?b !x Safe βγδ-traces: ?aδ?b{?b}?a ?a δ ?a ?b !x?aδ{?a} τ τ {?b} γ !y ?a ?b !x δ, {?a} τ ?b γ !x

17. Safety of Testing • Test should not destroy implementation • Safety hypothesis(weakening of input-enabledness hypothesis)Each safe βγδ-trace of S, which is also βγδ-trace of I, can be safely extended in I by each symbol safe after this trace in S • Such I is called safe for S

18. iocoβγδConformance Relation • Testing abilities • Ability to provide inputs • Ability to observe outputs • Ability to observe quiescence • Ability to observe input refusal • Constraints • Implementation is safe for specification • Tester has full control over action firing of implementation (synchronous testing)

19. iocoβγδConformance Relation IiocoβγδS Iis safe for S and for each safe βγδ-trace σof S • an output safe in S after σis possible in I after σonly if its is possible in S • quiescence safe in S after σis possible in I after σonly if it is possible in S • an input safe in S after σis possible in Ιafter σonly if it is possible in S • an input refusal safe in S after σis possible in I after σonly if it is possible in S

20. Examples S I1 I2 I5 I4 I6 I3 {?b} {?b} {?b} {?b} δ ?a ?a !y ?a !x !x !x ?b !x ?a {?a} {?a} {?a} ?b ,?b !y ?b !y ?a ?b !y !y τ {?a} {?a} ?b ?b δ {?a} ?a δ δ {?a} δ {?a} γ γ {?b} ?b ?b ?b ?b !y !y γ γ {?a,?b}, δ {?a,?b}, δ {?a,?b}, δ {?a,?b}, δ I2 is not safe for S I1iocoβγδ S I4iocoβγδ S I5iocoβγδ S I3iocoβγδ S I6iocoβγδ S

21. Test Case Derivation • Very similar to ioco • Differences • We should escape forbidden action – Use safe traces only • We should test input refusals – Try to provide an input and observe refusal as a deadlock

22. Examples S T1 T2 ?y,θ !b {?b} θ !x ?x,?y θ !a θ ?a !x !a θ !b {?a} θ ?x,?y ?x,θ ?y θ ?b !a ?x,?y !y τ θ θ !b θ δ ?a γ !b θ ?x,?y ?b θ !a θ {?a,?b}, δ !b θ

23. Outline • Introduction • Proposed conformance relation • LTS Completion • Trying to force composition to preserve conformance • Conclusion

24. Completion Operations • ioco is preserved when S is input-enabled • Replace S with S’ such thatIIiocoS  IiocoS’ ( S ~iocoS’)and S’ is input-enabled • Then,IiocoS  IiocoS’  C (I||C) ioco (S’||C) • S’ is more correct form of S – it can be used in any context

25. Demonic completion Ξ Tretmans et al. Ξ(S) S I Undefined inputs All inputs τ LTS !y !x τ ?a All outputs ?a ?a ?a !y ?a I ioco S !x !y ?a τ ?a I iocoΞ(S) May add more conforming implementations

26. Basic Completion • States Bc(S) = δ-traces of S • For each δ-traceσof S add the following transitions R(σ) means all δ-traces obtained from σby deleting some δ-s • Add σ σ<?a> marked with ?aif μ  R(σ)μ<?a> –δ-traceof S • Add σ σ<!x> marked with !xif μ  R(σ)μ<!x> –δ-traceof S • Add internal σ σ<δ>if μ  R(σ)μ<δ>–δ-traceof S and σdoes not end with δ • Add σ σ<!error> marked with !errorif μ  R(σ)μcannot be extended with any output, nor with δ !y !x τ ?a ?a

27. Proposed Completions Undefined inputs • Δ • Γ • For each I and S in ioco domainIiocoS IiocoβγδΔ(S) IiocoβγδΓ(S) Δ(S) Γ(S) S All inputs τ Bc(LTS) All outputs !y !x τ ?a Undefined inputs ?a ?a ?a Bc(LTS) γ γ !x !y τ ?a

28. Outline • Introduction • Proposed conformance relation • LTS Completion • Conclusion

29. Summary iocoβγδ ? Extension for non-input-enabled implementations Δ, Γ ioco Completion – construction of equivalent input-enabled spec

30. Announcement • Notation • U – non-empty subsets of reachable states of S • AU and z is safe in A  Az – states reachable from A by z and τ • AU and z is unsafe in A  Az = γ • AU and z is refusal set  Az – maximal stable subset of A that for each refusal in z it exists in each element of the subset • States of F(S) – U, U×{outputs, δ}, γ • Transitions • γ  γ marked with γ • If for symbol z Az is nonempty  A Azand (A, δ) Azmarked with z • If !y is safe and exists in Az where z is refusal set  internal A (Az,!y) and (Az,!y)(Az)!y marked with !y • If each !y is unsafe or not exists in nonemptyAz where z is refusal set  internal A (Az, δ) • If for ?a A?a is nonempty  (A, !y) A?amarked with ?a

31. Practical Implications • Not any specs are written correctly for use in component-based systems • The transformation rules can serve for specification adjustment • The rules can be rephrased into characteristics of correctly written specs

32. Contacts • Igor Burdonovigor@ispras.ru • Alexander Kossatchevkos@ispras.ru • Victor Kuliaminkuliamin@ispras.ru

33. Thank you!