Integration of Functional and Timed Testing of Real Time and Concurrent Systems

1. Integration of Functional and Timed Testing ofReal Time and Concurrent Systems V. Kuliamin, A. Petrenko, N.!Pakoulin, I.!Bourdonov, A.!Kossatchev kuliamin@ispras.ru ISP RAS, Russia

2. Classic testing scheme stimulus s1 Test System Target System r1 s2 r2 reaction s3 r3 Test = Test sequence s4 r4 s5 r5

3. Concurrency and real-time issues • Multiple access points • Simultaneous inputs • Multiple reactions • Spontaneous reactions • Temporal constraints Target System stimulus1 stimulus2 stimulus reaction1 reaction3 reaction reaction2 reaction stimulus Δt <= T0 reaction

4. Outline • Theoretical background • Practical considerations • Case studies

5. Proposed model IOSM – Input/Output State Machine • States S • Stimuli X • Reactions R • Transitions T • φ : T→ S×(X∪ R ∪ {i})× S ?b !z ?a Internal transition !y Spontaneous reaction !x Multiple reactions

6. Empty stimulus X = I∪ {e} Empty stimulus e models the knowledge of system that it has no inputs Asynchronous State Machine –IOSM with empty stimulus !x !y ?a ?e

7. B : X*→P(R*) ai↦ { xi } B : X*→P(R*) ai↦ { yjxk…xlyn : k+…+l = i } Behavior function !x !y !x !y ?a ?e ?a I.Burdonov, A.Kossatchev, V.Kuliamin. Asynchronous State Machines: Classification and Applications to Testing. Proceedings of ISP RAS, Moscow, 2003. In Russian

8. Modeling parallel inputs Plain concurrency : multistimulus is equivalent to some sequence of the stimulus contained ?a ?b How to model responses to all possible multistimuli? ?{a,a} ?{b,b} ?{a,b} ?{a,a,b} ?{a,b,b}

9. Outline • Theoretical background • Practical considerations • UniTesK test development method • Modifications for testing concurrent systems • Case studies

10. UniTesK test construction Behavior Model Testing Model ✕ Target System ✕  Test Input Construction Behavior Correctness Checking

11. Oracle  ✕ Mediator UniTesK test architecture Test sequence construction Test Engine Test sequence construction Test Action Iterator Test Scenario Specification Target system

12. Example of specifications specification intdeposit ( int x ) { pre { return x >= 0; } post { return balance == @balance + x; } }

13. Example of scenario scenario classAccountTest { AcountSpecification model; AbstractState getState() { return new IntState( model.balance ); } scenariodeposit() { iterate(int i = 0; i < 5; i++;) model.deposit(i); } scenariowithdraw() { iterate(int i = 0; i < 5; i++;) model.withdraw(i); } }

14. Test sequence construction Test Engine Test Action Iterator

15. Modifications for testing concurrency • Specifications • Asynchronous reactions • Temporal constraints • Testing • Providing parallel inputs • Collecting reactions • Checking the behavior as a whole

16. Specifying asynchronous reactions specification reactionUDPPacket UDPResponse ( ) { pre { return!ModelUDPPackets.isEmpty ( ); } post { return @ModelUDPPackets.clone().contains(UDPResponse) && !ModelUDPPackets.contains(UDPResponse); } }

17. Specifying temporal constraints specification reactionUDPPacket UDPResponse ( ) { pre { return!ModelUDPPackets.isEmpty ( ); } post { return @ModelUDPPackets.clone().contains(UDPResponse) && !ModelUDPPackets.contains(UDPResponse) && time – @ModelUDPPackets.clone().time(UDPResponse) <= 50*numberOfNodes; } }

18. Providing parallel inputs Multisequence is used instead of sequence of stimuli s14 s13 s12 s11 Target System s24 s23 s22 s21 s33 s32 s31

19. Collecting asynchronous reactions 11 21 31 Reactions form a partially ordered set r11 r12 Target System 12 22 32 r21 r22 r23 23 r31 r32 r33 33 Time

20.       ✕ Constructing model action series Stimuli Reactions Plain concurrency axiom 

21. Modified test architecture Test sequence construction Test Engine ✕  Synchronization Manager Test Action Iterator Serializer Oracle Mediator Target system Reaction Collector

22. Outline • Theoretical background • Practical considerations • Case studies

23. Case studies • Nortel Networks SOS – 1994-1996 • about 30 serious bugs found • MSR IPv6 – 2001-2002 • several serious bugs found • MS Mobile IPv6 – 2002-2003

24. References • http://www.ispras.ru/groups/rv/rv.html • I. Burdonov, A. Kossatchev, V. Kuliamin. Asynchronous State Machines: Classification and Applications to Testing. Proceedings of ISP RAS, Moscow, 2003. In Russian • V.Kuliamin, A.Petrenko, I.Bourdonov, and A.Kossatchev. UniTesK Test Suite Architecture. Proceedings of FME2002. LNCS2391, pp.!77-88, Springer-Verlag, 2002. • A. Petrenko. Specification-based Testing: Towards Practice.Proceedings of PSI’01, LNCS, Springer-Verlag. v.2244, pp. 287-300 • V. Kuliamin, A.Petrenko, I. Bourdonov, A. Demakov, A.Jarov, A.Kossatchev, S.!Zelenov. Java Specification Extension for Automated Test Development. Proceedings of PSI'01. LNCS, Springer-Verlag. v.2244, pp.301-307. • A. Petrenko, I. Bourdonov, A. Kossatchev, and V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT'01, India, Pune, 2001.

25. Thank you!