UniTesK: Model Based Testing in Industrial Practice

1. UniTesK:Model Based Testing in Industrial Practice Victor Kuliamin Alexander Petrenko Alexander Kossatchev Igor Burdonov Institute for System Programming Moscow, Russia

2. Outline • Origin of UniTesK Method • UniTesK Manifesto • UniTesK Solutions • Case Studies

3. Origin of UniTesK Method • 1994 – 1996ISP RAS – Nortel Networks project onfunctional test suite development for Switch Operating System kernel • Few hundreds of bugs found in the OS kernel, which had been in use for 10 years • KVEST technologyAbout 600K lines of Nortel code tested by 2000 BUT: Failed to be introduced in Nortel processes

4. UniTesK Test Construction Testing Model Behavior Model Coverage Model System under Test Test Input Behavior Correctness Checking

5. Test Development Process Interface Definition Design Requirements Specification Development Behavior Specifications Scenario Development Test Scenarios Test Quality Criteria Mediator Development Mediators System under Test Test Execution Test Reports Test Results Analysis

6. Engineering Problems To increase productivity, but not to loose flexibility • How to simplify transformation of requirements into formal specifications? • How to minimize human involvement in test development, but make it possible in necessary points? • How to support a large variety of problem domains and test completeness criteria? • How to decouple tests and implementation, but keep them working together? • How to measure testing quality without implementation? To make relation between them more clear To make easier their usage for ordinary software engineers To make possible early start of test development To make specifications and tests more reusable

7. UniTesK Manifesto • Maximum simplification of MBT techniques • Automation where possible • Accommodation to current industrial practice • Integration with widely used tools • Integration with widely used languages

8. Tools • J@T • CTesK • Ch@se

15. Oracle  ✕ Mediator UniTesK Test Architecture Test action construction Test Engine Test action construction Test Action Iterator Test Scenario Specification System under Test

16. Example Client Data Management System: • A number of clients • Clients can be related as parent-subsidiary • Client can have only one parent, but may have several subsidiaries • No cycles along parent-subsidiary links • When parent is removed, all its subsidiaries should be removed

17. Example : Interface class ClientManager { Client addClient ( String name ); boolean removeClient ( Client client ); } class Client { boolean addSubsidiary ( Client client ); }

18. Example : Client Data class Client { String name; ClientSpecification parent = null; HashSet children = new HashSet(); invariant ParentChildLink() { if(parent != null&& !parent.children.contains(this))return false; Iterator j = children.iterator(); while(j.hasNext())if(((Client)j.next()).parent != this)return false; return true; } invariant NoCyclesAlongLinks() { Client r = this; do{r = r.parent;if(r == this)return false; }while(r != null); return true; } }

19. Example : addClient() Method class ClientManager { specification Client addClient ( String name ) updates clients.? { post { if(name == null|| clients.containsKey(name)) { // Client cannot be created returnaddClient == null&& clients.equals(@clients.clone()); } else { // Client can be created HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null&& addClient.name.equals(name) && addClient.parent == null&& addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); } } } }

20. Test Coverage Criteria post { “Single branch” branches if(a || b) if(a || b) if(a || b) predicates ( a || b ) !( a || b ) disjuncts a !a && b !a && !b branch “Single branch” branch “Single branch” return }

21. Example : addClient() Method class ClientManager { specification Client addClient ( String name ) updates clients.? { post { if(name == null|| clients.containsKey(name)) { branch"Client cannot be created"; returnaddClient == null&& clients.equals(@clients.clone()); } else { branch"Client can be created"; HashMap oldClients = (HashMap)clients.clone(); oldClients.remove(name); return addClient != null&& addClient.name.equals(name) && addClient.parent == null&& addClient.children.isEmpty() && clients.get(name) == addClient && oldClients.equals(@clients.clone()); } } } }

22. Automaton from Coverage Goals parameters operation domain 2 3 coverage goals 1 states

23. Implicit Automata Description • Implicit specifications cannot be resolved • To decrease effort they should not be resolved • Huge automata can be described shortly

24. Test Scenarios User can describe an automatonimplicitly in the form of Test Scenario Functions: • Provide the current state identifier • Provide the next admissible stimulus in this state

25. Test Scenario : State scenario class ClientManagerTestScenario { ClientManager target; AbstractGenState getGenState() { IntToIntMapGenState genstate = new IntToIntMapGenState(); int n = 0, m = 0; Iterator j = objectUnderTest.clients.keySet().iterator(); while(j.hasNext()) { String s = (String)j.next(); n = Integer.parseInt(s); if(((Client)target.clients.get(s)).parent == null) genstate.setValue(n, 0); else { m = Integer.parseInt(((Client)target.clients.get(s)).parent.name); genstate.setValue(n, m); } } return genstate; } }

26. Test Scenario : Stimuli scenario class ClientManagerTestScenario { scenario boolean addClient() { iterate(int i = 0; i < numberOfClients; i++; ) { String name = (i == 0)?(null):("" + i); if(target.precondition_addClient( name ) ) target.addClient( name ); } return true; } scenario boolean removeClient() { iterate(int i = -1; i < numberOfClients; i++; ) { Client client = (i < 0)? ClientMediator.create(new ClientImpl("1")) :(Client)target.clients.get(("" + i)); if(target.precondition_removeClient( client ) ) target.removeClient(client ); } return true; } }

27. Test Execution : First Stage Test Engine Test Scenario

28. Factorization

29. Testing Concurrency : Modeling ?a ?b How to model responses on all possible multistimuli? ?{a,b} ?{a,a} ?{b,b} ?{a,a,a} ?{a,a,b} Plain concurrency axiom : reaction on multistimulus is the same as on some sequence of its stimuli

30. Asynchronous Reaction Specification specification reaction UDPPacket UDPResponse ( ) { pre{ return !ModelUDPPackets.isEmpty ( ); } post { return (@ModelUDPPackets.clone()).contains ( UDPResponse ) && !ModelUDPPackets.contains ( UDPResponse ); } }

31. Providing Concurrent Inputs Multisequence is used instead of sequence of stimuli s14 s13 s12 s11 System under Test s24 s23 s22 s21 s33 s32 s31

32. Collecting Asynchronous Reactions 11 21 31 Reactions form a partially ordered set r11 r12 System under Test 12 22 32 r21 r22 r23 23 r31 r32 r33 33 Time

33.       ✕ Evaluation of Reactions Stimuli Reactions Plain concurrency axiom 

34.  ✕ UniTesK Test Architecture, 2 Test input construction Test Engine ✕  Synchronization Manager Test Action Iterator Serializer Oracle Mediator System under Test Reaction Collector

35. UniTesK Tools • J@T (C++ Link)Java (C++) / NetBeans, Eclipse (plan) • CTesKC / Visual Studio 6.0, gcc • Ch@seC# / Visual Studio .NET 7.1 • OTKSpecialized tool for compiler testing

36. Technology Transfer • Forms • Training • Pilot projects • Joint projects • Project supervising • Successful transfers • Lanit-TercomCTesK, Feb 2002 • Luxoft (IBS group)J@T, Jul 2003 • IntelOTK, Nov 2003

37. Case Studies • IPv6 implementations - 2001-2003 • Microsoft Research • Mobile IPv6 (in Windows CE 4.1) • Oktet • Intel compilers - 2001-2003 • Web-based client management system in bank • Enterprise application development framework • Billing system

38. References • V. Kuliamin, A. Petrenko, N. Pakoulin, I. Bourdonov, and A. Kossatchev. Integration of Functional and Timed Testing of Real-time and Concurrent Systems. Proc. of PSI 2003. LNCS, Springer-Verlag, 2003. • V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. UniTesK Test Suite Architecture. Proc. of FME 2002. LNCS 2391, pp. 77-88, Springer-Verlag, 2002. • A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’01, India, Pune, 2001 • I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing. "Programmirovanije", 2000, No. 2 (in Russian). Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61-73 (English version) • I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp. 608-621 • http://www.ispras.ru/groups/rv/rv.html

39. Contacts Victor V. Kuliamin Alexander K. Petrenko E-mail: kuliamin@ispras.ru, petrenko@ispras.ru 109004, B. Kommunisticheskaya, 25 Moscow, Russia Web: http://www.ispras.ru/groups/rv/rv.html Phone: 007-095-9125317 Fax: 007-095-9121524

40. Thank you!