OO System Testing

1. OO System Testing Automatic test synthesis from UML models

2. Outline • System testing • Behavioral test patterns • Generating behavioral test patterns

3. Enhanced use cases model Simulation model Test casessynthesis Test objectivesgeneration General context Textual Requirements

4. Context & Objectives • Aim: Generating automatically test sequences from the use cases model • Dependencies between the use cases • Idea: • Define the input and output of a use case • Express the dependencies between the use cases (declarative approach) • Define criteria to generate relevant tests

5. Test système et UML • Meeting distribué

6. The use case scenarios • High level, simple, incomplete • Wildcards for genericity • Example: :Server :Server x:user x:user enter(*, x) enter(*, x) ok nok Exceptional case (a) Nominal case (b) (b) Enter use case scenario (x is a scenario parameter)

7. Cas d’utilisation Scénarios Nominaux Scénarios Exc. Rares Scénarios Exc. Echecs A Planifier NA1, NA2 EA1, EA2 B Ouvrir NB1 EB1, EB2 I Clôturer NI1 RI1 C Consulter NC1 EC1 D Entrer NC1 RD1 ED1, ED2 E Demander la Parole NE1 EE1 G Parler NG1, NG2 RG1 EG1, EG2 H Sortir NH1 EH1 F Donner la Parole NF1 EF1, EF2 Test système et UML

8. Test système et UML • Critère minimum: Couvrir chaque scénario avec une donnée de test • Ici 27 cas de test • Critère de couverture des combinaisons de use-cases • Prérequis : un diagramme d’activité des use-cases

9. A swimlane per actor Visually significant Within UML notations Suitable to apply algorithms Activity diagrams • Difficult to build • Hard or impossible to express certain behaviors • Not suitable for use cases shared by actors

10. Test système et UML

11. Test système et UML • Critère : tester chaque scénario de chaque cas d’utilisation dans chaque séquence nominale élémentaire (1 passage par boucle)

12. é ù N A 1 é ù N ê ú B 1 é ù é ù é ù é ù é ù é ù N N N N N N N ê ú [ ] [ ] ê ú A 2 A 1 A 1 I 1 A 1 I 1 H 1 · · · E N N ê ú ê ú ê ú ê ú ê ú ê ú ê ú ê ú B 1 1 1 B B E N N R N R E ë û ë û ë û ë û ë û ë û ê ú A 1 A 2 A 2 I 1 A 2 I 1 H 1 E ê ú ë û B 2 E ë û A 2 Test système et UML • Données de tests à générer pour la séquence A.B.I.H => 4 + 2x3 + 2x1x2 + 2x1x2x2 = 22 cas de test doivent être générés via ce « pattern » Cible de Test A B I H Combinaison des Cas de · · · Test

13. é ù é ù é ù é ù é ù N N N N E [ ] [ ] [ ] 1 1 1 1 1 A H A A A N N E ê ú ê ú ê ú ê ú ê ú 1 1 1 B B B N E N N E ë û ë û ë û ë û ë û 2 1 2 2 2 A H A A A Test système et UML • En évitant les redondances: ordonner les « cibles de test » => 10 cas de test doivent être générés via ce « pattern » Cible de Test H I B A Combina ison des Cas de · · · · · · N R I1 I1 Test

14. Benefits from test patterns • Generation of product specific test cases from product independant test patterns • But tedious to build test patterns especially for « basis tests » • Idea : being able to build automatically significant sets of test patterns

15. How to exploit use cases ordering ? • Generate pertinent paths of use cases • In order to reach a test criterion • Issues: • An algorithm to assemble the use cases taking into account the pre and post conditions • Defining pertinent test criterions

16. Conclusion • From early modeling to test cases : • From reusable and generic test pattern • To concrete test cases, specific to each product • Two ways of selecting test patterns: • manually (qualitative approach) • driven by use cases sequential dependencies (quantitative approach)

17. Requirements by Contracts allow Automated System Testing Clémentine Nebut Franck Fleurey Yves Le Traon Jean-Marc Jézéquel IRISA-INRIA Université de Rennes 1 FRANCE

18. Param Pre UC1 Post Param UC2 Post Pre • Enhance the use cases with parameters and contracts • Deduce a model of the correct orderings of use cases • Result: a set of test objectives satisfying a certain criterion • Cover this model with an adequate criterion Test objectives {UC1(p1,p2), UC3(p2),UC4(p1)} {UC3(p1),UC1(p2,p2)} … Overview of the Method

19. Example: a virtual meeting server VirtualMtg enter plan consult open leave manager close speak user hand over moderator connect ask

20. Outline • A contract language for the use cases • Test generation from the enhanced use cases • Building a model of the valid orderings of use cases • Definition of criteria • Results on a case study • Conclusion

21. Outline • A contract language for the use cases • Test generation from the enhanced use cases • Building a model of the valid orderings of use cases • Definition of criteria • Results on a case study • Conclusion

22. Contracts: logical expressions on predicates • Precondition, postcondition • Predicate= a name, an arity, and a semantic + set of typed parameters • Ex: created(m:meeting) manager(u:participant,m:meeting) • First-order logic: • classical boolean operators (and, or, implies, not) • + quantifiers (forall, exists)

23. Identifying the use case parameters • Parameters = Actors or Business concepts • Used to represent the inputs of the use case • May be reified during the design process • In the virtual meeting example: • Types: participant, meeting • Examples: • UC Enter (p: participant, m: meeting) • UC HandOver(p1: participant, p2: participant, m: meeting)

24. OPEN(u1,m1);CLOSE(u1,m1) is a correct sequence Contracts: example #use case OPEN UC open(u:participant;m:mtg) pre created(m) and moderator(u,m) and not closed(m) andnot opened(m) and connected(u) post opened(m) #use case CLOSE UC close(u:participant; m:mtg) pre opened(m) and moderator(u,m) postnot opened(m) and closed(m) and forall(v:participant) {not entered(v,m) and not asked(v,m) and not speaker(v,m) }

25. Outline • A contract language for the use cases • Test generation from the enhanced use cases • Building a model of the valid orderings of use cases • Definition of criteria • Results on a case study • Conclusion

26. Use Case Transition System (UCTS) : definition • Quadruple M=(Q,q0,A,) • Q = non empty set of states • State = set of instantiated predicates • q0 = initial state • A = alphabet of actions • action= instantiated use case •   Q x A x Q = transition function • Instantiated use case (resp. predicate) = use case (resp. predicate) with its operational parameters

27. open(p1,m1) enter(p1,m1) connected(p1), created(m1), manager(p1,m1), moderator(p1,m1)opened(m1) entered(p1,m1) close(p1,m1) connected(p1), created(m1), manager(p1,m1), moderator(p1,m1)closed(m1) close(p1,m1) UCTS: example connected(p1), created(m1), manager(p1,m1), moderator(p1,m1) connected(p1), created(m1), manager(p1,m1), moderator(p1,m1)opened(m1)

28. Building algorithm: principles • Need of: • the enumeration of each actor and business concept for example, 3 participants and 2 meetings • an initial state given in terms of true predicates • The algorithm tries to apply successively all the instantiated use cases • from all the unvisited nodes • until all the nodes are visited

29. UCTS in the testing context • Path of the UCTS = correct sequence of instantiated use cases = test objective • Aim: Finding a set of paths in the UCTS covering the requirements • Necessity to find test criteria

30. Test criteria • General objective: generate • few but efficient test objectives • short test objectives • Structural criteria • All edges • All vertices • All instantiated use cases • All instantiated use cases and all vertices • Semantical criterion • All precondition terms

31. All precondition terms criterion • Find all the combinations of terms that make the precondition true • Examples: UC over(u:participant; m:mtg) pre speaker(u,m) or moderator(u,m)  try to apply Over with: • speaker(u,m) and not moderator(u,m) • not speaker(u,m) and moderator(u,m) • speaker(u,m) and moderator(u,m)

32. Robustness tests • Generate paths leading to an invalid application of the use case • Exercise correctly the system, then make a non specified action • Criterion: similar to all precondition term (all the terms that makes the precondition fail) Correct path . . . Non specified action

33. Outline • A contract language for the use cases • Test generation from the enhanced use cases • Building a model of the valid orderings of use cases • Definition of criteria • Results on a case study • Conclusion

34. VirtualMtg enter plan consult open leave manager close speak user hand over moderator connect ask Results: virtual meeting case study Dead code Robustness code (w.r.t. env) Robustness code (w.r.t. spec) Nominal code

35. Test objective Test objective Test objective Test objective Test case Test case Test case Test case Experimental protocol Test synthesis System under test Code coverageTool Code coverage measures

36. Comparaison between criteria efficiency % of covered code 100 90 80 70 60 Robustness test cases Functional test cases 50 All edges All All IUC All IUC All vertices and all precond vertices terms

37. Results

38. Relative efficiency of the test cases # covered statements # test cases 20 16 12 8 4 0 All edges All All IUC All IUC All vertices and All precond vertices terms

39. Dead code 9 % Nominal code 65% Robustness code w.r.t. spec 8% Robustness code w.r.t. env 18% Covered code Code repartition and covered code

40. Outline • A contract language for the use cases • Test generation from the enhanced use cases • Building a model of the valid orderings of use cases • Definition of criteria • Results on a case study • Conclusion

41. Conclusion • Test objectives automatically generated • from use cases enhanced with contracts • with test adequacy criteria • The test generated are efficient in terms of code coverage • Prototype tool supporting the approach

42. Future work • Use scenarios to transform test objectives into test cases • use of a test synthesis tool • Give test priorities to the use cases • adapt the criteria to the priorities • On-the-fly computation of the test objectives

43. From system level test patterns to specific test cases : application to product-Line architectures

44. Product Line architectures • A product line : a set of systems which share a common software architecture and a set of reusable components. • Building a product line aims at developing once the common core of a set of products, and to reuse it for all the products. • Defining a product family • Variants and commonalities • Reuse assets • For our purpose: specify behavioural test patterns, that become reusable “test assets” of the product-line

45. Product Line architectures: a key challenge • Use case scenarios cannot be used directly for testing • Generic and incomplete. • Parameters are not known, nor object instances (scenarios concern roles). • Specify the general system functionality without knowing – at that stage - the exact sequence calls/answers. • Generating test cases from such test patterns for a given UML specification is thus one of the key challenges in software testing today.

46. PL • Variants • optional, when a component can be present or not, • alternative, when at a variation point, one and only one component can be chosen among a set of components, • multiple, when at a variation point, several components can be chosen among a set of components. • All the variants must appear in the architecture but not all the possible combination of variants • Extracting a product from the global product line architecture : product instantiation

47. Testing product lines • Benefiting from the PL specificities • Testing commonalities • Deriving tests according to the variants • Specific tests • Reusing tests • Building test assets • Defining test independently from the products • Using generic scenarios • Deriving product-specific test cases from those generic scenarios

48. Product Line architectures: example • Virtual Meeting Server PL offers simplified web conference services: • it aims at permitting several kinds of work meetings, on a distributed platform. ( general case of a ‘chat’ software). • When connected to the server, a client can enter or exit a meeting, speak, or plan new meetings. • Three types of meetings • standard meetings where the client who has the floor is designated by a moderator (nominated by the organizer of the meeting) • democratic meetings which are standard meetings where the moderator is a FIFO robot (the first client to ask for permission to speak is the first to speak) • private meetings which are standard meetings with access limited to a defined set of clients.

49. VirtualMtg enter plan consult open leave manager user close speak hand over moderator connect The Virtual Meeting Example • Connection to the server • Planning of meetings • Participation in meetings • Moderation of meetings Virtual meeting use case diagram

50. Product Line architectures: example • Due to marketing constraints, the Virtual Meeting PL is derivable into three products • a demonstration edition: standard and limited • a personal edition: any type but limited • an enterprise edition: any type, no limitations • Two variants : type (multiple)and participants limitation (optional) • (also OS, languages, interfaces etc.)