Building Trust into OO Components using a Genetic Analogy

1. Building Trust into OO Components using a Genetic Analogy Yves Le Traon and Benoit Baudry Yves.Le_Traon@irisa.fr

2. Summary • The problem: trustable components • Component qualification with mutation analysis • Genetic algorithms for test enhancement • A new model: bacteriological algorithms for test enhancement • A mixed-approach and tuning of the parameters • Conclusion

3. Safely Trust Testing for trust Object-oriented paradigm Reuse the better you test, the more trustable your component is

4. Testing for trust Specification Derived as executable contracts V & V: checking Implementation against Specification Trust based on Consistency Implementation

5. Design-by-contract What is the accepted domain of use ? What guarantees the component execution ? Executable Contracts Specifying the Component Implementing it... debug Building Unit Tests ImplementationOK Validating it... Trust No Trust Embedding Test Suite Self-Tests Test Quality Estimate Trusting Components? • The Design of a Trustable Component

6. How Can You Trust a Component? Specification Contract between the client and the component Measure of Trust based on Consistency V & V: checking Implementation against Specification (oracle) (e.g., embedded tests) Implementation

7. Component Trusting Components? • The point of view of the user... ? Components “off-the-shelf”

8. Component Trusting Components? • The point of view of the user... “replay” selftests 100% 85% 100% 55% Components “off-the-shelf”

9. selftest System tests Specification Test Component Impl. tests selftest tests Trusting Components? • Plug-in the component in the system Continuity Strategy Test dependencies between a component and its environment

10. Principle & Applications • Embed the test suite inside the component • Implements a SELF_TESTABLE interface • Component Unit Test suite = • Test data + activator • Oracle (mostly executable assertions from the component specification) • Useful in conjunction with • Estimating the Quality of the Component • Integration Testing

11. Example with UML SET_OF_INTEGER empty : Boolean full : Boolean has (x : Integer) : Boolean put (x : Integer) {pre: not full} {post: has (x); not empty} prune (x : Integer) {pre: has (x); not empty} {post: not has (x); not full}

12. class SET_OF_INTEGERS -- implements a simplified set of integers. creation make feature-- creation make -- allocate a set of integers ensure empty: empty; feature {ANY} -- status report empty: BOOLEAN -- is the set empty? full: BOOLEAN -- is the set full ? feature {ANY} -- access has(x: INTEGER): BOOLEAN -- is x in the set ? feature {ANY} -- Element changes put(x: INTEGER) -- put x into the set require not_full: not full; ensure has: has(x); not_empty: not empty; Example with Eiffel inherit SELF_TESTABLE , self_test prune(x: INTEGER) -- remove x from the set. require has: has(x); not_empty: not empty; ensure not_has: not has(x); not_full: not full; end-- class SET_OF_INTEGERS feature {TEST_DRIVER} -- for testing purpose only tst_add -- test add tst_prune -- test ‘prune’ and ‘add’ tst_suite -- Called by template method ‘self_test’ -- to execute all tst_* methods.

13. Execution of class SET_OF_INTEGERS Self Test Test sequence nb. 1 is add usable? put - empty - full -------------- - empty at create ... Ok - not empty after put ... Ok ..... >>>> DIAGNOSIS on class SET_OF_INTEGERS <<<< test(s) are OK. Method call statistics has : 37 full : 33 prune : 3 empty : 47 index_of : 40 put : 14 ....

14. Component qualification with mutation analysis Mutation operators Case study The problem of automating the test enhancement process

15. The Triangle View of a component Specification Contract between the client and the component Measure of Trust based on Consistency V & V: checking Implementation against Specification (oracle) (e.g., embedded tests) Implementation

16. Assessing Consistency • Assume the component passes all its tests according to its specification • Component’s implementation quality linked to its test & specification quality • How do we estimate test & spec consistency? • Introduce faults into the implementation • mutant implementations • Check whether the tests catch the faults • the tests kill the mutants

17. Limited Mutation Analysis put (x : INTEGER) is -- put x in the set require not_full: not full do 1 if not has (x) then 2 count := count + 1 3 structure.put (x, count) end-- if ensure has: has (x) - 1 not_empty: not empty end-- put Remove-inst

18. Class A Generation of mutants mutantA6 mutantA5 mutantA4 mutantA3 mutantA2 mutantA1 Selftest A Automated process Non automated process Test Execution Error detected Error not detected mutantAj  alive  mutantAj killed SelfTest OK ! Consider Aj as 3 Diagnosis Equivalent mutant 1 Enhance Selftest 2 Add contracts to the specification Incomplete specification Overall Process

19. About Living Mutants • What if a mutant is not killed? • Tests inadequate => add more tests • Specification incomplete => add precision • Equivalent mutant => remove mutant (or original!) • e.g., x<y ? x:y <=> x<=y ? x:y

20. d MS = m Component qualification • Trust estimating : Score of mutation analysis • d : Number of killed mutants • m : Number of generated mutants which are not equivalent

21. Quality Estimate = Mutation Score • Q(Ci) = Mutation Score for Ci = di/mi • di = number of mutants killed • mi = number of mutants generated for Ci • WARNING: Q(Ci)=100% not=>bug free • Depends on mutation operators (see next slide) • Quality of a system made of components • Q(S) = S di / S mi

22. 120/184 selftest System tests Specification 65/100 Test Impl. tests selftest 378/378 tests 234/245 System test Quality Q(S) = (120 + 65 + 378 + 234) (184 + 100 + 378 + 245) = 87,8 %

23. Type Description EHF Exception Handling Fault AOR Arithmetic Operator Replacement LOR Logical Operator Replacement ROR Relational Operator Replacement NOR No Operation Replacement VCP Variable and Constant Perturbation MCR Methods Call Replacement RFI Referencing Fault Insertion Component qualification Mutation operators

24. Mutation operators (1) • Exception Handling Fault • causes an exception • Arithmetic Operator Replacement • replaces e.g., ‘+’ by ‘-’ and vice-versa. • Logical Operator Replacement • logical operators (and, or, nand, nor, xor) are replaced by each of the other operators; • expression is replaced by TRUE and FALSE.

25. Mutation operators (2) • Relational Operator Replacement • relational operators (<, >, <=, >=, =, /=) are replaced by each one of the other operators. • No Operation Replacement • Replaces each statement by the Null statement. • Variable and Constant Perturbation • Each arithmetic constant/variable: ++ / -- • Each boolean is replaced by its complement.

26. Mutation operators (3) • Referencing Fault Insertion (Alias/Copy) • Nullify an object reference after its creation. • Suppress a clone or copy instruction. • Insert a clone instruction for each reference assignment.

27. Outline of a Testing Process • Select either: • Quality Driven: select wanted quality level = Q(Ci) • Effort Driven: Maximum #test cases = MaxTC • Mutation Analysis and Test Cases Enhancement • while Q(Ci) < Q(Ci) and nTC <= MaxTC • enhance the test cases (nTC++) • apply test cases to each mutant • Eliminates equivalent mutants • computes new Q(Ci)

28. id_mut EQ METHODE SOURCE MUTANT COMMENTAIRE 2 1 empty count = lower_bound – 1 count <= lower_bound – 1 jamais < 6 2 full count = upper_bound count >= upper_bound jamais > * count 2 16 3 index_of – loop variant count + 2 même or else or count structure count structure 24 4 index_of – loop until court test count := 0 ( nul) 30 5 make valeur défaut – 1 ( lower_bound ), upper_bound 45 6 make lower_bound, upper_bound redondance + 1 lower_bound, ( upper_bound ) 46 7 make lower_bound, upper_bound redondance + 1 test insuf. count = 60 I full count = – 1 test insuf. ( upper_bound ); 63 II full upper_bound; not has (x) true Spec inc. if then if then 72 III put not has (x) not false Spec inc. if then if then 75 IV put + 1 - Result ) 98 8 index_of – loop variant - Result) même - 1 - Result ) 99 9 index_of – loop variant - Result) même + 1 (count + 2 ) - 100 10 index_of – loop variant count + 2 - même – 1 (count + 2 ) - 101 11 index_of – loop variant count + 2 - même + 1 (count ) + 2 - 102 12 index_of – loop variant count + 2 - même – 1 (count ) + 2 - 103 13 index_of – loop variant count + 2 - même + 2 3 count - (count + - 104 14 index_of – loop variant même + 2 1 count - (count + - 105 15 index_of – loop variant même + 1 test insuf. > (count ) or 110 V index_of – loop until > count or NON EQUIVALENT EQUIVALENT A test report 119 mutants, 99 dead, 15 equivalents MS= 99/104=95%

29. equivalent mutants suppression remaining bugs correction automatic optimization of the initial tests set initial tests generation and bugs correction (tester’s work). Oracle function reconstruction Improve contracts measure contracts efficiency 2 contracts contracts contracts 1 contracts 3 4 contracts impl. MS= trust test test impl. impl. test MS=trust MS=trust contracts test impl. test impl. MS= trust test impl. Global Process Automated process 5 6

30. Oracle • Oracle par différence de comportements • Des traces • Des objets « programmes » • Oracle embarqués (contrats, assertions) • Interne au composant • Oracle associé aux données de test • Extérieur au composant • Oracle spécifique à la donnée de test

31. A short cases study • Building a self-testable library : date-time

32. p_date.e p_time.e p_date_time.e Total number of mutants 673 275 199 Nbr equivalent 49 18 15 Mutation score 100% 100% 100% Initial contracts efficiency 10,35% 17,90% 8,7% Improved contracts 69,42% 91,43% 70,10% efficiency First version test size 106 93 78 Reduced tests size 72 33 44 A short cases study • Results

33. A short cases study • Robustness of the selftest against an infected environment • p_date_time selftest robustness

34. Partial conclusion • An effective method to build (some level of) ‘trust’ • estimate the quality a component based on the consistency of its 3 aspects: • specification (contract) • tests • implementation • be a good basis for integration and non-regression testing • A tool is currently being implemented • for languages supporting DbC: • Eiffel, Java (with iContract), UML (with OCL+MSC) ...

35. Mutation for Unit and System testing • System testing • Combinatory explosion of the #mutants • Determination of equivalent mutant unrealistic • A subset of operators must be chosen to deal with these constraints • A strategy for injecting faults ? • Specific operators ?

36. Mutation for Unit and System testing

37. Mutation for Unit and System testing

38. Genetic algorithms for test enhancement

39. Test enhancement • Easy to reach a mutation score of 60% • Costly to improve this score • Test enhancement Genetic algorithms

40. Gas for test optimization • The issue : improving test cases automatically • Non linear problem • Gas may be adapted

41. Case study mutantA6 Class A mutantA5 mutantA4 mutantA3 Mutant Generation mutantA2 mutantA1 Autotest A 1st step i-th version of Autotest A Step i Automatic process Test Execution Automatic Improvement of selftest Non automatic process yes Genetic Algorithm New MS >Old MS no determinist Improvement of selftest 2 1 Look for equivalent mutants Diagnosis 3 Add contracts to the specification Incomplete specification

42. Praises population Class A Test6 mutantA3 mutantA7 mutantA9 mutantA4 mutantA8 mutantA2 mutantA5 mutantA6 mutantA1 Test Test5 Test1 Test3 Test4 mutantA10 Test2 Test enhancement

43. Test enhancement • The analogy • Test cases = predators = individuals • Mutants = population of prey

44. Genes Individual Operators reproduction crossover mutation Fitness function Test enhancement: Genetic algorithm

45. Test enhancement: Genetic algorithm • Fitness function Mutation score • Individual Tests cases • For unit class testing • Gene {Initialization, Function calls}

46. G = [I , S] G = [I , Smut] S = (m1(p1),…,mi(pi),…mn(pn)) Smut = (m1(p1),…,mi(pi mut),…mn(pn)) G1 = [I1 , S1] G2 = [I2 , S2] G3 = [I2 , S1] G4 = [I1 , S2] G5 = [I1 , S1 S2] G6 = [I2 , S2 S1] Test enhancement: Genetic algorithm • Operators • Crossover • Mutation for unit testing ind1 = {G1 1, ... G1 i, G1 i+1, .. G1 m} ind2 = {G2 1, ... G2 i, G2 i+1, .. G2 m} ind3 = {G1 1, ... G1 i, G2 i+1, .. G2 m} ind4 = {G2 1, ... G2 i, G1 i+1, .. G1 m}

47. Test enhancement: Genetic algorithm • Gene modeling for system testing • Must be adapted to the particular system under test • In the case of the studied system (a C# parser) • If there are x nodes N in the file a gene can be represented as follows G = [N1,…,Nx] • Mutation operator for system testing. The mutation operator, chooses a gene at random in an individual and replaces a node in that gene by another one: G = [N1,…, Ni,…, Nx]  Gmut = [N1,…, Nimut,…, Nx]

48. Test enhancement: Genetic algorithm The global process of a genetic algorithm

49. p_format.e +int_to-str_min(x,y : integer) Case study: unit testing p_date_const.e -Rfc_january -Rfc_february -Rfc_march -Rfc_april -Rfc_may -Rfc_june -Rfc_july -Rfc_august -Rfc_september -Rfc_october -Rfc_november -Rfc_december p_text_object.e hashable.e comparable.e p_time.e +is_equal(like current) +hash_code +set_hour(integer) +set_minute(integer) +set_second(integer) +set(x,y,z : integer) +is_am +is_pm +to_iso, out +to_rfc p_date_time.e +make +is_equal(like current) +hash_code +set_date(x,y,z : integer) +set_time(x,y,z : integer) +set(a,b,c,x,y,z : integer) +is_valid +is_local +is_utc +timezone_bias +set_timezone_bias(integer) -local_timebias -internal_tz +to_iso, out +to_iso_long +to_rfc -add_iso_timezone(string, boolean) p_date.e +is_equal(like current) +hash_code +set_year(integer) +set_month(integer) +set_day(integer) +set(x,y,z : integer) +is_valid_date(x,y,z : integer) +is_valid +is_leap_year -max_day(x :integer) +to_iso, out +to_iso_long +to_rfc Client Relation Inheritance Relation

50. p_date p_date_time p_time # of generated mutants 673 199 275 mutation score (%) 53 58 58 Case study: unit testing