The improvement of Mutation Analysis By Mutant Propagation Ability

1. The improvement of Mutation Analysis By Mutant Propagation Ability • Speaker: Hsiu-Yi Lin • Advisor: Chun-Yao Wang Date: 2011.02.25

2. Outline • Introduction • Functional Verification • Verification Quality Measuring • Functional Qualification • Mutation Analysis (MA) • The Limitation of MA • Previous Work • Cost Reducing Techniques • Mutant Reduction Techniques • Proposed Approach • Mutant Propagation Ability Estimation • Assignment Decision Diagram • Observation Point Insertion • The Flowchart for the algorithm • Experimental Results • Future work

3. Outline • Introduction • Functional Verification • Verification Quality Measuring • Functional Qualification • Mutation Analysis (MA) • The Limitation of MA • Previous Work • Cost Reducing Techniques • Mutant Reduction Techniques • Proposed Approach • Mutant Propagation Ability Estimation • Assignment Decision Diagram • Observation Point Insertion • The Flowchart for the algorithm • Experimental Results • Future work

4. Functional Verification • Functional verification is the step to ensure that the specifications and/or implementations of the design at various abstraction levels are in accord with the design intent. Transformation Specification HDL Coding Functional Verification

5. Functional Verification • Simulation-based simulation • While formal approaches are suffered from the complexity of execution, simulation has still a dominant role in the verification of the functional correctness of electronic and embedded systems.

6. Verification Quality Measuring (1/2) • Completeness Problem • “When can one claim that the verification is complete?” This is a perpetual and still unanswerable question. • Exhaustive simulation is impossible • Considering a logic block with 64-inputs, the combinatorial possibilities for its input space reach 16 x 10 18 billion. It is impossible to simulating them within a reasonable time.

7. Verification Quality Measuring (2/2) • Coverage Metric • To avoid simulating all tests exhaustively, coverage metric are developed to guide the selection of tests for simulation. • Coverage metrics can be classified into two categories • Structural coverage • Functional coverage

8. Structure coverage & Functional coverage • Structural Coverage (code coverage) • Most of them only concern the design code is activated, they don’t guarantee the design bugs will be propagated and detected. • Functional Coverage • It usually involves the interpretation of functionality and the related measurements from the specification. However, The extra effort to write the test plan is a major drawback. In addition, the implementation is subjective, and it is no way to define completion.

9. Functional Qualification • The limitation to the coverage metric • Because of the drawbacks of Structure & Functional coverage, we need a adequate metrics to track the progress of verification. • A new verification technique known as functional qualification addresses this problem. • Fault-based verification technique • To apply functional qualification, we can use fault-based verification technique, and consider all three portions of the verification process – activation, propagation, and detection.

10. Fault-based Verification (1/2) • In manufacturing stage • Stuck-at fault model at gate-level is used to guide the selection of product test data for exposing defects during the manufacturing processes. • In design stage • Theoretically, we can translate higher level faults to gate-level then apply ATPG methods to generate the test data. However, this mapping imposes high complexity and inefficiency, especially with complex design.

11. Fault-based Verification (2/2) • OCCOM ( Observability-Based Code Coverage Metric ) • By F. Fallah- DAC1998 • Compute the probability that an effect of the fault would be propagated. • Insert tags into the design when simulating, and observe the propagation result. • Show the problem with coupling code coverage and consider the propagation ability.

12. Mutation Analysis (1/3) • Mutation Analysis (MA) is a fault-based test data selection technique. • History • Mutation Analysis can be sued for testing software. The paper introducing mutation analysis was written by R. A. DeMillo, R. J. Lipton, and F. G. Sayward in 1978.

13. Mutation Analysis (2/3) • Implementation • Analyze small behavioral change to a software program, and intend to find weakness in the functional testing of a program. • Similar to manufacturing test • Looks for a change in values seen on an output • Define a fault model which is similar to designer errors.

14. Mutation Analysis (3/3) • Problem • The number of such potential faults for a given program is enormous; it is impossible to generate mutants representing all of them.

15. Fundamental Hypotheses of MA • Mutation Analysis targets only a subset of these faults, with the hope that these will be sufficient to simulate all faults. This theory is based on two hypotheses: • Competent Programmer Hypothesis (CPH) • The programmers or engineers write code that is close to being correct. • Coupling Effect • A test that distinguishes the good version from all its mutants is also sensitive to more complex errors.

16. Terminology • Mutant • The “small change” we make in the code (as the “fault” we know in ATPG). • Kill • If we detect the error in the simulation result, it is said we kill the mutant. • Living Mutants • After we tested all verification data, the mutants that still existed are living mutants • Living Mutants show that there are some weaknesses in our test pattern.

17. Terminology • Non-Activated Mutants (NAM) • No stimulus reaches the mutation. • Non-Propagated Mutants (NPM) • The mutants will be activated and generate the error events. These error events cannot be propagated to the observation point (ex: assertion checkers) in the testbench. • Non-Detected Mutants (NDM) • The error events have been propagated to the observation point, but cannot be detect by the checker.

18. Mutation Example • Original program code a = b or c Mutated program a = b and c use “and” to replace “or” is a mutant • Check whether we catch the difference in the simulation result

19. The Verification Environment Improvement by Mutation Analysis • Final goal • Identify the weakness in the verification environment, then generate the test data and improve the checker to modify them. • Experimental goal • Kill the mutants as many as possible.

20. The Limitation of Mutation Analysis (1/2) • Limitation 1: High computational cost of executing the enormous number of mutants against a test set. • Considering a design under test with L lines and M mutation operators. Assuming the simulation cost linear to the design size, We need O(M * L) = O(L2) to simulate the DUV for each test case. Now we have T test cases, so we can evaluate the total cost is O (T * L2)

21. The Limitation of Mutation Analysis (2/2) • Limitation 2: It is related to the amount of human effort involved in using Mutation Analysis. • Analyze the result • To clarify the reason for the living mutants • Equivalent mutants • It is similar to the “redundant fault” of manufacturing fault model.

22. Previous Work • For limitation 1 • Cost Reducing Techniques • For limitation 2 • Just a few research

23. Outline • Introduction • Functional Verification • Verification Quality Measuring • Functional Qualification • Mutation Analysis (MA) • The Limitation of MA • Previous Work • Cost Reducing Techniques • Mutant Reduction Techniques • Proposed Approach • Mutant Propagation Ability Estimation • Assignment Decision Diagram • Observation Point Insertion • The Flowchart for the algorithm • Experimental Results • Future work

24. Cost Reducing Techniques • Most cost reduction techniques have been proposed. These techniques are classified into two types • Reduction of the generated mutants • Reduction of execution cost.

25. Mutant Reduction Techniques • Mutant Sampling (1980) • Choose a small subset of mutants from the entire set. • Mutant Clustering (2008) • Chooses a subset of mutants using clustering algorithm. • Selective Mutation (1990) • Seek to find a small set of mutation operators that generate a subset of all possible mutants than others. • Higher Order Mutation (2008) • First order mutants(FOMs) vs.. higher order mutants(HOMs)

26. Execution Cost Reduction Techniques (1/2) • Strong Mutation Weak and Firm Mutation • Strong: for a given program p, a mutant m of program p is said to be killed only if mutant m gives a different output from the original program p. • Weak: a program p is assumed to be constructed from a set of components C = {c1, ..., cn}. Suppose mutant m is made by changing component cm, mutant m is said to be killed if any execution of component cm is different from mutant m.

27. Execution Cost Reduction Techniques (2/2) • Firm Mutation: the “compare state” of Firm Mutation lies between the intermediate states after execution (Weak Mutation) and the final output (Strong Mutation). • Except the different types of mutation, there are other important reduction techniques such as Compiler-Based Technique or Schema Generation. But these techniques are related to the run-time optimization, we won’t concern them in our topic.

28. Strong Mutation Weak and Firm Mutation Line 1 Line 2 Line 3 Line 4 … … … Line 101 Line 102 Output = … Detect Error Mutatnt Observation point  Weak Propagation!

29. Strong Mutation Weak and Firm Mutation Line 1 Line 2 Line 3 Line 4 … … … Line 101 Line 102 Output = … Mutatnt Strong Propagation! Detect Error Observation point 

30. Strong Mutation Weak and Firm Mutation Line 1 Line 2 Line 3 Line 4 … … … Line 101 Line 102 Output = … Mutatnt Detect Error Observation point  Firm Propagation!

31. Outline • Introduction • Functional Verification • Verification Quality Measuring • Functional Qualification • Mutation Analysis (MA) • The Limitation of MA • Previous Work • Cost Reducing Techniques • Mutant Reduction Techniques • Proposed Approach • Mutant Propagation Ability Estimation • Assignment Decision Diagram • Observation Point Insertion • The Flowchart for the algorithm • Experimental Results • Future work

32. Overview of Our Approach • Create a new mutant analysis method: Mutation Propagation Ability(MPA) Analysis • Estimate the “propagation ability” for each mutant in the design. • We reduce the execution effort by adding observation points and determine the state between the intermediate states, similar to firm mutation, . • MPA analysis is a static method, which is not a burden. • Automatic select the internal observation signals • Although living mutants represent the weakness of the test-bench, designers need more practical information to modify their test plan. • The signals with low MPA will be the candidates for debugging.

33. Mutant Propagation Ability Estimation • Mutation Propagation Ability(MPA) • Represent the probability of error occurring in each signal by a single mutant. • The signal with higher probability stands for the error will be detected with the higher probability. • The MPA on the location of the mutant is 1, and decrease from there to the output. On the other hand, MPA=0 means the mutant won’t effect that signal.

34. Mutant Propagation Ability Estimation • MPA Estimation Rule in RTL design • Transfer the RTL to the Assignment Decision Diagram format. • Derive the MPAs for each operation in every data path. • Consider the feedback condition, so it can support sequential circuit.

35. Example x = a & b y = c & d z = x & y Injecting mutant 1 a MPA1= 1 & MPA1=0.5 x b MPA1= 0 & z MPA1=0.25 c y & MPA1= 0 MPA1=0 d MPA1= 0

36. Example x = a & b y = c & d z = x & y a MPA1= 1 MPA2= 0 & MPA1=0.5 MPA2=0 x b MPA1= 0 MPA2= 0 & z MPA1=0.25 MPA2=0.5 Injecting mutant 2 c y & MPA1= 0 MPA2= 0 MPA1=0 MPA2=1 d MPA1= 0 MPA2= 0

37. MPA Estimation Rule • For any two input operation “out = (a op b)” • MPAa/b: MPA of signal a/b • MPA’a/b: (1-MPA) of signal a/b MPAout = MPAa‧MPA’b‧(1 – P(a op b = a’ op b) ) + MPA’a‧MPAb‧(1 –P(a op b = a op b’) ) + MPAa‧MPAb‧ (1 – P(a op b = a’ op b’) ) op a out b

38. MPA Estimation Example • For operator “+”, ausumeMPAa = 1, MPAb= 0 Assume a’ = a + ∆a (∆!=0) , b’ = b + ∆b(∆!=0) P(a’+b=a+b) = P(a+b=a+b’) = P(a+∆a+b=a+b) = 0 P(a+b=a’+b’) = P(a+∆a+∆a+b=a+b) = 0  MPAout = MPAa‧MPA’b‧(1 – 0) + MPA’a‧MPAb‧(1 – 0 ) + MPAa‧MPAb‧ (1 – 0 ) = MPAa + MPA’a‧MPAb = 1 + a out b It means the error will be certainly propagated.

39. Assignment Decision Diagram • Assignment Decision Diagram was published in DAC 1993, which was intermediate format for the design. • Many related works such as RTL ATPG, RTL testability analysis adopt this method.

40. Assignment Decision Diagram • The ADD representation consists of four parts: • the assignment value • the assignment condition • the assignment decision • the assignment target

41. The Assignment Value • Consists of read nodes and operation nodes. • The computation is represented as a data-flow graph that contains operator nodes, corresponding to the type of operations that are performed.

42. TheAssignmentcondition • Consists of read nodes and operation nodes that are connected as a data-flow graph to represent the computation of a condition. • The end product of the condition-computation is a binary value that evaluates to true or false. This true/false value is used as a guarding condition for the assignment value.

43. The Assignment-Decision • The assignment-decision part consists of an Assignment Decision Node (ADN). The ADN selects a value from a set of values that are provided to it. • The selection is based on the conditions computed by the assignment condition part of the ADD. If one of the conditions to the ADN evaluates to true then the corresponding input value is selected. It

44. The Assignment-Target • The write node is associated with the selected value from the corresponding ADN.

45. Assignment Decision Diagram if(State_reg == ST2) begin if(b==2) X = Y+Z else X = Y-Z end

46. Observation Point Insertion • Choose the signals with low MPA • Check whether the error caused by the mutant propagates to these locations. • If we cannot detect the error on these location, we must stop the simulation for the mutated design. • the low MPA represents it is hard to propagate the error to this location. Therefore we choose the signals with low MPAfor each mutant.

47. The Flowchart for the algorithm Define the location for observation points according to the MPA analysis result Transfer the RTL design to the ADD Mutation Propagation Ability Analysis Extract the information from the observation points Mutation Analysis, and compute the mutant coverage

48. Outline • Introduction • Functional Verification • Verification Quality Measuring • Functional Qualification • Mutation Analysis (MA) • The Limitation of MA • Previous Work • Cost Reducing Techniques • Mutant Reduction Techniques • Proposed Approach • Mutant Propagation Ability Estimation • Assignment Decision Diagram • Observation Point Insertion • The Flowchart for the algorithm • Experimental Results • Future work

49. Future Work • Finish the MPA estimation rule for more operations. • Deal with the MPA in condition node. • Write the program!