Automatic Test Generation

1. Automatic Test Generation • Generate a test sequence for a given behavioral description • Some coverage metric is typically assumed • Advantages of Automation: • Less designer time/cost • Less dependence on designer experience/ability Disadvantages of Automation: • Must rely on behavioral description (not spec) • Must rely on coverage metrics

2. Fault Directed Test Generation • A subset of potential faults are chosen for test generation • Tests are generated specifically for the chosen set • Continue until all potential faults are detected Key Points: The faults are defined by the coverage metric - “a statement may be faulty” for statement coverage 2. The input space must be searched 3. These algorithms are typically “complete” - If a fault can be detected then it will be detected

3. SAT-Based Test Generation • SATISFIABILITY (SAT) is the problem of solving a set of boolean constraints • Ex. (A AND B) OR (C AND D) • An ATG problem can be solved as a SAT problem if the coverage constraints are boolean • if (x AND y) • then out=1; (x AND y) • else out=2;NOT (x AND y) • Both statement constraints are boolean • Boolean constraints are common in logic

4. SAT-Based Problems • Some constraints are arithmetic, not naturally boolean • if (x > y) • then out=1; (x > y) • else out=2;NOT (x > y) • Arithmetic constraints can be converted to boolean but it is expensive • The > operator can be replaced by a gate-level implementation of a comparator • A better solution would solve both boolean and arithmetic constraints

5. Integer Linear Programming (ILP) • ILP is basically solving simultaneous equations (ala high school algebra) • X + Y = 10 • X - Y = 16 • 2X = 26 • X = 13 • Equations must be linear, variables must have integral values • A better solution would solve both boolean and arithmetic constraints

6. ILP + SAT • Use ILP for arithmetic constraints and SAT for boolean constraints • ((A > B) OR (B > C)) AND ((A > 3) OR (A < B)) • First abstract the arithmetic and solve the boolean • (w OR x) AND (y OR z) • SAT solution - w is TRUE and y is TRUE • Next, solve each arithmetic clause which is TRUE • (A > B), (A > 3) • 3 > B, A > 3

7. ILP + SAT Complexities • When solving the SAT, many solutions may exist • Which clauses will be selected to be TRUE? • ((A > B) OR (B > C)) AND ((A > 3) OR (A < B)) • (w OR x) AND (y OR z) • SAT solution - w is TRUE and z is TRUE • Next, solve each arithmetic clause which is TRUE • (A > B), (A < B), No Solution • A clause selection heuristic is needed • - Shortest clause, clause with least common vars, etc. • A backtracking mechanism is needed • - When conflict occurs, what decisions are reversed?

8. Constraint Logic Programming (CLP) • Need a constraint solving approach which can handle arithmetic and boolean constraints • CLP was developed in the AI field but it can be applied to a wode range of problems • CLP constaints are declarative: • - They express a relationship between between variables which must hold without specifying a computational procedure to enforce the relationship

9. Declarative vs. Procedural Example: The greatest common divisor between two numbers X and Y Procedural Solution: Write a program which solves computes the GCD - Describe how to solve the problem, step by step. 1. Divide the larger by the smaller and note the remainder. 2. Divide the remainder (92) into the previous divisor. 3. Repeat steps 1 and 2 until the remainder is 1 or zero. 4. When the remainder is zero the last divisor is the GCD. Declarative Solution: Describe the relationship between X, Y, and other variables - Do not describe how to compute the GCD Z. X / Z is an integer, Y / Z is an integer X / Q is an integer, Y / Q is an integer For all possible Q, Z >=Q

10. CLP Problem Formulation • A CLP problem consists of: • * a set of variables X={x1,...,xn}, • * for each variable xi, a finite set Di of possible values (its domain), • * and a set of constraints restricting the values that the variables can simultaneously take. • A solution to a CSP is an assignment of a value from its domain to every variable, in such a way that every constraint is satisfied

11. X x != y x != z Y Z y != z Internal Constraint Representation • An n-ary constraint is one that involves n variables • x > 3 is unary, x > y is binary, x + y > z is 3-ary • Unary constraints are solved trivially by restricting a variable’s domain. • Ex. x > 0, restrict the value of x to positive numbers • Binary constraints can be represented as a constraint network x != y y != z z != x