160 likes | 337 Views
Generating High-Quality Tests for Boolean Circuits by Treating Tests as Proof Encoding. Eugene Goldberg, Pete Manolios Northeastern University, USA. TAP-2010, July 1-2, Malaga, Spain. Summary. Introduction Test generation algorithm based on TTPE Experimental results and conclusion.
E N D
Generating High-Quality Tests for Boolean Circuits by Treating Tests as Proof Encoding Eugene Goldberg, Pete Manolios Northeastern University, USA TAP-2010, July 1-2, Malaga, Spain
Summary • Introduction • Test generation algorithm based on TTPE • Experimental results and conclusion
Motivation • Testing is easy (scalable) but incomplete • Formal verification is complete but hard • How do we get the best of both worlds (e.g. by generating a small test set that is complete or close to such)? • We study testing by the example of combinational circuits • We describe circuits by propositional logic
Main Idea TTPE: Treating Tests As Proof Encoding Q: Why does testing work at all? A: Short proof small encoding test set Q: What is the best coverage metric? A: A formal proof is an ideal coverage metric (contains a complete set of corner cases) Q: When does one stop generating tests? A: When the test set encodes a proof
Description of the Setup N is a combinational circuit, Itis correct if N 0, It has a bug if N(x) = 1. z N Y Circuit N CNF formula FN(X,Y,z), N 0 (FNz) 0 …. X Point p is (x,y,z) i.e. a complete assignment to X Y {z} Test t extracted from p is the assignment to X, i.e. t = x
A Naive Testing Procedure Si = {p1,…,pi} z p1, t1, N(t1)=0, Enc(S1) = no N Y p2, t2, N(t2)=0, Enc(S2) = no …….. pi, ti, N(ti)=0, Enc(Si) = no …. X pi+1, ti+1, N(ti+1)=1, Enc(Si+1) = no pi+1, ti+1, N(ti+1)=0, Enc(Si+1) = yes
Summary • Introduction • Test generation algorithm based on TTPE • Experimental results and conclusion
High-Level Description • We use the resolution proof system • Checking if a set of points encodes a resolution proof is hard • Instead, we generate points (called boundary) that encode mandatory fragments of a proof • So, instead of trying to encode an entire proof we target essential parts of it • We stop when a counterexample is found or a resource is exceeded
Encoding a Resolution Proof C = x1 x2 ~x5, C =~x1x7 C = x2 ~x5 x7 p, p legalize the resolution of C and C on v iff a) p and p are different only in v b) C (p ) = 0, C (p ) = 0 c) C(p ) = 0, C(p ) = 0, where C is the resolvent S={p1,..,pk} encodes a proof if the resolutions legalized by points of S are sufficient to derive an empty clause. Tests T encode a proof if they are extracted from S encoding a proof
Small Complete Test Sets A proof of (FNz) 0 of k resolutions is encoded by 2k points (at most) A complete set of 2k tests for checking N 0. N with 1000inputs: 106 tests instead of 21000
Boundary Points • Given a CNF formula F, an unsatisfying assignment to Vars(F) is a v-boundary point p iff • If C F and C(p)=0, then v Vars(C) • If p is a v-boundary point of F and F is unsatisfiable, any proof has to have a resolution such that • it is a resolution on v, • the resolvent is falsified by p Adding the resolvent of this resolution to F eliminates p as a v-boundary point. If F is satisfiable, there is a boundary point of F that cannot be eliminated.
Extracting Tests from Boundary Points Circuit N , CNF formula F=FN z, Is N(t)=1? Generate a v-boundary point pi of F (finding pirequires a SAT-check) Extract test ti from pi N(ti)=1? If so, stop. Add to F a resolvent on v (mandated by pi) Go to step 1 (to generatepi+1)
Summary • Introduction • Test generation algorithm based on TTPE • Experimental results and conclusion
Faults in Arithmetic Components G is a 24-input AND gate
Comparing SAT, Random Tests and Tests from Boundary Points • 17 faults. Random tests failed (106 per fault) • Reused tests generated for previous faults • Precosat is a winner of SAT-2009 competition • Used Precosat for finding boundary points too
Some Concluding Remarks • TTPE is not a trick. There is a deep relation between proofs and tests • Many ways to use TTPE (e.g. encoding mandatory parts of a proof) • TTPE for more expressive logics (describing sequential circuits and software)?