1 / 50

DART: Automated Random Testing

Learn about DART, a novel approach to overcome limitations in unit testing by automating the testing process and generating random test inputs.

lworsham
Download Presentation

DART: Automated Random Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DARTDirected Automated Random TestingPatrice Godefroid, Nils Klarlund,and Koushik Sen Syed Nabeel

  2. Motivation • Software Testing emerges as one of the most important aspects in Software Engineering • Unit Testing • Limitations in Unit Testing • DART a novel approach to counter limitations in Unit Testing

  3. Unit Testing Testing small portions of programs, such as methods, groups of methods, or classes. • a sequence of calls to the units (for e.g. methods) • BEFORE the calls some code to generate parameters for the methods • AFTER the calls some code checking whether the method has performed correctly.

  4. Goals of Unit Testing • Detect Errors in the components logic • Check ALL corner cases • Provide 100% code coverage

  5. Question What are some limitations of Unit Testing?

  6. Limitations of Unit Testing • Requires test driver and harness code which is representative of ALL external environment • Expensive and hard to perform manually (rarely done properly)

  7. DART To The Rescue Automates unit testing removing need for writing test drivers • Automated extraction of the program interface by static code parsing • Automatic generation of test driver for this interface that performs random testing • Dynamic Analysis of the program behavior to generate specific test inputs for the direction of program execution through alternate program paths

  8. Execution Model For DART • Concrete Execution: based on a memory model M that maps addresses to values • Symbolic Execution: based on symbolic memory S that maps addresses to expressions

  9. Directed Testing by Constraint Building • Progressively constraints are built as branches are encountered • Linear constraints are solved to explore various execution paths • In case of getting stuck or non linear constraints symbolic execution falls back to concrete execution

  10. Consider An Example …. int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  11. Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2); }

  12. Random Testing int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10){ printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } main(){ int tmp1 = randomInt(); int tmp2 = randomInt(); test_me(tmp1,tmp2); } Evident Problem: Probability of Reaching Abort is very low

  13. DART Approach main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } • Full example taken from Presentation by Koushik • at http://osl.cs.uiuc.edu/~ksen/slides/dart-fm.ppt

  14. concrete state symbolic state constraints t1=36 t1=m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  15. concrete state symbolic state constraints t1=36, t2=-7 t1=m, t2=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  16. concrete state symbolic state constraints t1=36, t2=-7 t1=m, t2=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  17. concrete state symbolic state constraints x=36, y=-7 x=m, y=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  18. concrete state symbolic state constraints x=36, y=-7, z=72 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  19. concrete state symbolic state constraints x=36, y=-7, z=72 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m != n

  20. concrete state symbolic state constraints x=36, y=-7, z=72 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m != n

  21. concrete state symbolic state constraints x=36, y=-7, z=72 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } solve: 2m = n m=1, n=2 solve: 2m = n m=1, n=2 2m != n

  22. Question Which portion of code identifies a new constraint ?

  23. The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  24. The Example Again main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  25. concrete state symbolic state constraints t1=1 t1=m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  26. concrete state symbolic state constraints t1=1, t2=2 t1=m, t2=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  27. concrete state symbolic state constraints t1=1, t2=2 t1=m, t2=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  28. concrete state symbolic state constraints x=1, y=2 x=m, y=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  29. concrete state symbolic state constraints x=1, y=2 x=m, y=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  30. concrete state symbolic state constraints x=1, y=2 x=m, y=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  31. concrete state symbolic state constraints x=1, y=2 x=m, y=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m != n+10

  32. concrete state symbolic state constraints x=1, y=2, z=2 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m != n+10

  33. concrete state symbolic state constraints x=1, y=2, z=2 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m != n+10

  34. concrete state symbolic state constraints x=36, y=-7, z=72 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } solve: 2m = n m=1, n=2 solve: 2m = n and m=n+10 m= -10, n= -20 2m = n m != n+10

  35. concrete state symbolic state constraints t1=-10 t1=m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  36. concrete state symbolic state constraints t1=-10, t2=-20 t1=m, t2=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  37. concrete state symbolic state constraints t1=-10, t2=-20 t1=m, t2=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  38. concrete state symbolic state constraints x=-10, y=-20 x=m, y=n DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  39. concrete state symbolic state constraints x=-10, y=-20, z=-20 x=m, y=n, z=2m DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } }

  40. concrete state symbolic state constraints x=m, y=n, z=2m x=-10, y=-20, z=-20 DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n

  41. concrete state symbolic state constraints x=m, y=n, z=2m x=-10, y=-20, z=-20 DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } 2m = n m = n+10

  42. concrete state symbolic state constraints x=m, y=n, z=2m x=-10, y=-20, z=-20 DART Approach Concrete Execution Symbolic Execution main(){ int t1 = randomInt(); int t2 = randomInt(); test_me(t1,t2); } int double(int x) { return 2 * x; } void test_me(int x, int y) { int z = double(x); if(z==y){ if(x != y+10) { printf(“I am fine here”); } else { printf(“I should not reach here”); abort(); } } Program Error 2m = n m = n+10

  43. Properties of DART • Sound w.r.t errors found (no false positives) • Complete (in a limited sense) If DART terminates without reporting a bug, no bug exists

  44. Advantages: • Dynamic Data Analysis • Sound Errors

  45. Dynamic Data Analysis struct foo { int i; char c; } bar (struct foo *a) { if (a->c == 0) { *((char *)a + sizeof(int)) = 1; if (a->c != 0) abort(); } }

  46. Dynamic Data Analysis • Static Analyzers would not be able to detect the change in the value of a->c and declare the program to be safe • DART locates the error by means of satisfying the constraint a->c==0

  47. Sound Errors foobar(int x, int y){ if (x*x*x > 0){ if (x>0 && y==10) abort(); } else { if (x>0 && y==20) abort(); } }

  48. Sound Errors • Static analysis using predicate abstraction state that both aborts may be reachable • Test-generation using symbolic execution get stuck at first conditional • DART detects first abort with high probability

  49. DART For C • Interface Extraction done by light weight static parsing of source code • Test Driver Generation involves initialization of external variables with random values, stubs for external functions • Directed Search comprises of implementation of constraint satisfaction, parsing and analysis of C code

  50. Discussion

More Related