1 / 64

Realization of solver based techniques for Dynamic Software Verification

Realization of solver based techniques for Dynamic Software Verification. Andreas S Scherbakov Intel Corporation andreas.s.scherbakov@intel.com. What’s program testing here?. The problem: to test a program means Find at least one set of input values such that

andrew
Download Presentation

Realization of solver based techniques for Dynamic Software Verification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Realization of solver based techniques for Dynamic Software Verification Andreas S Scherbakov Intel Corporation andreas.s.scherbakov@intel.com

  2. What’s program testing here? • The problem: to test a program means • Find at least one set of input values such that • a crash/an Illegal operation occur or • some user defined property has violated (unexpected results/behaviour) or • Prove correctness of the program • or at least demonstrate that it’s correct with some high probability

  3. SW testing: basic approaches • Random testing -> You execute your program repeatedly with random input values.. + covers a lot of unpredictable cases ─ too much redundant iterations -> out of resources • “Traditional “ testing - Custom test suites -> You know you code and therefore you can create necessary examples to test it?.. + targets known critical points ─ misses most of unusual use cases ─ large effort, requires intimate knowledge of the code • Directed testing -> Try to get a significantly different run each attempt.. + explores execution alternatives rapidly + effective for mixed whitebox/blackbox code ─ usually needs some collateral code ─ takes large resources if poorly optimized

  4. SW testing: basic approaches - 2 • Static Analysis • Commercial tools: Coverity, Klocwork, … ─ Find dumb bugs, not application logic errors ─ Finds some “false positive” bugs, misses many real bugs + Good performance + Little expertise required • Model Checking • Academic and competitor tools: BLAST, CBMC, SLAM/SDV + Finds application logic errors ─ Finds some “false positive” bugs, but doesn’t miss any real ones ─ Significant user expertise required • Formal Verification • Academic tools: HOL, Isabelle, … + Ultimate guarantee: proves conformance with specification ─ Scaling constraint is human effort, not machine time ─ Ultimate user expertise required: multiple FV PhDs

  5. Directed Testing:as few runs as possible • executes the program with two test cases: i=0 and i=5 • 100% branch coverage

  6. DART: Directed Automated Random Testing • Main idea has been proposeded in Patrice Godefroid, Nils Klarlund, and Koushik Sen. DART: Directed Automated Random Testing. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. PLDI 2005: 213-223. • Dependent upon a Satisfiability Modulo Theories (SMT) solvers -> SMT solvers are applications able to solve equation sets. A theory here implies methods related to some set of allowed data types/operands

  7. What does it check? • Does not verify the correctness of the program UNLESS YOU HAVE Express the meaning of CORRECTNESS in form of ASSERTION CHECKERs • Can not infer what the ‘correct’ behavior of the program is • What does it check • allows users to add assumptionsto limit the search space and assertions (‘ensure‘) to define the expected behavior. • Assertions are treated as (‘bad’) branches– so test process will try to reach them, or formally verify it is impossible. • ‘built in’ checks for crashes, divide by 0, memory corruption • requires some familiarity with the software under test for effectiveness.

  8. Looking for a Snark in a Forest Looking for a Bug in a Program • A bug is a like a snark • A program is like a forest with many paths • Source code is like a map of the forest Just the place for a Snark! I have said it twice: That alone should encourage the crew.Just the place for a Snark! I have said it thrice: What I tell you three times is true. The Hunting of the Snark Lewis Carroll

  9. Proof Rather than Snark Hunting forest searching can be a very effective way to show the presence of snarks, but is hopelessly inadequate for showing their absence. The Humble Snark Hunter • How can we prove there no snarks in the forest? • Get a map of the forest • Find the area between trees • Assume a safe minimum diameter of a snark • If minimum snark diameter > minimum tree separation no snarks in forest • The gold standard, but: • You need a formal model of the forest • A mathematician • Substantial effort • As good as your model of forests and snarks (are snarks really spherical?)

  10. Snark Hunting Via Random Testing • REPEAT • Walk through the forest with a coin. • On encountering a fork, toss the coin: • heads, go left • tails, go right • UNTIL snark found or exhausted • Easy to do: You don’t even need a map! • But: • Very low probability of finding a snark

  11. Traditional Snark Hunting • Study the forest map and use your experience to choose the places where snarks are likely to hide. • For each likely hiding place, write a sequence of “turn left”, “turn right” instructions that will take you there. • REPEAT • Choose an unused instruction sequence • Walk through the forest following the instructions • UNTIL snark found or all instructions used • But… • Snarks notoriously good at hiding where you don’t expect

  12. Snark Hunting Via Static Coverage Analysis • Get a map of the forest • Have a computer calculate instruction sequences that go through all locations in the forest. • REPEAT • Choose an unused instruction sequence • Walk through the forest following the instructions • UNTIL snark found or enough of the forest covered • But… • Lot of computing power to calculate the paths • there will be a lot of paths

  13. Effective Snark Hunting Without A Map • Start with a blank Map He had bought a large map representing the sea,        Without the least vestige of land:And the crew were much pleased when they found it to be        A map they could all understand. • REPEAT • REPEAT • Walk through the forest with • a map (initially blank) • sequence of instructions (initially blank) • Add each fork that you haven’t seen before to your map. • When encountering a fork: • If there is an unused instruction, follow it • Otherwise, toss a coin as in random testing • UNTIL you exit the forest • If there is a fork on your map with a branch not taken • Write a sequence of instructions that lead down such a branch • UNTIL snark found, no untaken branches on map, you’re tired

  14. Comparison of alternatives Formal Verification Model checking DART Accuracy Traditional testing Static analysis Expertise/Effort

  15. How it Works voidf (intx, inty) { if (x > y) { x = x + y; y = x – y – 3; x = x – y; } x = x – 1; if (x > y) { abort (); } return; } = 0 x y = 9 • f(x,y) run: 1 • Arbitrary inputs: • x = 0 • y = 9 false x > y =-1 x1= x – 1; false x1 > y

  16. How it Works x y voidf (intx, inty) { if (x > y) { x = x + y; y = x – y – 3; x = x – y; } x = x – 1; if (x > y) { abort (); } return; } • f(x,y) run: 2 • choose x, y so • (x > y) = false • x1 = x – 1 • (x1 > y) = true • no such x, y! x > y x1= x – 1;  x1 > y

  17. How it Works voidf (intx, inty) { if (x > y) { x = x + y; y = x – y – 3; x = x – y; } x = x – 1; if (x > y) { abort (); } return; } =9 x y =0 • f(x,y) run:2 • choose x, y so • (x > y) = true • Inputs • x = 9 • y = 0 true x > y =9 x1= x + y; y1 =x1 – y; x2 =x1 – y1 – 3; x3 = x2 – 1; x1= x – 1; =9 =-3 =-4  x1 > y false x3 > y1

  18. How it Works =1 x y voidf (intx, inty) { if (x > y) { x = x + y; y = x – y – 3; x = x – y; } x = x – 1; if (x > y) { abort (); } return; } =0 • f(x,y) run: 3 • choose x, y so • (x > y) = true • x1 = x + y • y1 =x1 – y • x2 =x1 – y1 + 3 • x3 = x2 – 1 • (x3 > y1) = true • Inputs: • x = 1 • y = 0 true x > y =1 x1= x + y; y1 =x1 – y; x2 =x1 – y1 – 3; x3 = x2 – 1; =1 =-3 =-4 true x3 > y1 abort

  19. A Simple Test Harness The Program • intmain () { • constintx = • choose_int ("x"); • constinty = • choose_int ("y"); • snarky (x, y); • return 0; • } • instrumentation library routine voidsnarky (intx, inty) { if (x > y) { x = x + y; y = x – y – 3; x = x – y; } x = x – 1; if (x > y) { abort (); } }

  20. Quick Example int main () { constsize_tsource_length = choose_size_atmost (…); constchar *source = choose_valid_string (…); constsize_ttarget_size = choose_size_atleast (…); constchar *target = choose_valid_char_array (…); string_copy (source, target); ensure (string_equal (source, target)); return 0; } void string_copy (constchar *s, char *t) { inti; for (i=0; s[i] != '\0'; ++i) { t[i] = s[i]; } } int string_equal (constchar *s, constchar *t) { inti = 0; while (s[i] != '\0' && s[i] == t[i]) { ++i; }

  21. Quick example: Bug found Bug found with the parameters: target_size = 1 target[0] = 1 source_length = 0 (Killed by signal)

  22. Overall Design • Harness Library • Supply specified values for inputs, or arbitrary values • Check required/ensured constraints • Instrumentation • Modify a C program to produce an execution trace with the required execution • Observed Execution • Observe path taken by a run and calculate predicate describing a new path • Constraint Solver • Solver used to discover for a specified path condition • If the path is feasible • Inputs that would cause it to be executed

  23. Testing Time • Don’t expect to test all paths for realistically sized data • You can, however, run many useful tests quickly

  24. Harness code Code under test Stub code You Provide The Controllability • For each “unit” you write • A harness to call unit’s functions • Stubs for functions the unit calls • Provides functions to generate values • For harnesses to call with • For stubs to return with • Declarative specification of constraints on the values • This provides • A model of the unit’s environment • Controllability over the unit

  25. Front End: Instrumentation

  26. Why do we track symbolic data? We want to be able to choose another branch next run.. if (x==y+3) { /* branch A */ } else { /* branch B */ } To choose given branch, we need to solve: ( x==y+3 ) == false/true To pass it to solver, we need to have x==y+3 expression in a symbolic form at if In order to know it at this point, we should track assignments of constituent components..

  27. Tracing symbolic data • Solution: adding special tracing statements to source statements x = y*z; tmp=trace_multiplication(VAR_Y,VAR_Z); x = y*z; trace_assign(VAR_X,tmp); x = y[i]; tmp=trace_array_element(VAR_Y,VAR_I); x = y[i]; trace_assign(VAR_X,tmp);

  28. CIL • “CIL (CIntermediate Language) is a high-level representation along with a set of tools that permit easy analysis and source-to-source transformation of C programs.” http://www.cs.berkeley.edu/~necula/cil/ • CIL enables user application to explore and re-factor various types of C source constructs (functions, blocks, statements, instructions, expressions, variables etc) in a convenient way while keeping the remaining code structure.

  29. Tool Framework Frontend User Input Backend User written harness Scoreboard track coverage CIL Instrument -ation Run Instrumented Program Software under test Input Generator SMT Solver Problem: CIL Based Frontend does not support C++ Solution: Replace the CIL based frontend with LLVM to support C++

  30. How CIL simplifies handling the code.. • Automatically rewrites C expressions with side effects: a = b+= --c---> c = c-1; b = b+c; a = b; • Uniformly represents memory references: (base+offset) • Converts do,for,while loops to while (1) {if (cond1) break; /* if needed */if (cond2) continue; /* if needed */ body;} • Traces control flow

  31. What is LLVM? • LLVM – Low Level Virtual Machine • Modular and reusable collection of libraries • Developed at UIUC and at Apple® • LLVM Intermediate Representation (IR) is well designed and documented. • Has a production quality C++ frontend that is compatible with GCC • Open-source with industry friendly license. • More info at www.llvm.org

  32. LLVM frontend LLVM Based Frontend User written harness Clang C/C++ Parser Compiler Pass Rest of Compile LLVM IR Software under test Instrumented Program Backend LLVM provides modular libraries and tool infrastructure to develop compiler passes

  33. Using C++ overloads • Idea: redefine operators such a way that they output trace data: my_intoperator + (my_int x, my_int y) { symbolic s = trace_addition(x.symbol(),y.symbol()); int c = x.val() + y.vall(); returnmy_int (s,c); } • Instrumentation is still needed (control tracing, types..)

  34. Reducing branches • This 2-branch control: if (x && y) action1; else action2; really produces 3 branches in C/C++: if (x){ if (y) action1; else action2; } else action 1; • x && y is not really a logical and. • We cannot simply supply (x && y) to a SMT solver..

  35. Reducing branches: solution • But.. Sometimes it IS logical and • Namely, if y may be safely evaluated at x==false or y cannot be safely evaluated at any x value which means • y has no side effects and • y crash conditions don’t depend on x • If we can prove this statically, use the form: if (logical_and(x,y)) action1; else action2; • Else use 3-branch form 

  36. Solver Theories • Different solver theory • Linear Integer Arithmetic: (a*x + b*y + ….) {><=} C • Linear Real Arithmetic • BitVector Arithmetic • Most conditions in C source code fits one of them. But some mixed/complex don’t • alas, sometimes using random alternation • luckily, theories are being developed actively • Need to recognize theory patterns for better performance -> Sometimes supported scope is wider then declared theory scope

  37. Path exploration strategy • Usually we explore all paths in Depth First Search mode: • alternate deeper ones first • when complete, return one level and try again • But execution path count may occur to be extremely high to explore all of them

  38. Path exploration strategy -2 • If we have no resources to explore all path, DFS is not the best strategy: some nodes never be visited while some others are carefully explored • low coverage coverage • most of dumb bugs may be missed • Good strategy principle: first visit new nodes, next explore new paths • Details are subject to research explored unexplored

  39. An optimization: Get function properties • Idea: Taking advantage of code hierarchy: using I/O properies for function/procedure call -> try to go with the assumptions only rather than deepening into subroutine body • Example: y = string_copy(x) require valid_pointer(x) property valid_pointer(y) /* assuming we have yet memory */ property length(x) == length(y) property i < length(x)  y[i] == x [i] • For black box (external library) code, assumptions should be supplied as collaterals • For available source code, they can also be extracted automatically -> but it’s a question what to extract If (length(s) > 2) { p = string_copy(s); if (length(s) >1) { } else { do_something(); }} If (length(s) > 2) { p = ???; assumelength(p) == length(s); if (length(p) >1) { } else { /* lenghts(p) <=1 && length(p) == length(s) && lengths(s) > 2) ---- Infeasible */ }}

  40. An optimization: Separate independent alternations if (z == 2) { x=b; do_something1(); } if (y == x) { do_something2(); } • Dependent choices • We should try 2*2 combinations: • z=2, y=b • z=2, y≠b • z≠2, y=x • z≠2, y≠x • (all variables are sampled at the beginning of code piece presented)

  41. Separate independent alternations -2 if (z == 2) { q =b; } if (y == x) { p = c; } • Independent choices • We can try only 2 combinations, for example: • z=2, y=x • z≠2, y≠x • (provided that do_something1() and do_something2() effects don’t interdepend)

  42. Separate independent alternations -3 if (z == 2) { q =b; } if (y == x) { p = c; } if (q == p) … Dependent choices again!

  43. An optimization: re-using unsatisfied conditions if(a && b && c) { … } if (a && b && c) { … } if (a && b && c&& d &&e) { … } Let we’ve proved that we cannot get here • Then, we can be sure that we cannot get there too • No need to call a solver again

  44. Handling Black Boxed Code

  45. Contents • Motivation • Losing control with black boxes • Return Value Representation • Randomization • Characterizing • Learning • Stubbing/Wrapping • Example: The encryption problem • Selective/Dynamic Black-Boxing • Embedded White-Boxes • Afterwords

  46. Motivation • Testing a portion of a code within a large system. E.g: • Code over infrastructure/library functions • Firmware over hardware API/Virtual Platform • Binary infrastructure • Hiding Code solver can’t cope with • Non Linear arithmetic (a*b = C) • Assembly • Handling Deep paths/Recursion

  47. Losing Control with Black Boxes • Black-boxes impair our controllability when program paths are influenced by black-box outputs. • We have no information to pick “a” such that it drives (b > 10) in both directions. int a = choose_int(“a”); int b = bb(a); if (b > 10) { … } else { … }

  48. Return Value Representation • The flow treats the return value of an uninstrumented function as concrete only (not symbolic). • But it can be explicitly assigned a fresh symbolic variable with fresh_* • The reverse could be done as well with concrete_* (later). int a = blackboxed_func(…); // a is concrete fresh_integer(a, “a”); // a is symbolic

  49. Example: The Encryption Problem ulong x = choose_uint ("x%d", count); ulong y = choose_uint ("y%d", count); if (y == encrypt(x)) <…>; else <…>; • We pathologically can’t guess x and y beforehand such that y == encrypt(x). • coping with it by: • Running once with y=x=0, the condition fails. • “see”: (y == <concrete encrypt(0)>) • choose x=0, y = encrypt(0) for the 2nd run.

  50. Randomization • We can increase our chances of gaining coverage by adding randomization int a = choose_random_int(“a”); int b = bb(a); if (b > 10) { … } else { … }

More Related