1 / 48

Hybrid Concolic Testing

Hybrid Concolic Testing. Rupak Majumdar Koushik Sen UC Los Angeles UC Berkeley. Automated Test Generation. Studied since 70’s King 76, Myers 79 30 years have passed, and yet no effective solution What Happened???. Automated Test Generation. Studied since 70’s

yvon
Download Presentation

Hybrid Concolic Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hybrid Concolic Testing Rupak Majumdar Koushik Sen UC Los Angeles UC Berkeley

  2. Automated Test Generation • Studied since 70’s • King 76, Myers 79 • 30 years have passed, and yet no effective solution • What Happened???

  3. Automated Test Generation • Studied since 70’s • King 76, Myers 79 • 30 years have passed, and yet no effective solution • What Happened??? • Program-analysis techniques were expensive • Automated theorem proving and constraint solving techniques were not efficient

  4. Automated Test Generation • Studied since 70’s • King 76, Myers 79 • 30 years have passed, and yet no effective solution • What Happened??? • Program-analysis techniques were expensive • Automated theorem proving and constraint solving techniques were not efficient • In the recent years we have seen remarkable progress in static program-analysis and constraint solving • SLAM, BLAST, ESP, Bandera, Saturn, MAGIC

  5. Automated Test Generation • Studied since 70’s • King 76, Myers 79 • 30 years have passed, and yet no effective solution • What Happened??? • Program-analysis techniques were expensive • Automated theorem proving and constraint solving techniques were not efficient • In the recent years we have seen remarkable progress in static program-analysis and constraint solving • SLAM, BLAST, ESP, Bandera, Saturn, MAGIC Question: Can we combine program analysis with classical testing techniques to Scale Automated Test Generation?

  6. Our Approach • Concolic Testing: • Combines Dynamic and Static Program Analysis • Exhaustive • Fails to scale • Random Testing: • Fast • Non-exhaustive • Redundant Executions and poor coverage + = Hybrid Concolic Testing

  7. Goals of Test Generation (Simplified) • Generate test inputs • Execute program on generated test inputs • Catch assertion violations • Problem: how to ensure that all reachable statements are executed • Solution: • Explore all feasible execution paths

  8. T F T F T F F T T T F T Execution of Programs Non-Conditional Statements • All Possible Execution Paths • Binary tree • Computation tree • Internal node! conditional statement execution • Edge! execution of a sequence of non-conditional statements • Each path in the tree represents an equivalence class of inputs Conditional Statements

  9. Random testing Random Testing [Bird and Munoz 83] Fuzz testing Windows NT [Forrester and Miller 00] QuickCheck [Claessen & Hughes 01] JCrasher [Csallner and Smaragdakis 04] RUTE-J [Andrews et al. 06] Randoop [Pacheco et al. 07] Very low probability of reaching an error Problematic for complex data structures Fuzz (Random) Testing

  10. Random testing Random Testing [Bird and Munoz 83] Fuzz testing Windows NT [Forrester and Miller 00] QuickCheck [Claessen & Hughes 01] JCrasher [Csallner and Smaragdakis 04] RUTE-J [Andrews et al. 06] Randoop [Pacheco et al. 07] Very low probability of reaching an error Problematic for complex data structures Fuzz (Random) Testing Example ( ) { s = readString(); if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && s[4]==‘2’ && s[5]==‘0’ && s[6]==‘0’ && s[7]==‘7’) { printf(“Am I here?”); } } Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’} Probability of reaching printf = 7-8» 10-7

  11. Random testing Random Testing [Bird and Munoz 83] Fuzz testing Windows NT [Forrester and Miller 00] QuickCheck [Claessen & Hughes 01] JCrasher [Csallner and Smaragdakis 04] RUTE-J [Andrews et al. 06] Randoop [Pacheco et al. 07] Very low probability of reaching an error Problematic for complex data structures Fuzz (Random) Testing Example ( ) { s = readString(); if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && s[4]==‘2’ && s[5]==‘0’ && s[6]==‘0’ && s[7]==‘7’) { printf(“Am I here?”); } } Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’} Probability of reaching printf = 7-8» 10-7 Fast and Inexpensive

  12. Concolic Testing • Combine concrete testing (concrete execution) and symbolic testing (symbolic execution) [PLDI’05, FSE’05, FASE’06, CAV’06, HVC’06] Concrete + Symbolic = Concolic

  13. Example int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } }

  14. Example int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } 2*y == x Y N x > y+10 N Y ERROR

  15. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 22, y = 7 x = x0, y = y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } }

  16. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 22, y = 7, z = 14 x = x0, y = y0, z = 2*y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } }

  17. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 22, y = 7, z = 14 x = x0, y = y0, z = 2*y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } 2*y0 != x0

  18. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 22, y = 7, z = 14 x = x0, y = y0, z = 2*y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } Solve: 2*y0 == x0 Solution: x0 = 2, y0 = 1 2*y0 != x0

  19. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1 x = x0, y = y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } }

  20. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x0, y = y0, z = 2*y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } }

  21. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x0, y = y0, z = 2*y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } 2*y0 == x0

  22. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x0, y = y0, z = 2*y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } 2*y0 == x0 x0 · y0+10

  23. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 2, y = 1, z = 2 x = x0, y = y0, z = 2*y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } Solve: (2*y0 == x0) Æ (x0 > y0 + 10) Solution: x0 = 30, y0 = 15 2*y0 == x0 x0 · y0+10

  24. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 30, y = 15 x = x0, y = y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } }

  25. Concrete Execution Symbolic Execution concrete state symbolic state path condition x = 30, y = 15 x = x0, y = y0 Concolic Testing Approach int double (int v) { return 2*v; } void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR; } } } Program Error 2*y0 == x0 x0 > y0+10

  26. Traverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors T F T F T F F T T T F T Explicit Path (not State) Model Checking

  27. Traverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors T F T F T F F T T T F T Explicit Path (not State) Model Checking

  28. Traverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors T F T F T F F T T T F T Explicit Path (not State) Model Checking

  29. Traverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors T F T F T F F T T T F T Explicit Path (not State) Model Checking

  30. Traverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors T F T F T F F T T T F T Explicit Path (not State) Model Checking

  31. Traverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors T F T F T F F T T T F T Explicit Path (not State) Model Checking

  32. Entire Computation Tree Limitations • Path Space of a Large Program is Huge • Path Explosion Problem

  33. Explored by Concolic Testing Entire Computation Tree Limitations • Path Space of a Large Program is Huge • Path Explosion Problem

  34. Limitations: A Comparative View Concolic: Broad, shallow Random: Narrow, deep

  35. Limitations: Example Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } } } • Want to hit COVER_ME • input() denotes external input • Can be hit on an input sequence • s = “ICSE” • c : ‘:’ ‘\n’ • Similar code in • Text editors (vi) • Parsers (lexer) • Event-driven programs (GUI)

  36. Limitations: Example Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } } } • Pure random testing can get to • state = 2 • But difficult to get ‘ICSE’ as a • Sequence • Probability 1/(28)6» 3X10-15 • Conversely, concolic testing • can generate ‘ICSE’ but explores • many paths to get to state = 2

  37. Hybrid Concolic Testing • Interleave Random Testing and Concolic Testing to increase coverage Motivated by similar idea used in VLSI design validation: Ganai et al. 1999, Ho et al. 2000

  38. Hybrid Concolic Testing • Interleave Random Testing and Concolic Testing to increase coverage • while (not required coverage) { • while (not saturation) • perform random testing; • Checkpoint; • while (not increase in coverage) • perform concolic testing; • Restore; • }

  39. Hybrid Concolic Testing • Interleave Random Testing and Concolic Testing to increase coverage • while (not required coverage) { • while (not saturation) • perform random testing; • Checkpoint; • while (not increase in coverage) • perform concolic testing; • Restore; • } Deep, broad search Hybrid Search

  40. Hybrid Concolic Testing Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } } } • Random Phase • ‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … • Saturates after many (~10000) iterations • In less than 1 second • COVER_ME is not reached

  41. Hybrid Concolic Testing Example ( ) { 1: state = 0; 2: while(1) { 3: s = input(); 4: c = input(); 5: if(c==‘:’ && state==0) state=1; 6: else if(c==‘\n’ && state==1) state=2; 7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) { COVER_ME:; } } } • Random Phase • ‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … • Saturates after many (~10000) iterations • In less than 1 second • COVER_ME is not reached • Concolic Phase • s[0]=‘I’, s[1]=‘C’, s[2]=‘S’, s[3]=‘E’ • Reaches COVER_ME

  42. Hybrid Concolic Testing • 4x more coverage than random testing • 2x more coverage than concolic testing

  43. Results

  44. Results: Red Black Tree test_driver() RedBlackTree rb = new RedBlackTree(); while(1) { choice = input(); data = input(); switch(choice) { case 1: rb.add(data); break; case 2: rb.remove(data); break; case 3: rb.find(data); break; default: rb.add_if_not_member(data); break; } } }

  45. Results

  46. Concolic Testing Random Testing Summary

  47. Concolic Testing Random Testing Summary Hybrid Concolic Testing

  48. Thank You!

More Related