1 / 45

An Approach for Selecting Tests with Provable Guarantees

An Approach for Selecting Tests with Provable Guarantees. Mahadevan Subramaniam and Bo Guo University of Nebraska at Omaha. Multi-version QA. Changes. Requirements. No bug claimed. No bug claimed. R1. R2. bugs. bugs. Development Team. Development Team. Testing Team. Testing

baruch
Download Presentation

An Approach for Selecting Tests with Provable Guarantees

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Approach for Selecting Tests with Provable Guarantees MahadevanSubramaniam and Bo Guo University of Nebraska at Omaha

  2. Multi-version QA Changes Requirements No bug claimed No bug claimed R1 R2 bugs bugs Development Team Development Team Testing Team Testing Team codes Modified codes Two or Three weeks Testing is very costly and time consuming [NIST] 1) in critical path for release 2) development deadline slips can reduce test time 3) development team needs time to fix bugs UNO

  3. Multi-version QA Scope • Predictive Changes: new tests to validate new features • Corrective Changes: Regression testing of existing features • Problems • Regression Test Minimization – remove redundant tests • Regression Test Prioritization – fault-finding tests first • Regression Test Selection (RTS) – select relevant tests UNO

  4. Regression Test Selection -- Motivation • Regression testing is most frequent [NIST] and has significant impact on product quality. • System behavior re-validated in practice using large test suites. • Re-running the entire test suite each time is impractical. • Regression test selection identifies tests from a test suite to validate a change in an evolution step. UNO

  5. Current State of Art – Code based UNO

  6. Current State of Art – Model-based UNO

  7. Current State of Art • Store all test traces on original version • Static analysis • Compare new and modified Control Flow Graphs • If a trace includes the differences, select test including the trace. • Use data flow graphs and program dependency graphs UNO

  8. Problems • Storing all traces is expensive • Control flow graphs comparing may involve untested parts of programs • No guarantee on the regression test suite • Relevant tests may be skipped (Incomplete) • Irrelevant tests may be selected UNO

  9. Goal • Develop an automatic approach for regression test selection to create a regression test suite with provable guarantees and without using any history information. • Main Challenges • Provably predict test execution behavior without actually executing the test ? • Keep regression test selection economical • Selection cost + regression test cost <= brute-force run cost. UNO

  10. Model-based Proposal Key ideas • Analyze test descriptions to provably predict their behavior. • Test descriptions based on EFSMs and GUI event diagrams have sufficient information in terms of input and output events. • Exploit overlap among descriptions to keep regression economical [See tech report] UNO 11/30/2012 10

  11. s0 EFSM – extended finite state machine 1 6 • List of states • The machine is in only one state at a time • Triggering condition for each transition s1 Initially a = 0; state is s0 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 2 5 3 s2 3 4 UNO

  12. Overview of the Approach • Analyze a given test description to determine if the test execution will exercise a given change and select the test. • Identify descriptions having sufficient information about their execution (fully-observable tests) • Analyze such descriptions to determine if the change will be executed. • Patch other descriptions to obtain information about their execution. • Changes add / delete / replace transitions. • Test description is a sequence of input and output messages with parameters with values over booleans, integers, arrays, queues, and records. UNO 12

  13. Overview of the Approach (Contd.) • From a given test description and a change • find sets of matching transitions that can process inputs in the description • descriptions having a matching change transition are candidates • determine feasibility of an execution path formed by matching transitions • automatically attempt to patch infeasible paths to obtain an execution path • Test selected if the change transition appears in the feasible execution path. UNO 13

  14. s0 A Simple Example 1 6 s1 Initially a = 0; state is s0 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7 2 5 3 s2 3 4 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). Not Selected Tests λ1: open(100)/ack(100), deposit(50)/ack(150), close/ack(150) λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) UNO

  15. Φ = [ { 1 } { 2 } { 6 } { 3, 7 } ] s0 A Simple Example – λ2 1 6 λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) s1 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). 7 2 5 3 3 s2 Find transitions matching each input of descriptionλ2 4 UNO

  16. s0 A Simple Example – λ2 (conti…) 1 6 λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) Φ= [ {1} {2} {3, 7} {6} ] s1 7 2 5 3 3 s2 First test input of λ2must be processed by transition 1 4 Transition 2 can immediately follow 1 to process next input of λ2 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). UNO

  17. s0 A Simple Example – λ2 (conti…) Selected 1 6 λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) Φ= [ {1} {2} {3, 7} {6} ] s1 7 2 5 3 3 s2 Transition 7 can immediately follow 2 since (160 > 0) ˄ (160 > 150) satisfiable. 4 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). UNO

  18. A Simple Example – λ3 s0 1 6 λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) s1 7 2 Φ = [ {1} {2} {3, 7} {3, 7} {6} ] 5 3 3 First two inputs processed by transition 1 followed by transition 2. s2 Transition 7 cannot immediately follow 2 since (100 > 0) ^ (100 > 150) is unsatisfiable. 4 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). UNO

  19. A Simple Example – λ3(conti..) s0 1 6 λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) s1 7 2 Φ = [ {1} {2} {3, 7} {3, 7} {6} ] 5 3 3 s2 But 3 can immediately follow 2 since (100 > 0) ˄ (100 <= 150) is satisfiable. 4 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). UNO

  20. A Simple Example – λ3(conti..) s0 1 6 λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) s1 7 2 ? Φ = [ {1} {2} {3, 7} {3, 7} {6} ] Can we patch λ3 to obtain a path with transition 7? 5 3 3 Neither 7 nor 3 can immediately follow transition 3. s2 4 7 can follow 3 after some transitions not using test inputs. 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). UNO

  21. A Simple Example – λ3(conti..) s0 Selected 1 6 λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) s1 7 2 ? Φ = [ {1} {2} {3, 7} {3, 7} {6} ] 5 4 cannot patch since (50 > 0) ^ (50 < 50) is unsatisfiable. 3 3 s2 5 can patch since (50 >= 50) is satisfiable and 7 can immediately follow 5. 4 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). UNO

  22. Salient Aspects • Regression test selection for test descriptions with rich data types. • Fully-observable tests: descriptions with sufficient information. • A simple structural invariant to identify fully-observable tests. • Automatically analyze failure of invariant to patch test descriptions. • Procedures to select tests guaranteed to exercise changes to EFSMs. • Experiments with 10 web services and protocols • Study costs of running the full test case and selective test cases. • Proposed approach reduces test running times in all examples. UNO

  23. Outline • Preliminaries • Fully-Observable Tests • Tests with Non-Observable Regions • Experiments • Conclusion UNO

  24. EFSMs • E = (Ii, Oi, Si, Vi, Ti) • Ii, Oi : input and output messages • Si: local states • Vi: Data and queue variables • Ti: Deterministic transition relation • Transitiont: mj, Pt, st qt, ml, At • mi ml: parameterized input and output messages • Pt: a predicate over variables from Vi • st, qt : states from Si • At: ordered sequence of assignments to variables from Vi • Explicit transition is transition having both input and output messages UNO

  25. Test Descriptions and Changes • Test description • λ = < g0, [i1/o1, i2/o2, …, in/on] > • g0 is concrete initial global state • Finite sequence of input/output elements ik/ok • Test run • rλ= g0t0…tmgm…g0 is an EFSM run produced by applying λ to the EFSM in state g0. • All the global states in rλ are concrete global states. • Changes to the EFSM • Performed at the transition level • δ = < sign, tn >, sign { +, - } (addition, deletion) • δ = < to, tn > (replacement) UNO

  26. Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion UNO 26

  27. Transitions Matching a Test Description Transition matches a test if an input-output element of test is an instance of input-output message of the transition and satisfies the input condition of the transition. Transition 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a) matcheswithdraw(160)/ack(150). Φ: sequenceof sets of transitions point-wise matching the sequence of input-output elements in a test description. Φ= [ {1} {2} {3, 7} {6} ] for test λ2 : sequence of transitions selected point-wise from the sets.  = [ 1 2 3 6 ] UNO

  28. Fully-Observable Tests  has a disjunct for each sequence of transitions ρ in Φ. kth conjunct states that execution path with k transitions can be extended using k+1st transition. Test is fully-observable if its invariant  is a satisfiable formula A disjunct with value true denotes execution path for a satisfiable  At most one disjunct can evaluate to truesince EFSMs are deterministic ( n-1  = ˅˄ Pos([t1,…, tk], g0)  Pre(tk+1) ρ ϵ Φ k=0 UNO

  29. Identifying Fully-Observable Tests λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) ρ1: conjunct 0: (a0 == 0) ˄ (100 > 0) ρ1: conjunct 1: (a0 == 0) ˄ (100 > 0) ˄ (a1 == a0+100) ˄ (50 > 0) ρ1: conjunct 2: (a0 == 0) ˄ (100 > 0) ˄ (a1 == a0+100) ˄ (50 > 0) ˄ (a2 == a1+50) ˄ (160 > 0) ˄ (160 <= a2) 3 1 2 ρ2: conjunct 2: (a0 == 0) ˄ (100 > 0) ˄ (a1 == a0+100) ˄ (50 > 0) ˄ (a2 == a1+50) ˄ (160 > 0) ˄ (160 > a2) 6 7 Initially a = 0; state is s0 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). Φ= [ {1} {2} {3, 7} {6} ] 3 1 2 7 UNO

  30. Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Preliminary Experiments Conclusion UNO 30

  31. Tests with Non-Observable Regions • Identify a non-observable region (tk, tk+1) for a conjunct that fails to extend to transition tk+1 • EFSM paths from output state of tk to input state of tk+1. • All intermediate transitions in all paths have no test inputs. • Eliminating regions without loops • Merge the intermediate transitions with tk to tmerge • Patch conjunct C = Pos([t1,…, tmerge], g0)  Pre(tk+1) and check if satisfiable. • Fails if patched conjuncts for all paths in region unsatisfiable. UNO 31

  32. Patching Failures – Example s0 1 6 λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) s1 7 2 1 5 3 3 s2 ignore 2 ? 4 3 3 7 5 ? 7 7 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). UNO 32

  33. Regions with Terminating Loops • Regions having loops cannot be eliminated by simple merging. • Each loop iteration requires additional test input. • Merge intermediate transitions in the path until the loop end-points and attempt to construct a path by executing loop transitions. • Process terminates since all loops are terminating over concrete global states. UNO 33

  34. Patching Involving Loop – Example s0 1 6 λ4: open(100)/ack(100), deposit(50)/ack(150), withdraw(50)/ack(100), withdraw(60)/ack(40), withdraw(60)/ack(50), close/ack(50) s1 7 Φ= [ {1} {2} {3, 7} {3, 7} {3, 7} {6} ] 2 1 5 3 3 s2 ignore 2 4 ? 3 7 Patch fails ? ? 5 1. open(v), v > 0, s0  s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a), 4. a > 0  a < 50, s2  s2, {a = a + 1}, 5. a >= 50, s2  s1, {}, 6. close, true, s1  s0, {}, ack(a). 7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a). 3 3 7 4 ? 7 7 7 5 UNO 34

  35. Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion UNO 35

  36. Experiments • Regression Cost Model [Leung and White 91] • C1: Cost of running the full test suite • C2: Cost of running the selected tests • C3: Cost of analyzing the regression test selection - > Atm: automatic teller machine; Thp: third party call; Bnk: back web services; Ven: a vending machine (Cmp, Tcp, Cnf) : completion, two-phase commit, and conference protocol

  37. Time Savings across Multiple Changes UNO

  38. Test Selected across Multiple Changes UNO

  39. SPG with Code-based approach s0 E P1: incr1 (x) { p1 while(++x <= 0) { } s1 return x; } s1 P1 t2: while() [x+1<=0] /{x=x+1} T s1 F X t3: jump() [x+1>0] /{x=x+1; return(x)} t1: incr1(p) /{x = p} UNO

  40. SPG with Code-based approach F E E T P1: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} } s1 return x; } P2 P1 P1 T T s1 s1 F F X X x = 0 : (E, P1) (P1, s1) x = 0 : (E, P1) (P1, s1) UNO

  41. SPG with Code-based approach t3: jump() [x+1>0] /{x=x+1; return(x)} s0 s0 t1: incr1(p) /{x = p} P1: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} } s1 return x; } s1 s1 t4: while(), [x+1<=0] /{x=x+1} t6: jump(), [x+1>0] /{x=x+1} t2: while() [x+1<=0] /{x=x+1} s2 t5: while() [x+1<=0] /{x=x+1} t3: jump() [x+1>0] /{x=x+1; return(x)} t1: incr1(p) /{x = p} Incr1(0)/null, jump()/return() : t1,t3 Incr1(0)/null, jump()/return() : t1,t3 UNO

  42. SPG with Code-based approach E E T P3: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} s2 return x; } s1 return x; } P2 F P1 P1 T s2 T s1 s1 F F X X x = -1 : (E, P1) (P1, P1) (P1, s1) (s1, X) x = -1 : (E, P1) (P1, P2) (P2, s2) (s2, X) UNO

  43. SPG with Code-based approach t3: jump() [x+1>0] /{x=x+1; return(x)} s0 s0 t1: incr1(p) /{x = p} P3: incr1 (x) { p1 while(++x <= 0) { p2 while(++x <= 0) {} s2 return x; } s1 return x; } s1 s1 t4: while(), [x+1<=0] /{x=x+1} t6: jump(), [x+1>0] /{x=x+1; return(x)} t2: while() [x+1<=0] /{x=x+1} s2 t5: while() [x+1<=0] /{x=x+1} t3: jump() [x+1>0] /{x=x+1; return(x)} t1: incr1(p) /{x = p} incr1(-1)/null, while()/null, jump()/return(0) : t1, t2, t3 incr1(-1)/null, while()/null, jump()/return(0) : t1, t4, t6 UNO

  44. Conclusions • Proposed an approach for regression test selection for high-level EFSM test descriptions supporting common data types. • Test descriptions are automatically analyzed to identify tests that exercise a given change. • Procedures to identify fully-observable tests, patching non-observable regions, and reducing the test suites are described. • Initial results of our experiments are promising. • Comparison with Code-based approach. UNO

  45. Pre- and Post-Images of Transitions Pre-image of t: a symbolic global state that triggers the transition t. 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a) Pre(3) = < s1, ( v > 0 ^ v <= a0 ^ IQ0.hd == withdraw(v) ) Post-image of t: a symbolic global state that is obtained after executing t. Pos(3) = < s2, (v > 0 ^ v <=a0 ^ IQ1 == deq(IQ0) ^ IQ0.hd == withdraw(v) ^ OQ1 == <OQ0, ack(v)> ^ a1 == a0 – v ) > Executable path: a transition sequence [t1,…, tn] from a given global state g if Pos([t1,…tn], g) is satisfiable. n-1 Pos([t1,..,tn], g) = pred ˄ ( ˄Pos(ti) ˄ Pre(ti+1)) ˄ Pos(tn) i=1 UNO

More Related