1 / 28

03.03.2014, Copenhagen

Multi- Objective Test Suite Optimization for Incremental Product Family Testing ( joint work with Hauke Baller, Sascha Lity and Ina Schaefer). 03.03.2014, Copenhagen. Contents. „ What is a good ordering for testing the members of a product family ?“

lucian-woodward
Download Presentation

03.03.2014, Copenhagen

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multi-Objective Test Suite OptimizationforIncrementalProduct Family Testing(jointworkwith Hauke Baller, Sascha Lity and Ina Schaefer) 03.03.2014, Copenhagen

  2. Contents • „Whatis a goodorderingfortestingthemembersof a productfamily?“ • Extension oftheclassicalTest Suite Minimization Problem • Weights • Bounds • Variability • Approx. optimal solution(s) for Multi-Objectives via ILP Solving • Incrementalheuristicsolutionbased on dynamicprogramming toappear at ICST‘2014

  3. Test Suite Minimization Problem [Harrold et al., 1993] r3 Test Requirements r1 r2 r4 r5 r6 t1 t2 t3 t4 … Test Cases Complete Test Suite

  4. Example: Model-Based Testing • All-TransitionsCoverageR = {r1,r2, …, r8} • Complete Test Suite TS(4 Test Cases) • Minimal Complete Test Suite TS‘(1 Test Case) • Test Suites • TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} • TS‘= {(r1r2r3r5r4r7r8r5r6)} r1 r6 test model r2 r4 r5 r3 r7 r8

  5. Minimum Set Cover Problem • Assumptions: • All Test Cases are relevant: • Each Requirement is satisfiable: • Given: • Set of Requirements • Set of Test Cases • Sub sets satisfying Requirement • Find: • Minimum sub set such that • NP-Complete

  6. Cost/Profit Test Suite Optimization[Malishevsky et al. 2006, Xu et al. 2012, Rout et al. 2013] + + + + + = 2 1 4 3 1 2 Testing Profits pTS ≥ Profit Goal kp r3 Test Requirements r1 r2 r4 r5 r6 t1 t2 t3 t4 … Test Cases + + + = 4 2 1 4 TestingCostscTS ≤CostBoundkc Minimal Test Suite ∑=8 Adequate Test Suite ∑=7

  7. Example: Model-Based Testing withCosts/Profits Test Suite CostscTS • TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} • cTS = 2+2+3+3 = 10 • TS‘= {(r1r2r3r5r4r7r8r5r6)} • cTS = 9 • TS‘‘ = {(r1r4r5r2),(r3r7)} • cTS = 5+2 = 7 • Test Suite Profits pTS • TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} • pTS=(2+1)+(2+1)+(2+2+4)+(2+2+1)=15 • TS‘= {(r1r2r3r5r4r7r8r5r6)} • pTS=(2+1+2+2+1+2+4+2+1)=15 • TS‘‘ = {(r1r4r5r2),(r3r7)} • pTS=(2+1+2+1)+(2+2)=10 TS‘‘isAdequateTest Suite forkp= 10 r1(2) kp=10 r6(1) r2(1) r4(1) r5(2) Profits arecountedonce r3(2) r7(2) r8(4)

  8. Minimum Partial Weighted Set Cover Problem [Könemann et al., 2011] • Given: • Set of Weighted Requirements R • Set of Weighted Test Cases T • Profit goal kp • Find: • Sub set with minimal such that • pTS ≥ kp • where and

  9. Product Line Analysis

  10. Sample-basedProduct Line Analysis [Cohen et al. 2006], [Gustafsson 2007], [Oster et al., 2010], [Perrouin et al, 2010], [Kim et al. 2011], [Johansen et al., 2012], [Henard et al., 2013], [Haslinger et al., 2013], ….

  11. Family-basedProduct Line Analysis [Krishnamurti et al., 2006], [Gruler et al., 2010], [Classen et al. 2010], [Gnesi et al. 2011], [Cordy et al., 2013], …

  12. IncrementalProduct Line Analysis [Batoryet al. 2007], [Engström et al. 2011], [Lochau et al. 2012], …

  13. Family-basedProduct Line Test Suite Design • 4 ProductVariants • T={t1, …, t15} • t7=(r1r6r8r7) • t8=(r1r4r5r2) • t11=(r1r4r7) • t12=(r9r10r8) • t13=(r3r11) • t14=(r1r4r11) • t15=(r3r7)} r9(2) r10(2) r6(1) r1(2) r2(1) r4(1) r5(2) r3(2) r7(2) r8(4) r11(8)

  14. Complete Minimal Product Line Test Suite [Cichos et al. 2011] Assumptions • ifthen satisfies on • Given: • Set of Requirements R • Set of Test Cases T • Set of Product Variants P • Sub sets satisfying Requirement • Sub sets executable on product variant • Sub sets relevant for product variant • Find: • Minimum sub set such that

  15. Constraint Minimum Set Cover Problem 2 1 4 3 1 2 Testing Profits pTS≥kp rpTS≥krp r3 r1 r2 r4 r5 r6 Test Requirements t1 t2 t3 t4 … Test Cases 4 2 4 1 Testing Costs p2 p1 p3 ProductVariants ProductCosts/Profits ppTS = + + 5 2 3 ≥ Profit Goal kpp

  16. Constraint (Minimum) Partial Weighted Set Cover Problem • Given: • Set of WeightedRequirements • Set of WeightedTest Cases • Set of WeightedProduct Variants • Profit Goals • (Cost Bound • Find: • Sub set (with minimal such that • andand is an AdequatProduct Line Test Suite

  17. Integer Linear Programming (ILP) • (Integer) Decision variables • Linear objectivefunction • Linear constraints

  18. ILP forProduct Line Test Suite Optimization ppTS≥ kpp ppTS-Δpp=kpp Test Cases must beappliedto a Product t For all Requirements (of a Product) r rpTS≥krp cTS≤kc p t Binary Variables ppTS≥kpp Priotization Normalization Value p t Productselected Requirements must besatisfied pj select Test Case from Test Case Set NOTpjOROne Test Case from Test Case Set ppTS Requirementsatisfied For all Products, a Test Case must beselected Again (ri  Set)  (NOTriOR Set) Differences must not be negativ

  19. Re-FormulationasProductSelection Problem • ILP findsAdequate Test Suite (ifexists), but Problem isNP-hard • Another Point-of-View: IncrementalProductSelection r1 r2 r3 r4 r5 r6 rpTS t1 t2 t3 t4 cTS p1 p2 p3 ppTS

  20. AdequateProduct Sub Set • Given: • Set of WeightedRequirements • Set of WeightedTest Cases • Set of WeightedProduct Variants • Profit Goals • (Cost Bound • Find: • Sub set P such that an adequate test suite exists with

  21. IncrementalProductSelection p1 p1 + p3 + p5 p3 p2 p2 p4 p4 selectnextproduct profit goalsreached? costboundsexceeded? p7 p8 p5 p6 p9 noadequatesolutionfound isadequate isadequate

  22. ProductSelectionHeuristic • Are Profit Goals reached? (rpTS ≥ krp and ppTS ≥ kpp) • No: • For each unselected Product • While unsatisfied Requirements exist, select one: • With high profit and uniqueness/rareness • Select a satisfying Test Case : • With the ability to satisfy many other open Requirements for many other Products and low cost • Rank Products and Select • By: Product profit, resulting Test Case costs, and Requirement profit (weighted) • Recalculate ppTS,rpTSand cTS • Is Test budget excelled (cTS > kc) • Yes: “Error”  End • Goto 1. • Yes: “Success”  End cf. [Harrold et al., 1993] [Harrold et al., 1993] + Products ? ? ? n t n r ? ?

  23. Experimental Results • Varying: • Number of: Requirements, Test Cases, and Products • Mapping: Requirements/Test Cases and Test Cases/Products • Weights • In Total: 51 generated Data Sets • Time Limit: 24h 50, 100, and 500 Products 50, 100, and 500 Requirements 50, 100, and 500 Test Cases r1 r2 r3 r4 r5 r6 -/o/+ Heuristic is 21.5 timesfaster MeanandVariance MeanandVariance t1 t2 t3 t4 Efficencyimprovementfactor: 127.48 68.6% less accuracy p1 p2 p3 -/o/+

  24. Discussion & Future Work Summary • Heuristicproducesusefulresultsandscales • Feasibletradeoffbetweenaccuracyandefficiency Ongoingand Future Work • Backtracking ifboundsareexceeded • Search-basedoptimizationtechniques • Methodsandcriteriato find appropriateweightsandbounds • Beyond ILP: • Dynamic weightfunction, • On-the-flygenerationoftestartifacts, … • Applicationto Real-World Case Studies

  25. Industrial Case Study: Heidelberg Ion-Beam Therapy Center • DCU Case Study forBeam Cycle Creation • Complex Automation Engineering Software • Highly-Configurableand Runtime Adaptive • Re-Engineered as (D)SPL • ~ 200 Features (Boolean andNon-Boolean) • 150% Test Model ~ 640 Transitions • SPL Implementation in C~ 175.000 LOC DCU-R DCU-Z DCU-R DCU-P Gantry TreatmentRooms

  26. Model-Driven Development of DCU Code States Automata Stubs .c generate .h SFT Transition Trigger deploy insert Parameters + UserData Hand-written ECA Code + .h .c .h

  27. DCU SPL Test Model Specification + 150% SFT

  28. ThankYourForYour Attention! Multi-Mechanismen Adaption für das künftige Internet Integrated Model-basedTesting ofContinuouslyEvolving Software Product Lines

More Related