1 / 62

Algorithms For Optimizing Test Cases

Algorithms For Optimizing Test Cases. Presented by team 4 Jim Kile Don Little Samir Shah. Software Testing – So What?. July 28 1962 – Mariner I space probe Mission control destroys the rocket 1985-1987 – Therac-25 medical accelerator At least five patients die Others seriously injured

hisa
Download Presentation

Algorithms For Optimizing Test Cases

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Algorithms For Optimizing Test Cases Presented by team 4 Jim Kile Don Little Samir Shah

  2. Software Testing – So What? • July 28 1962 – Mariner I space probe • Mission control destroys the rocket • 1985-1987 – Therac-25 medical accelerator • At least five patients die • Others seriously injured • November 2000 – National Cancer Institute Panama City • At least eight patients die • Another 20 receive significant overdoses • Physicians indicted for murder

  3. What Is Test Case Optimization? • Typically applies to unit test cases where coverage approaches 100% • Implies ordering execution such that: • Rate of fault detection is increased • Amount of time to perform regression is reduced • Elimination of unnecessary test cases during regression runs

  4. What Are Goals For Test Case Execution? • Increase the rate of early fault detection and correction • Find bugs early so they can be corrected early • Regression test only those areas that have changed • Reduce the amount of time to execute full unit regression test suites

  5. Why Is Test Case Optimization Important? • Execution complexity • Non-linear problem • Elapsed time to execute • Computing resources • Human resources • Wait time for test completion • Late defect detection and correction

  6. Who Develops Test Cases? • Developers • Dedicated quality assurance staff • Automated test generation techniques

  7. What Are Benefits Of Automated Test Generation Techniques? • No cognitive bias • Better able to: • Generate test cases that concentrate on error prone areas • Produce highly novel test cases • Better coverage overall

  8. Why Automated Test Optimization? • Generating set of basic test cases - easy • Easily cover 50–70% of faults • Improving a test set’s quality - hard • Improving to 90–100% • Time consuming and expensive

  9. What Is Difference Between Optimization And Test Case Generation? • Optimization problem • Single goal is sought • Test case generation • No single goal • Optimized coverage of code under test

  10. What Are Benefits Of GA Vs. Random Test Generation? • Random test generation • Uniform distribution • GA-based search process • More focused test set • Set focuses on identified flaws • Some highly novel test cases

  11. How Do Genetic Algorithms Work? • A GA operates on strings of digits called chromosomes • Each digit that makes up the chromosome is called a gene • Collection of such chromosomes makes up a population

  12. How Do Genetic Algorithms Work? • Each chromosome has a fitness value • Fitness value determines probability of survival in next generation

  13. How Do Genetic Algorithms Work? • Algorithm begins with random population • Algorithm evolves incrementally – generation • Produces a structure from iterative development • Reproduction • Combine with another chromosome (crossover) • Adjusted slightly (mutation) • Original chromosome may have a poor/ low fitness • Create offspring with much higher fitness

  14. Uses of GA’s In Testing • Generate a range of effectivetest data with fault revealing power • Both papers used this technique • Introduce faults in software under test to determine effectives of test cases • Only one paper used this technique

  15. How Is Quality Of Test Cases Evaluated? • Number of detected injected faults • Killed by the test case • Otherwise, it’s alive • Develop a score • Errors killed by test case • Divided by test set

  16. How Does GA Work Specifically?Example • Genetic algorithm • Generates random population of binary digits • For example each chromosome may be 36 bits long • Each twelve bit segment representing one of the sides of a triangle • Chromosomes cross between gene 5 and 32

  17. How Does GA Work Specifically?Crossover Before crossover between gene 5 and 32 1)111001110101 100101100110 001010111000 2)111101011010 100101101010 101110110100 After crossover 1)111001011010 100101101010 101110111000 2)111101110101 100101100110 001010110100

  18. How Does Mutation Work Specifically? • Mutation will randomly switch genes in population • Gene 23 in chromosome 1 was switched from a 1 to 0

  19. How Does Mutation Work? Before mutation of gene 23 chromosome 1 1)111001110101 100101100110 001010111000 After mutation 1)111001110101 100101100100 001010111000

  20. Generation Of Next Population • Based on a “roulette wheel” • Where fitness determines the probability of selection • Those with higher fitness more chance of offspring in the next generation in comparison to their less fit companions

  21. Dynamic Software Testing Techniques • Structural – first paper • Code coverage • Boundary conditions • Individual or combined statement traversal • Path coverage • Functional – second paper • Confirms that a function from specification is correctly implemented • No analysis of the structure of the program

  22. First paper Automatic test case optimization: A bacteriologic algorithm Benoit Baudry, Franck Fleurey, Jean-marc Jézéquel And Yves Le Traon

  23. Contribution • Finding an optimal set of test cases through revealing a test case’s “fault revealing power” • Building confidence in the test suite through “mutation analysis”

  24. Bacteriologic AlgorithmTheoretical Basis • Adapted from genetic algorithms • Inspired by evolutionary ecology and bacteriologic adaptation • Similarities in this problem domain • Can’t generate a single perfect test suite

  25. Bacteriologic AlgorithmFramework

  26. Bacteriologic AlgorithmBasic Functions • Initialization • Iterate incrementally creating new generation • Limitation • Only works on test cases of similar size

  27. Bacteriologic AlgorithmInitialization • Initial test cases either written by hand or automatically generated • For the experiment test cases were randomly generated • Initial size set to 25 nodes

  28. Bacteriologic AlgorithmComputing Fitness • Tool used to generate test case mutants • Uses the mutation score of a set of test cases as that set’s fitness function MS(T) = 100(d/(m - equiv)) • Test cases are executed to determine how many mutants they can kill • Global mutation score computed

  29. Bacteriologic AlgorithmMemorization • Used to compute relative fitness • Test case mutation score relative to the solution set’s mutation score • Test cases are selected whose relative score exceeds the memorization threshold

  30. Bacteriologic AlgorithmMutation • Randomly selects test cases • Selection is weighted by relative fitness of the test case • Selected cases and code are mutated to create new cases for the next generation • Code is represented by an abstract syntax tree • Nodes in the tree are replaced

  31. Bacteriologic AlgorithmMutation – Abstract Syntax Tree • A finite, labeled directed tree • Nodes are labeled by operators • Edges represent operands • Leaves contain variables or constants • Used in a parser • Range of all possible structures defined by the syntax

  32. Bacteriologic AlgorithmMutation – Abstract Syntax Tree Example x = a + b; y = a * b; while (y > a) { a++; x = a + b; }

  33. Bacteriologic AlgorithmFiltering • Filtering = removing • Two different implementations • Delete any test case whose relative mutation score is equal to 0 • That is the function kills no mutant that the test cases in the solution set haven’t killed • Reduce the coverage matrix by deleting redundant test cases

  34. Bacteriologic AlgorithmResults

  35. Bacteriologic AlgorithmResults • Comparison with genetic algorithm • Both ran 50 times • Genetic algorithm results • 200 generations created • Average mutation score of 85 • Required executing an average of 480,000 test cases

  36. Bacteriologic AlgorithmResults • Bacteriologic algorithm results • 30 generations created • Average mutation score of 96 • Required executing an average of 46,375 test cases

  37. Second paper Breeding software test cases with genetic algorithms D. Berndt, J. Fisher, L. Johnson, J. Pinglikar And A. Watkins

  38. Focus • Breeding software test cases using genetic algorithms as part of a software testing cycle • Uses automated test generation techniques • Evolving fitness function • Relies on fossil record of organisms • Search behaviors • Novelty • Proximity • Severity

  39. Genetic AlgorithmSimple Triangle Classification Program (TRITYP) • Classify triangle by type • Three sides of the triangle • Parameters x, y, and z • Range 0 – 2000 • Search space • Illegal / legal triangle

  40. Genetic AlgorithmApproach • Flaws were intentionally introduced into data for testing purposes • Errors introduced for specific ranges of x and y parameters • X coordinate between 500 - 1000 • Y coordinate between 0 – 500 • Result in error

  41. Genetic AlgorithmSearch Space Illegal/Legal Triangle

  42. Genetic AlgorithmHow does it work specifically? • Generates random population of x, y and z coordinates as binary digits • Each chromosome is 36 bits long • Each twelve bit segment representing one of the sides of a triangle

  43. Genetic AlgorithmFitness • Relative fitness function • Compares • Particular chromosome’s fitness • Historical information from the fossil record

  44. Genetic AlgorithmGenerating Software Test Cases • Variety of sources into test case breeding with genetic algorithms • Powerful evolutionary • Naturally parallel computational engine • Balance fitness with diversity • Wide variety of test cases can be bred • Concepts of novelty,proximity and severity • Used to create a relative or changing fitness function

  45. Genetic AlgorithmBreeding Software Test Cases • Using genetic algorithms • Evolving fitness function • Fossil record of organisms • Interesting search behaviors • Novelty • Proximity • Severity

  46. Genetic AlgorithmNovelty • Measure of the uniqueness of particular test case • Quantified by measuring distance in parameter space from previous invocations stored in the fossil record

  47. Genetic AlgorithmProximity • Measure of closeness to other test cases that resulted in system failures

  48. Genetic AlgorithmSeverity • Measure of the seriousness of a system error

  49. Genetic AlgorithmDiversity • Used to avoid being trapped by local maxima • Generation of test cases diversity means • Emphasizing novelty • Downplaying proximity • Simple rules complex behavior • Explorers • Prospectors • Miners

  50. Genetic AlgorithmExplorer • Highly novel test case • Spread across the lightly populated regions of the test space • Once an error is discovered - fitness function encourages more thorough testing of the region

More Related