1 / 25

When to Test Less

When to Test Less. Presented by Lan Guo. Introduction (1). Methods of software testing: functional, coverage, and user-oriented Phases of software testing: unit, integration, product, or regression

gavan
Download Presentation

When to Test Less

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. When to Test Less Presented by Lan Guo

  2. Introduction (1) • Methods of software testing: functional, coverage, and user-oriented • Phases of software testing: unit,integration, product, or regression • White-box (coverage) testing uses the structure of the software to measure the quality of testing.

  3. Introduction (2) • Statement coverage • Decision coverage • Data flow coverage • Mutation testing • Black-box testing aims at the specific features of the software. • Generate an operational profile, which is a list of occurrence probabilities of each element in the input domain of the program

  4. Introduction (3) • The operational profile is dependent on the input from the customer. • Difficulties in black-box testing • new software, new feature, no customer base • inadequate test set • coarse features, interacting features • multiple and unknown user groups

  5. Introduction (4) • There is a saturation effect associated with ALL testing methods. Reliability mutation Data flow decision functional Testing effort

  6. Introduction (5) • Elaborate and expensive testing does not tend to find more faults than inexpensive manual or simple automatic testing schemes.

  7. Theory of Testing (1) • In black-box testing, the probability to find a fault in N random tests is y = 1 - (1 - x)N x: the probability of finding a fault for a random input • Black-box testing can be expensive. If x = 0.0001, 46,000 tests are needed to be 99% sure to reveal a fault.

  8. Theory of Testing (2) • Formal-methods testing is even more expensive compared to black-box testing. • While-box testing only slightly reduces the number of tests required by black-box testing. In addition, the partition cost is very high.

  9. Reality of Testing (1) • Software-in-the-small projects produce software without rigorous testing. • The software does not crash more often. • A program’s shape is the shape of the pathways within the program. The more tangled the pathways, the more complex the program shape.

  10. Reality of Testing (2) • The simple shapes are common, which is demonstrated by du-pathways, control-flow diagrams and saturation effect. • Saturation effect is consistent with programs containing either many portions with simple shapes--so the portions are easily reached-- or many portions that are so twisted in shape that we will never reach them.

  11. Average Shape of Software • Right shape: the program is either too complex to test or too simple to test elaborately. • In either case, there is no point to conduct a lengthy and expensive test. • Most programs are indeed the right shape for simple testing. Proof: A mathematical model of building a pathway across a program from randomly selected inputs to some randomly selected state.

  12. Assumption 1 • Programs are networks connecting system concepts. (NAYO) • Optimizing compiler builds a network (control and data-flow graphs) from the code. • An executiontrace of a procedural programshows what statements are executed, which subroutines are care called, and in what order. • A verification condition generator creates a proof tree (execution tree), including preconditions and post-conditions

  13. Assumption 2 • A fault explanation tree is a tree whose leaves are inputs and whose root is a fault. • If we arrive at a state which should not happen in this tree, it is a bug in our system.

  14. Assumption 3 • Testingis a process of trying to generate a fault explanation tree from the program network. • If we can’t generate such tree, we are confident that there is no fault in our program.

  15. Assumption 4 • NAYO tree (explanation tree) N: no edge A: and node Y: yes edge O: or node • No edge only connects or nodes • Yes edge connects both and nodes and or nodes

  16. Assumption 5 • Testing is the process of extracting NAYO trees (the explanations) from NAYO networks (the program).

  17. Mathematics Equations (1) • Given in inputs to a NAYO network with V nodes, the probability to find a fault is: xo = in/V (A) • The probability of reaching an and node with andp parents is the probability of reaching all its parents: xand = xiandp (B)

  18. Mathematics Equations (2) • The probability of reaching an or node with orp parents is the probability of not missing all of its parents: xor = 1 - (1 - xo)orp (C) • If the ratio of and nodes in a NAYO tree is andf, then the ratio of or nodes in the same tree is orf = 1 - andf. The probability to reach some random node xj is: xj = andf * xand + orf * xor (D)

  19. Mathematics Equations (3) • Rearrange Equation A to isolate the number of tests required to be 99% sure of finding a fault with probability xj is: y = 0.99 = 1- ((1 - xj)N) Therefore, N = log(1- 0.99) log(1-xj)

  20. Implementation • We can approximately categorize the tree’s shape by the number of tests N required to reach that tree’s root to find a fault. • Results from the Simulation Runs Classification Threshold Percent Simple 0 < N < 102 36 Moderate 102 <=N<104 19 25 Hard 104<=N<106 N >= 106 20 Overly complex

  21. Conclusion • Black-box mathematics is blind to the internal structures of the program. If we include internal structure in the analysis, the result is more optimistic. • The average case analysis of the shape of programs indicates that simple and RANDOM testing can be very valuable.

  22. Related Research • Simple and random testing can probe non-determinate systems. • Limited testing is adequate for knowledge based systems.

  23. Potential Future Research • Extending static code analysis tools (optimizer and verification condition generator) to extract the parameters in the mathematical model from real-world programs, in order to monitor the change of program’s shape. • Offering design guideline to avoid hard-shaped programs.

  24. References (1) • T. Menzies and B. Cukic, “When to Test Less”, IEEE Software, September/October 2000, pp. 107-112 • D. Hamlet and R. Taylor, “Partition Testing Does not Inspire Confidence”, IEEE Trans. Software Eng., Vol. 16, No. 12, Dec. 1990, pp. 1402-1411 • J. Horgan and A. Mathur, “Software Testing and Reliability”, The Handbook of Software Reliability Eng., M.R. Lyu, ed., McGraw-Hill, New York, 1996, pp. 531-565.

  25. References (2) • T. Menzies and B. Cukic, “Adequacy of limited testing for knowledge based systems”, International Journal on Artificial Intelligence Tools, Vol. 9, No. 1, 2000, pp. 153-172 • T. Menzies, B. Cukic, H. Singh, and J. Powell, “Testing Nondeterminate Systems”

More Related