1 / 26

Firewalls for regression testing

Firewalls for regression testing. Spring 2013. Use of the word “Firewalls”.

kermit-burt
Download Presentation

Firewalls for regression testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Firewalls for regression testing Spring 2013

  2. Use of the word “Firewalls” The way the authors White, Jaber, Robinson and Rajlich use the term “Firewall” in this work is special for their work on regression testing and must not be mixed with the common use of the term “firewall” which is part of networks and security.

  3. Why firewalls in regression testing Most regression tests are large and time-consuming to run. The concept of a “firewall” is used to reduce the set of classes – or components – that need to be tested. A firewall in regression testing separates the classes that depend on the class that is changed from the rest of the classes. There are two central concepts – dependency and encoding

  4. Dependency The main concept to understand when we apply the firewall idea is dependency. The main question is: Is the component dealing with the plain values of the input or Must the component account for how those inputs were generated

  5. Definitions – 1 Let C be a component with • inputs {Ii} • outputs {Oj} • post-conditions {Qk} If Q  {Qk} and I  {Ii} so that Q depends on I and on a value generated by a component C1 that is connected to I through a data flow through one or more intermediary component C2,…Cm then • I is an encoded input • C is an external component in scope of C2 and visa versa

  6. Definitions – 2 No such component C1 => I is a plain input, i.e. not encoded All inputs of C are plain => C is an I/O component C is external in scope of D and there is no message sent from C to D => there is a hidden dependency between C and D. I I c c … c1 c2 c1 cm I is a plain input I is an encoded input

  7. Dependency example – 1 Component 1 generates the key k, which is converted to an integer in component 2. Component 3 converts the integer to a hash address A. Component 4 uses the hash address to do a table look-up to generate the value f(k). If component 2 or 3 is changed, the result r may change.

  8. Dependency example – 2 The post condition of component 4 depends on k, generated by component 1 => input A is encoded by component 2. Component 4 is external in scope of component 2 and visa versa. r = f(k) but k is changed by component 2 There is a hidden dependency between 2 and 4

  9. Example of hidden dependency A’s post-condition is dependent on the value “colour” that is • Generated in class I • Encoded as an integer in class A (0 is red, 1 is yellow etc.) • Decoded in class B (receives colour code, decodes and paints the screen) Thus class B is external in scope of A. The dependency between A and B is hidden since no message is sent between A and B

  10. Global variables Activation path: 1, 2, 3, 4 Messages path: 1, 2, 3, g, 4 The output v3 from component 3 is used to modify the global variable g which is used as input to component 4. Component 4 is external and dependent on variable v1 and is activated by a message from component 3 to component 4 which has no data associated with it

  11. Simple firewall example: a TFW TFW does not examine dependencies further away than one level from changed component

  12. Process for determining TFW in OO systems • Given two successive versions of an oo-system, find the difference of the two versions and identify those classes that have changed • If any of the changed classes are in an inheritance hierarchy, also consider descendants of the changed class as changed • For each changed class, identify all the classes that send messages to the changed class or receive messages from the changed class and include them in the TFW • See previous slide for a graph

  13. Extended Testing Firewall - ETW • EFW extends the TFW by taking into account the data flows, external components and hidden dependencies (see previous slides) • EFW is obtained by following the process: • First apply steps 1-3 in order to obtain TFW • Identify all data paths from and to a modified class • Find all external classes in scope of the modified class (E in the example on next slide). Include all such classes in EFW

  14. Extended firewall – example

  15. Difference between TFW and EFW

  16. Empirical Study 1: TFW vs. EFW • In order to compare the concepts of Simple (TFW) and Extended Firewall (EFW), the authors tested several builds of a TelComm system. • 66 classes, 11000 lines of code, C++ • This system had already been tested by the developers using their standard test suite. • Common – found using both TFW and EFW: 8 errors • Found only using EFW: 4 errors. Two of these had not been found by the company’s test suite.

  17. TelComm system- results

  18. TelComm system: Severity of faults • All faults in previous slide were of severity 3 except for the EFW fault in build 1.1 which is of severity 2 • No faults of severity 1 or 4

  19. TelComm system: Extent of extra effort • In the table on next slide, total time for testing is the sum of • Analysis time – identifying the firewall • Testing time is the time needed for • test setup • test execution • analysis of test results. • EFW testing requires approximately • 20 – 30% more tests • 40 – 50% more person hours

  20. TelComm system: Extent of extra effort • TFW: 46 person hours, 8 errors => 5.7 person hours per error found • EFW: 58 person hours,12 errors => 4.8 person hours per error found

  21. TelComm system: Reused vs. new tests For TFW some testes were reused from the company’s original test suite All extra tests needed for EFW were written from scratch.

  22. Empirical Study 2 TFW vs. ETF • An empirical study was also performed on a very large system • more than 1 mill lines of code • thousands of classes • C++ code • Real time system • Windows environment

  23. Real time system: TFW vs. EFW • 13 common faults and 3 faults found only by EFW • Version 1: Common faults: 3 at sev. level 4, 1 at level 3, 1 at level 2 (EFW) • Version 2: Common faults 8 (6 at lev. 4, 2 at lev. 2), • EFW faults, one at level 3 one at level 2

  24. Real time system: number of tests and effort • TFW: 93 person hours, 13 errors => 7.2 person hours per error found • EFW: 120 person hours, 16 errors => 7.5 person hours per error found

  25. Conclusion • Extended Firewall testing (EFW) finds more faults than Simple Fire Wall (TFW) testing • 50% in the first study, 23 % in the second study • The cost in person hours per fault found is approximately the same: 5 - 8 person hours per fault • EFW and TFW • should only be used to test updates and changes • assumes that the initial test suite exists and has a high quality

  26. Reference Most of the information on these slides are from the paper (available from It’s learning) “Extended firewall for regression testing: an experience report” By: Lee White, Khaled Jaber, Brian Robinson and Václac Rajlich From: Journal of Software Maintenance and Evolution: Research and Practice, 2008, 20:419-433

More Related