1 / 20

Injection of realistic software faults for experimental software risk assessment

Injection of realistic software faults for experimental software risk assessment. Henrique Madeira Critical Software, SA and University of Coimbra, DEI-CISUC Coimbra, Portugal. NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005. Universidade de Coimbra.

september-aguilar
Download Presentation

Injection of realistic software faults for experimental software risk assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Injection of realistic software faults for experimental software risk assessment Henrique Madeira Critical Software, SA and University of Coimbra, DEI-CISUC Coimbra, Portugal NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005 Universidade de Coimbra

  2. What’s the point of injecting software faults? • Software faults are most probably the major cause of computer system outages • Goals: • Experimental risk assessment in component-based software development • Dependability evaluation of COTS components • Robustness testing • Fault tolerance layer evaluation • Dependability benchmarking

  3. Output Input SW component under test Interface faults Output Input Target SW component Software faults Two possible injection points • Injection of interface faults in software components (classical robustness testing) Injection of realistic software faults inside software components (new approach)

  4. Example of results obtained with interface faults (robustness testing) Robustness failure in RTEMS 4.5.0 Excerpt of application code: requestedSize1 = 4294967295; returnStatus = rtems_region_get_segment (regionId, requestedSize1, option, timeout, ptsegment1); Result: Memory exception at fffffffc (illegal address) Unexpected trap (0x09) at address 0x0200aaac Data access exception at 0xfffffffc This software fault was discovered automatically by the Xception robustness testing tool!

  5. Software faults are injected in a device driver using the G-SWFIT technique. • The device is heavily used by programs running during test. Windows XP Availability Feedback Stability Windows 2000 Windows NT Worst Worst Worst Best Best Best Example of results obtained with injection of software faults What happens if a software bug in a device driver becomes active?

  6. Component-based software development • Vision: development of systems using pre-fabricated components. Reuse custom components or buy software components available from software manufactures (Commercial-Off-The-Shelf: COTS). • Potential advantages: • Reduce development effort since the components are already developed, tested, and matured by execution in different contexts • Improve system quality • Achieve of shorter time-to-market • Improve management of increased complexity of software • Trend→use general-purpose COTS components and develop domain specific components.

  7. Some potential problems • COTS • In general, functionality description is not fully provided. • No guarantee of adequate testing. • COTS must be assessed in relation to their intended use. • The source code is normally not available (makes it impossible white box verification & validation of COTS). • Reuse of custom components in a different context may expose components faults. Using COTS (or reusing custom components) represent a risk! How to assess (and reduce) that risk?

  8. Case-study: I-don’t-care-about software architecture diagram Software components Different sizes Different levels of granularity

  9. Question 1 This is a COTS! What’s the risk of using it in my system?

  10. Question 2 This is custom component previously built! What’s the risk of reusing it in my system?

  11. Question 3 This is a new custom component! What’s the risk of using it without further testing?

  12. Injection of software faults Software complexity metrics Experimental risk assessment Example of question: What’s the risk of using Component 3 in my system? Risk = prob. of bug * prob. of bug activation * impact of bug activation

  13. Again, two possible injection points Injection of interface faults Injection of realistic SW faults

  14. Injection of SW faults Injection of SW faults Why injection or real software faults? • Error propagation through non conventional channels is a reality. • Faults injected inside components are more representative.

  15. How to inject software faults? • Use G-SWFIT (ISSRE 2002, DSN 2003, DSN 2004) • Injects the top N most common software faults. • This top N is based on field data (our study + ODC data from IBM) and corresponds to ~65% of the bugs found in field data. • Injects faults in executable code. • Largely independent on the programming language, compiler, etc that have generated the executable code. • G-SWFIT is now a reasonably mature technique.

  16. 01011000100X001 010110001X01001 010110X01001001 01X110001001001 G-SWFITGeneric software fault injection technique Target executable code Low-level code mutation engine Low level mutated versions 010110001001001 ... Library of software fault injection operators Emulate common programmer mistakes The technique can be applied to binary files prior to execution or to in-memory running processes

  17. Injection of software faults Software complexity metrics Experimental risk assessment (again) Example of question: What’s the risk of using Component 3 in my system? Risk = prob. of bug * prob. of bug activation * impact of bug activation

  18. Target code Estimation of the probability of residual bugs • Many studies indicate that fault probability correlates with the software module complexity • Metrics of software complexity base on: • Static feature of the code; • Dynamic features; • Possible information on the development process (type of tests, etc); • ...

  19. Target code Software faults Estimation of bug activation probability and bug impact • Test campaigns to evaluate the activation probability and the impact of software faults (bugs) inside the component in the rest of the system. • Use software metrics to choose the modules to inject faults and define trigger locations accordingly.

  20. Conclusions and current work on experimental risk assessment • Experimental software risk assessment seems to be viable. • Risk is a multi-dimensional measure. Many software risks can be assessed, depending on the property I’m interested in. • Current work: • Improve the G-SWFIT technique: • Improving current tool. • Expansion of the mutation operator library • Construction of a field-usable tool for software fault emulation in Java environments • Study of software metrics and available tools. • Composeability measures. • Real case-studies to demonstrate the methodology.

More Related