1 / 24

Automating Software Module Testing for FAA Certification

Automating Software Module Testing for FAA Certification. Usha Santhanam The Boeing Company. Automating Software Module Testing for FAA Certification. Challenges in code based software testing FAA certification testing Test Set Editor (TSE) tool set and process Regression testing

lloyd
Download Presentation

Automating Software Module Testing for FAA Certification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automating Software Module Testing for FAA Certification Usha Santhanam The Boeing Company

  2. Automating Software Module Testing for FAA Certification • Challenges in code based software testing • FAA certification testing • Test Set Editor (TSE) tool set and process • Regression testing • Benefits of TSE • Limitations of TSE • Summary

  3. Challenges in Code Based Software Testing • Software complexity and accelerated development schedules make avoiding defects difficult • Untested code in user environment • Difficulty of accurately capturing and analyzing the sequence of events that occur during program execution

  4. FAA Certification Testing • Structural coverage is often not achieved through requirements based testing alone • Testing each module in isolation provides the desired coverage • The three levels of structural coverage to deal with are statement coverage, decision coverage, and modified condition and decision coverage (MCDC)

  5. Structural Coverage • Statement coverage: execute each source level statement at least once • Decision coverage: take each execution branch at least once • MCDC: designed to demonstrate the independence of individual conditions that constitute a complex decision

  6. TSE Tool Set and Process • TSE is designed to achieve shorter turnaround time and cut cost during regression testing • TSE is domain-independent and removes the language-dependent test driver chores • TSE requires the test engineer to construct the test cases, but the rest of the testing process is automated

  7. TSE Tool Set and Process(cont.) • Tasks of testing a module for structural coverage are • Identifying inputs and outputs of a module (or collection of modules) • Selecting test cases and tagging them with the requirements they address • Constructing a test driver and stubs for subprograms called by the module under test

  8. TSE Tool Set and Process (cont.) • Compiling and running the tests • Analyzing coverage • TSE assists the tester with all these tasks except the selection of the test cases.

  9. package Io is Debugging : Boolean := False; procedure Get_End; end Io; with Text_Io; package body Io is K : constant := 1024; Max_Line_Length : constant := 16*K; Line : String (1 .. Max_Line_Length); procedure Error (Msg : String) is begin null; end Error; procedure Get_Line is begin null; end Get_Line; procedure Get_End is begin if Line (1 .. 4) = ".end" then Get_Line; else Error ("Missing .END, assume here."); end if; end Get_End; end Io; Example: Procedure Get_End in Package Io

  10. TSE_Tools.tcl

  11. Regression Testing • All test cases are self-checking • A batch file is used to run the test files through the TSE testing process • A batch log file is used to check for failures • Failed tests are examined and justified or updated

  12. Benefits of TSE • Tests can be developed by engineers who are not necessarily expert programmers in Ada • TSE includes a tool to analyze the source code and identify inputs and outputs of a given module • Provides coverage information

  13. Benefits of TSE(cont.) • TSE tests are easy to maintain due to uniformity of test definition files • TSE tests are repeatable with no manual interaction necessary • TSE tests can be executed on host or target platform using different support packages

  14. Productivity Improvements • Based on our experience we have seen a range of 1:5 to 1:10 productivity advantage. • Depends on the complexity of the module

  15. Estimate of TSE Based Testing • We have developed a tool that estimates the number of hours needed to test a module with TSE • Based on Statement count, number of If’s, Case’s, Loop’s, Exit’s, Logical operators, short circuit control forms, and stub calls • The Get_End procedure estimated testing time is 7 hours

  16. Estimate of Manual Testing • Estimated size of the get_end test driver is 90 SLOC • Including main driver and stubs • Programmer productivity for the test code development is 2 SLOC per hour (Hihm) • This yields the productivity improvement of 1:6.5 for get_end procedure

  17. Limitations of TSE • It is not possible to construct one generic module test to verify all instantiations • Handled on an instantiation by instantiation basis • Tasking is difficult to test because tasks run in independent parallel threads • Each subprogram can be tested independently regardless of the number of tasks sharing it

  18. Limitations of TSE(cont.) • For modules using private type parameters or returning private type, construction of external test driver is awkward • Embed the test driver in the body of the package that defines the limited private type • Nested subprogram testing not available • Requires the test driver to be nested • Must be tested in the context of enclosing subprogram

  19. Summary of TSE Benefits • Minimizes the need for programming knowledge • Cuts testing effort several fold over manual approach • Minimizes cost of retesting • Allows us to meet FAA/DO-178B requirements in cost effective manner

More Related