Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing

1. Test-Suite Reduction for Model Based Tests:Effects on Test Quality and Implications for Testing Written by Heimdahl and DevarajDept. of Comput. Sci. & Eng., Minnesota Univ.(19th International Conferenceon Automated Software Engineering 2004) Presented by Jae Ho Jung (Some slides from George Devaraj)

2. Naive test set generationresults in lots of redundanttest cases Cost of maintaining andrunning large test sets is high Test suite reduction techniques based on structural coverage help reduce these costs Drawback: This mightaffect the quality of test sets Test-suite Reduction What is the effect of test reduction on the fault finding ability of test sets?

3. Previous Empirical Studies • Wong et.al [1994] • “Test set minimization keeping coverage constant results inlittle or no reduction in its fault detection effectiveness” • Rothermel et.al [1997] • “Fault finding capability was significantly compromised on test-set reduction” • Same results were also observed by Jones and Harrold [2003] • How is this paper’s work different? • Focusing on testing of formal specifications • Addressinga wider range of test adequacy criteria

4. Model-Based Development This paper focusedon test-suites for Conformance testing Specification Centered Development Process

5. ※ Specification based test case generation using model checkers Test sequence generation architecture using model checker (cited from “Auto-generating test sequences using model checkers:A case study”) ※ RSML−eis based on the Statecharts like language Requirements State MachineLanguage (RSML). RSML−e is a fully formal and synchronousdata-flow language without any internal broadcast events (the absence of events isindicated by the−e).

6. Hypothesis • Hypothesis 1: • Hypothesis 2: Test reduction of a naively generatedspecification based test-set can producesignificant savings in terms of test-set size Test reduction will adversely affect the fault-finding capability of the resulting test set

7. Case Example:FGS (Flight Guidance System)

8. VerificationResult TestSequences User RSML-e NIMBUS Environment Verification PropertySpecification Simulator NuSMV(Model Checker) Test Generation Translator NuSMV InputLanguage Test Case Generator TrapProperties Verification Framework NIMBUS tools framework

9. The Experimental Setup • Fault Classes • Variable Replacement (VR) • Condition Insertion (CI) • Condition Removal (CR) • Condition Negation (CN) • Experimental Subject • FGS Model expressed in the RSML-e formalism • Fault Injection • Creation of “mutant” FGS specifications • Full Test-set generation for six test coverage criteria • Reduced Test-set generation • Simple greedy based heuristic algorithm used • Fault Detection • Running reduced as well as full versions against the “mutant” specs • Test Coverage Criteria • Variable Domain Coverage • Transition Coverage • Decision Coverage • Decision Coverage withSingle Uses • Modified Condition andDecision Coverage (MC/DC) • MC/DC with Single Uses

10. Algorithm for test-suite reduction • Reduced Set O  {F} • Greedy choice: Add a testcase to O which improvesthe existing coverage

11. Modified Condition/Decision Coverage S1: a decision statement A,B,C: three conditions The truth vector “FFF” is not a truth value for C(F) because it is not part of MC/DC pair for C. “FTT” and “TFF” does not satisfy the MC/DC Criterion for C <Partial Program P> • MC/DC is a stricter form of decision coverage • MC/DC require execution coverage at the condition level • MC/DC requires that each condition in a decision be shown by execution to independently affect the outcome of the decision (MC/DC requires that MC/DC pairs be covered for each condition) FFT FFF <Truth value, Truth vector, and MC/DC pair>

12. Experimental Results and Analysis:Full Test Set Full test set generation for various criteria along withtheir fault detection capability

13. Experimental Results and Analysis:Test-Suite reduction Reduced test set sizes for various test reduction runs

14. Experimental Results and Analysis:Effect on Fault Finding Capability Fault finding capability of the reduced test-sets

15. Threats to Validity • Using only one instance for investigations • Nevertheless useful because it was performed on a real world case example • Using seeded faults in experiment (not actual faults) • Using a single fault per specification • In practice, specifications may have more complex error patterns • However, single fault specifications more easier to run and control than multiple fault specifications

16. Future Works • Further experimentation needed • More experimental instances and multiple fault specifications • More robust coverage metrics needed • Skeptical towards the use of test-suite reduction techniques in the formal model based domain • More works are needed to determine how to prioritize test cases and empirically evaluate • Test-case prioritization would be a better approach than test-suite reduction or minimization

17. Questions • What makes the conflicting evidence in the previous empirical studies on this issue (the effect on the fault finding capability of the reduced test-suites)? • Considering test-suite prioritization algorithms. What could be a attribute of that algorithms?

18. Conclusions • Reduction of test-suite size has an unacceptable effect on the suite’s fault finding capability - test-suite reduction may not be an effective means of reducing testing time • More rigorous criteria, such as MC/DC, provide a better fault finding capability than less rigorous criteria, such as variable domain and transition coverage