1 / 33

Test purposes: adapting the notion of specification to testing

Test purposes: adapting the notion of specification to testing. Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble, France. ASE2001, Nov 27-29 2001, San Diego. Given a black box And its specification.

gladys
Download Presentation

Test purposes: adapting the notion of specification to testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Test purposes:adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble, France ASE2001, Nov 27-29 2001, San Diego

  2. Given a black box And its specification Test cases compare the black box to its specification ?sqrt(4) sqrt !2 accept X = sqrt(X)*sqrt(X) and sqrt(X) >= 0 Conformance testing

  3. Unfortunately, complex applications often result in complex test cases! ?Money_Transfer(1000000€) e-Bank Money_transfer !OK accept Complex test cases

  4. It then makes sense to specify these complex test cases « I want a test case that will execute the Money_transfer procedure successfully! » It is the notion of test purpose! ?Money_Transfer(1000000€) !OK accept Specifying test cases

  5. The COTE project • COTE: • conformance testing of components vs UML specifications • Softeam, France Telecom, Gemplus, IRISA, LSR/IMAG • Languages and tools to express and exploit test purposes and test cases • (compilers from test cases to target technologies) • This work comes from a brainstorming activity around the notion of test purpose within the COTE project

  6. The notion of test purpose Test purpose : Description of a precise goal of thetest case, in terms of exercising a particular execution path or verifying the compliance with a specific requirement [ISO IEC 9646-1 …conformance testing methodology and framework…] • This notion is used: • As a structuring element in industrial test suites • In some tools based on formal specifications(SAMSTAG, TGV, TorX)

  7. Test purposes in TGV • TGV is a tool developed jointly by IRISA (Rennes) and Vérimag (Grenoble) • Test synthesis tool • Given • A specification of the system under test • A test purpose It generates one corresponding test case.

  8. Pay 1 or 2 coins ?coin(1) ?coin(2) ?coin(1) Ask for sugar (optional) ?sugar ?coffee Ask for tea or coffee ?tea ?coffee ?tea Get the beverage !coffee !tea !tea !sugar !coffee !sugar Loop Example : coffee machine

  9. Dynamic specification Test purpose Test case ?coin(1) ?coin(1) ?coin(2) ?coin(2) ?coin(2) ?coin(1) ?coin(1) ?sugar ?sugar ?coffee ?coffee ?coffee ?tea ?coffee ?tea ?tea ?coffee ?tea !coffee !coffee !tea accept !coffee !tea !tea !tea !sugar !coffee !coffee !sugar !coffee !sugar !sugar A sample synthesis TGV

  10. ?coin(1) ?coin(2) ?coin(1) ?coin(1) ?coin(2) ?coin(1) ?sugar ?coffee ?sugar ?tea ?coffee ?coffee ?tea !coffee !coffee accept !tea !coffee !sugar !coffee Test purposes as specification • A test purpose (P) is an abstraction of test cases • In TGV, a partial sequence of the test cases • A test purpose specifies a set of test cases

  11. !coffee accept Benefits of abstraction • Capture the essence of the test(here test the delivery of coffee) • Shorter to state than the full test case(clerical completion left to the tool) • Robust to evolutions of the dynamic specification(test cases are affected more often than test purposes, e.g. pay 3 instead of 2)

  12. Relations between test purposes (P), test cases (C) and specifications (S)

  13. ?coin(2) ?coffee !coffee !coffee accept !coffee test case not conforming to the specification Weakly_refines(C,P) • A test case « refines » the test purpose(if it includes the partial sequence) • But several irrelevant test cases refine the test purpose also.Þ Weak refinement!

  14. ?coin(1) ?coin(2) ?coin(2) ?coin(1) ?sugar ?coffee ?coffee ?tea ?coffee ?tea !coffee accept !coffee !tea !tea !coffee !sugar !coffee !sugar refines(C,P,S) • The refinement takes place in the context of the specification refines

  15. weakly_refines vs refines • Refines is a ternary relation • Weakly_refines is defined as weakly_refines(C,P) ÛE S ¥ refines(C,P,S) weakly_refines Test case (C) Test Purpose (P) refines Specification (S)

  16. Semantics for refines(C,P,S) • Depending on the languages used for C, P, and S, a semantics can be defined for refines(C,P,S) • Semantics of refines in TGV: C is one of the paths of the synchronous product of P and S • See also, in the paper:semantics of a test refinement relation in the context of the B method

  17. Semantics of weakly_refines • As a consequence: weakly_refines(C,P) ÛE S ¥ refines(C,P,S) gives a semantics to weakly_refines on top of the semantics of refines

  18. Test case (C) Test Purpose (P) refines conforms Specification (S) conforms(C,S) conforms(C,S) ÛE P ¥ refines(C,P,S) • Conforms checks that the test case corresponds to the specification

  19. Test case (C) Test Purpose (P) refines consistent Specification (S) consistent(P,S) consistent(P,S) ÛE C ¥ refines(C,P,S) • Consistent checks if the test purpose is compatible with the specification

  20. weakly_refines Test case (C) Test Purpose (P) refines conforms consistent Specification (S) Summary • If refines(C,P,S) has a semantics,it gives a semantics to the binary relations! • In some cases, it is possible to systematically construct the third element of the relation

  21. Test specificationvssoftware specification

  22. Software Specification (SS) refinesSW Program (M) Test case (C) Test Purpose (P) refines Specification (S)

  23. Automated Software Engineering • Formal specs as a basis for software tools: • Verification of specification consistency • Program synthesis • Program verification

  24. !sugar ?coin(*) !coffee reject accept Specification consistency often demonstrated by proving: E M ¥ refinesSW(M,SS) (there exists a model which refines the specification) corresponds to consistent(P,S) ÛE C ¥ refines(C,P,S) (a test purpose makes sense if there exists a test case) Tool Detection of inconsistent purposes (e.g. !sugar comes after !coffee) avoids useless searches

  25. Synthesis Major theme at ASE synthesisSW(SS)  M corresponds to test synthesis e.g., TGV can be seen as a function synthesisTGV(P,S)  C Good news for tools The search domain is restricted by S (more decidable?, less complex?)

  26. Verification checks refinesSW(M,SS)(program M is a valid refinement of specification SS) corresponds to3 possible test case verifications • refines(C,P,S)C is a refinement of P in the context of S • weakly_refines(C,P)C is a refinement of P in some context • conforms(C,S)C is a test case for S

  27. Tools for test verification • Synthesis involves searching a huge space of solutions • If verification is cheaper than synthesis,it provides a natural complement. • Two examples: • Evolution of specifications • Test case retrieval

  28. ?tea ?coffee ?choc Evolution of specifications • Evolution of S into S’ • Given a set of (Ci,Pi) for S • Verify refines(Ci,Pi,S’) for each (Ci,Pi) • When it fails, verify conforms(Ci,S’) • A second failure reveals regression of S’ • Example: No need to recompute existing test cases if the coffee machinealso serves hot chocolate

  29. ?coin(2) ?coffee !coffee accept ?coin(2) !coffee accept Test case retrieval Given a large set of test cases • Retrieve a test that weakly_refines a new test purpose • Minimize the set by identifying test cases that weakly_refine several test purposes weakly_refines weakly_refines

  30. Related workand conclusion

  31. 1st category: domain independent Coverage criteria Fault models (mutation testing) Analysis rules, strategies Random testing 2nd category: exploits domain knowledge Test purposes(based on reqts) Operational profiles Must be consistent with the reference Test selection criterion (aka test requirement) Reference (specification, program) Synthesis tool Test case(s) Test selection criteria

  32. Conclusion • Test purposes are a specification of test cases! • 4 relations between C, P, and S • Not specific to a given tool or language • Not specific to an application domain (TGV: reactive systems, B: transf. systems) • Tools: from ASE to Automated Test Eng. • Based on these 4 relations • Test verification complements test synthesis • Applications to specification evolution, test case retrieval,…

  33. Interactive Coffee Crafting

More Related