1 / 31

A Methodology and a Framework for Test Case Generation

A Methodology and a Framework for Test Case Generation. University of Geneva L.Lúcio, L.Pedro, D.Buchs. Talk Overview. (Our) Model-Based test case generation theory based on the work of Bernot, Gaudel, Marre; The user’s role in the test case generation framework; Test execution;

elmo
Download Presentation

A Methodology and a Framework for Test Case Generation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Methodology and a Framework for Test Case Generation University of Geneva L.Lúcio, L.Pedro, D.Buchs

  2. Talk Overview • (Our) Model-Based test case generation theory • based on the work of Bernot, Gaudel, Marre; • The user’s role in the test case generation framework; • Test execution; • The test generation machinery. L.Lúcio, L.Pedro, D. Buchs

  3. Program (P) Specification (SP) ╞ Exhaustive Test Set (TSP) ╞O Specification-Based Test Generation ╞ : program P satisfies (has the same semantics as) specification SP; ╞o: program P reacts according to test set TSP(as observed by an Oracle O). L.Lúcio, L.Pedro, D. Buchs

  4. TExhaustive= { formula,result | formula  composition of input,output pairs result = trueif formula models a valid behavior result = falseif formula models an invalid behavior } Exhaustive test set - Definition The exhaustive set of tests for a given specification can be formalized as: The exhaustive test set describes fully the expected semantics of the specification, including valid and invalid behaviors… L.Lúcio, L.Pedro, D. Buchs

  5. Pertinence and Practicability According to the previous slide, the following formula holds: IF test set TSP is pertinent – valid and unbiased • valid – accepts only correct programs (P╞o TSP) => (P╞ SP) • unbiased – doesn’t reject correct programs (P╞ SP) => (P╞o TSP) (P╞ SP) <=> (P╞o TSP) But, exhaustive test sets are not practicable in the real world (infinite testing time)… L.Lúcio, L.Pedro, D. Buchs

  6. Test Selection We thus need a way of reducing the exhaustive (infinite) test set to a test set that is practicable, while keeping pertinence… How do we do this? By stating hypotheses about the behavior of the program – the idea is to find good hypotheses that generalize correctly the behavior of the SUT! L.Lúcio, L.Pedro, D. Buchs

  7. Stating Hypotheses Example: consider as SUT a (simplified) embedded controller for a drink vending machine: Drink Vending Machine accept_coin insert_coin reject_coin select_drink (X) give_drink (X) not_enough_money eject_money Drinks available: Coke (2 coins), Water (1 coin) L.Lúcio, L.Pedro, D. Buchs

  8. Example test 1 Example test 2  <select_drink(water), give_drink>, <insert_coin, accept_coin>, false   <insert_coin, accept_coin>, <select_drink(water), give_drink>, <select_drink(water), not_enough_money>, true  Stating Hypotheses (2) Hypotheses 1: if the SUT works well for sequences of at least 3 operations, then the system works well (regularity); AND Hypotheses 2: if the system works well while choosing one kind of drink, then it will work well for choosing all kinds (uniformity). L.Lúcio, L.Pedro, D. Buchs

  9. Specification – Complementing human Hypotheses Example: imagine that from the two hypotheses in the last but one slide the test selection mechanism has chosen Water. There are then 3 interesting valid behaviors (tests): • The buyer doesn’t insert enough coins, chooses Water and gets nothing; • The buyer inserts enough coins, chooses Water and gets it; • The buyer inserts too many coins, chooses Water, gets it and the change. The points of choice stated in the operation of the specification can be used to add further behavior classification that can be combined with the hypotheses stated by the test engineer. We call this sub-domain decomposition. L.Lúcio, L.Pedro, D. Buchs

  10. Test set Reduction Process <H0, T0> Oracle hypothesis and exhaustive test set T0 H0 Reduce Test set Increase hypothesis T1 H1 … … Tk-1 Hk-1 Tk Hk <Hk, Tk> defined hypothesis and practicable test set L.Lúcio, L.Pedro, D. Buchs

  11. Full Picture Does P satisfy SP? P SP Test Selection (Hypotheses H Application) Test Set T Test Procedure Execution of P using T Correction of P Oracle P satisfies, or not, T! no inconclusive yes P satisfies, or not, H no yes P does not satisfy SP Undefined P satisfies SP! L.Lúcio, L.Pedro, D. Buchs

  12. Full pictureUser’s Role in the Framework Test validation + Spec. hypotheses Model Tests SUT (implementation) Selection Criteria Discussed later in the presentation + Hypotheses about SUT behavior Oracle Test Driver Test Results L.Lúcio, L.Pedro, D. Buchs

  13. Tests SUT (implementation) Selection Criteria + Oracle Test Driver Test Results Providing the Model… Test Validation + Behavior classification Model Tests SUT (implementation) Selection Criteria + Oracle Test Driver Test Results L.Lúcio, L.Pedro, D. Buchs

  14. Providing the ModelThe Modelling Language UML (Fondue specification) • Software development process for reactive systems • Purpose is to produce a description of: • The problem domain • The functional requirements of the system • What it provides • Concept Model: Static structure of the information in the system • Behavior Model: Input and Output communication of the system • Uses UML (Class / Collaboration / State) and OCL L.Lúcio, L.Pedro, D. Buchs

  15. OCL Operation Schema For X System State post-condition Operation X Output Event Providing the ModelExpressing Model Semantics • Describes operations on the system by operation schemas which specify operations by pre- and post conditions using OCL System State pre-condition • State change implies changing objects, attributes and relations described by the class model … L.Lúcio, L.Pedro, D. Buchs

  16. SUT Interface Operation’s signatures Tests Tests Selection Criteria Selection Criteria + + Mapping Oracle Oracle Test Results Test Results Model - SUT Mapping… Model SUT (implementation) Test Driver L.Lúcio, L.Pedro, D. Buchs

  17. Model – SUT MappingThe Test Driver • The Test Driver is the piece of software that allows executing (abstract) tests on the SUT; • Each model operation and operation output has to be connected with an SUT operation and output respectively; • Heterogeneous SUT interfaces: • Programmatic APIs • Web pages • Specialized graphical interfaces • Command line L.Lúcio, L.Pedro, D. Buchs

  18. Model – SUT MappingExample Model’s Operations insert_coin select_drink (X) eject_money Operations’ Output accept_coin reject_coin give_drink (X) not_enough_money L.Lúcio, L.Pedro, D. Buchs

  19. Model Model Tests Tests SUT (implementation) SUT (implementation) Selection Criteria Selection Criteria + + Oracle Test Driver Observes Observability hypotheses Test Results Test Results Providing the Oracle Hypotheses L.Lúcio, L.Pedro, D. Buchs

  20. Providing the Oracle HypothesesObserving test results • The Oracle will observe the driver while it applies <input,output> pairs to the SUT; • If the output is not exactly what the expected one , the Oracle Hypotheses provide an error margin… • If the output goes outside the established hypotheses, the test result is undefined! Example: An operation was performed on a web page and after 40 seconds there is still no reply… Example Oracle hypotheses: if in a test there is a delay of more than 30 seconds then the test doesn’t pass. L.Lúcio, L.Pedro, D. Buchs

  21. Model Model Tests Tests SUT (implementation) SUT (implementation) + + Hypotheses about SUT behavior Oracle Oracle Test Driver Test Driver Test Results Test Results Providing Hypotheses about the SUT Model Tests SUT (implementation) Selection Criteria + Oracle Test Driver Test Results L.Lúcio, L.Pedro, D. Buchs

  22. Providing Hypotheses about the SUTThe Hypotheses’ language We are concentrating both the languages to: • Express hypotheses– or constraints on tests • Express tests in Prolog!... Example: produce a set of tests where the user inserts coins and selects a drink. The number of coins should be inferior to 5, testing with one type of drink is enough and the behavior should be valid. tests(L) :- pattern(_,and(star(ev(insertMoney(1),_),N),ev(selectDrink(S),_)),L), lowerT(nat,N,5),uniform(L),valid(L,true). Result with(selectDrink(Coke),[insufficientFunds]) with(insertMoney(1),[]), with(selectDrink(Coke),[insufficientFunds]) with(insertMoney(1),[]), with(insertMoney(1),[]), with(selectDrink(Coke), [giveDrink(Coke)]) L.Lúcio, L.Pedro, D. Buchs

  23. Providing Hypotheses about the SUTThe Hypotheses’ language (2) Using Prolog we can now express hypotheses on: • action sequences: • Pattern of paths based on regular expressions: • Zero or more times the same operation(s): * • One or more times the same operation(s): + • Logical operators between operation sequences: AND,OR • variable value domains: • Explore all possible values of a given domain; • Regularity hypotheses: • Constrain the values of a domain. • Uniformity hypotheses: • Choose only one value belonging to a variable’s domain L.Lúcio, L.Pedro, D. Buchs

  24. Test Translation - example We have used a demo version of a test tool for web pages – QuicktestTM L.Lúcio, L.Pedro, D. Buchs

  25. Test Execution L.Lúcio, L.Pedro, D. Buchs

  26. Model Model Syntactic test creation Tests Tests SUT SUT Selection Criteria Selection Criteria + + Test validation Oracle Oracle Test Driver Test Driver Test Results Test Results Subdomain decomposition Test case generation Machinery L.Lúcio, L.Pedro, D. Buchs

  27. Operational specification model • Syntactically building tests while applying hypotheses isn’t a problem… • ... but validating tests requires an operational specification model! • …also required to find sub-domains for operation’s parameters! L.Lúcio, L.Pedro, D. Buchs

  28. Operational specification model: UML→CO-OPN→Prolog • CO-OPN is a formal OO specification language based on: • ADTs : data types (no persistent state); • Petri Nets : behavior and concurrency handling. • Prolog allows constrained execution of the Model… L.Lúcio, L.Pedro, D. Buchs

  29. UML→CO-OPN→PrologTranslation Tools • We have already a tool that translates CO-OPN into an operational Prolog model! • UML->CO-OPN (translation from UML MOF to CO-OPN MOF) UML CO-OPN Prolog Under Construction Exists L.Lúcio, L.Pedro, D. Buchs

  30. a: a1 PC: True (1) a: a1+1 PC: True (2 -true) (2 - false) a: a1+1 PC: a1 > 9 a: a1+1 PC: a1<=9 Subdomain decomposition with Prolog • We wish to find which are the possible behaviors of an operation; • Prolog’s backtracking mechanism is very useful for that task… Int foo (int a) { 1 a++; 2 if (a>10) 3 ...; 4 else 5 ...; 6 return a } backtracking point The constraints in PC (Path Condition) define a decomposition of the domains of the operation’s entry parameters… L.Lúcio, L.Pedro, D. Buchs

  31. Conclusions • The framework and the process seem to be in place; • We are working on the UML/COOPN translation; • Unified user interface for the framework needed; • Validation in industry needed; • Come up with domain specific hypotheses templates L.Lúcio, L.Pedro, D. Buchs

More Related