1 / 38

Specification-based Protocol Testing

Specification-based Protocol Testing. Hyoung Seok Hong Oleg Sokolsky CSE 642. Outline. Introduction Specifications, implementations Conformance relations Tests, testing architectures Assumptions, etc. Formal protocol specifications Finite-state machine (FSM) testing

Samuel
Download Presentation

Specification-based Protocol Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Specification-based Protocol Testing Hyoung Seok Hong Oleg Sokolsky CSE 642

  2. Outline • Introduction • Specifications, implementations • Conformance relations • Tests, testing architectures • Assumptions, etc. • Formal protocol specifications • Finite-state machine (FSM) testing • Extended FSM (EFSM) testing • Test generation for StateCharts

  3. Testing problem • A specification gives a description of the system behavior. • An implementation should behave according to the specification. • Is this really true? a a b c b b a

  4. Black-box vs. White-box testing • Black-box: unknown system structure • Testing against reference specification • Use the specification interface for • test selection, coverage criteria, result analysis • White-box: known system structure • Testing against reference specification • Use system structure for • test selection, etc.

  5. IUT – implementation under test • PCO – point of control and observation Tester PCO IUT Protocol testing architectures Local architecture

  6. Coordinated architecture Remote architecture PCO1 Upper Tester Lower Tester IUT PCO2 underlying service layer Protocol testing architectures Distributed architecture

  7. General testing architecture IUT Tester Tester PCO PCO PCO Tester

  8. Tests and test suites • A test case defines: • A finite sequence of inputs • A finite sequence of expected outputs • A test suite is a collection of tests to achieve a certain test coverage • Testing process • test generation: construct a test suite • test application: execute each test • result evaluation: interpret outputs

  9. Test structure • Test purpose • e.g., identify a state of the implementation • Test preamble • lead the IUT into known state • Test body • invoke behavior corresponding to the test purpose • Checking of test results

  10. Abstract models make reasoning easier conformance relation abstract model of specification abstract model of implementation assumptions and test hypothesis assumptions and test hypothesis Modeling approach • Testing criteria on a concrete system are hard to formulate detailed specification implementation ?

  11. Outline • Introduction • Formal protocol specifications • Formal specification languages • Finite-state machine (FSM) testing • Extended FSM (EFSM) testing • Test generation for StateCharts

  12. Formal specification languages • Extended FSM • SDL • Estelle • StateCharts • Process Algebras • LOTOS • Algebraic Specification • Z, VDM

  13. Outline • Introduction • Formal protocol specifications • Finite-state machine (FSM) testing • FSM model • FSM fault models • FSM test selection • Extended FSM (EFSM) testing • Test generation for StateCharts

  14. A set of states • An initial state • Transfer function • Output function x1 /y1 x3 /y3 x2 /y2 FSM model • Finite sets of input (X) and output (Y) events

  15. FSM-based fault model • Output faults • Transfer faults • Transfer faults with additional states • Additional or missing transitions • Control and data flow faults?

  16. Testing based on FSM models • Testing methods based on control flow • Transition-tour (TT) method • a single sequence of inputs to traverse all transitions • simple but weak • Unique-input-output (UIO) method • a test to identify each state in the specification • Distinguishing sequence (DS) method • identify each state by the same test body • provides full coverage, but may not have a DS

  17. UIO testing method • Assumption: for each state, there is an input sequence that produces a unique output • Each test identifies one state • s1 b/1 • s2 a/0 a/0 b/1 • s3 b/1 a/1 • preamble body b/1 s1 s2 b/0 b/0 a/0 a/0 a/1 s3

  18. Outline • Introduction • Formal protocol specifications • Finite-state machine (FSM) testing • Extended FSM (EFSM) testing • EFSM model • EFSM fault models • EFSM test selection • Test generation for StateCharts

  19. A set of states • An initial state • Transition relation b1 x1 {output y1} • Guards • Update blocks {output y3} b2 x2 b3 x3 {update v2} EFSM model • Finite sets of input (X) and output (Y) events • A finite set of variables (V)

  20. Tests for EFSM models • EFSM executions depend on input signals and data • A test consists of: • test sequence • test data • Executability problem: • Find data to execute the test sequence • Undecidable, in general 

  21. test validation: keep only executable tests • test application: execute each test • result evaluation: interpret outputs Test validation • Executability problem introduces an additional step in the testing process: • test generation: construct a test suite • test application: execute each test • result evaluation: interpret outputs

  22. Control and data flow testing • EFSM executions are data-dependent • Control flow FSM testing methods are not adequate for EFSM models • Data flow testing methods account for data dependencies

  23. Data flow: definitions and uses • Data variables are • defined by inputs and updates (def) • used in • updates (c-use) • guards (p-use) • Data-flow graph captures data dependencies

  24. Data flow graph • Directed graph with nodes labeled with definitions and uses of variables p-use v def u v=0{input u} u<0{v:=u+1} p-use u c-use u def v

  25. Data-flow coverage criteria • all-def • test suite covers each definition at least once • all-use • cover each def-use association at least once • all-du-paths • exercise all paths from each definition of a variable to every its use.

  26. all-use coverage • Find at least one definition-free path for every def-use association no definitons of v def v use v

  27. Outline • Introduction • Formal protocol specifications • Finite-state machine (FSM) testing • Extended FSM (EFSM) testing • Test generation for Statecharts

  28. EFSMs (FSMs + variables) + concurrency + hierarchy + communication + real-time Widely used for specifying real-time embedded HW/SW controllers Also used in most of object-oriented methodologies, e.g., UML Statecharts

  29. off EFSMs + power-on/ light-on; m:=0 power-off/ light-off Concurrency + on Hierarchy + coffee Communication + coffee[m>0]/start;dec coffee[m>0]/start idle busy Real-time tm(coffee,[3,5])/stop done/stop money inc/m:=m+1 inc/m:=0 not empty empty dec[m=1]/m:=0 dec[m>1]/m:=m-1 A coffee vending machine Statecharts =

  30. Test sequences for Statecharts • The main purpose • What should implementations do? • {power-on}/{light-on}, {inc}/{}, {coffee}/{start} • What should not implementations do? • {power-on}/{light-on}, {coffee}/{} • The main issue • How can we generate a finite and reasonable number of test sequences?

  31. off power-on/ light-on; m:=0 power-off/ light-off on coffee coffee[m>0]/start;dec coffee[m>0]/start idle busy done/stop tm(coffee,[3,5])/stop money inc/m:=m+1 inc/m:=0 not empty empty dec[m=1]/m:=0 dec[m>1]/m:=m-1 State coverage

  32. off power-on/ light-on; m:=0 power-off/ light-off on coffee coffee[m>0]/start;dec coffee[m>0]/start idle busy done/stop tm(coffee,[3,5])/stop money inc/m:=m+1 inc/m:=0 not empty empty dec[m=1]/m:=0 dec[m>1]/m:=m-1 Transition coverage

  33. off power-on/ light-on; m:=0 power-off/ light-off on coffee coffee[m>0]/start;dec coffee[m>0]/start idle busy done/stop tm(coffee,[3,5])/stop money inc/m:=m+1 inc/m:=0 not empty empty dec[m=1]/m:=0 dec[m>1]/m:=m-1 Data flow oriented coverage

  34. Test generation from Statecharts (I) • Basic idea • Transforms Statecharts into EFSMs • Advantage • Can reuse the existing methods and tools for EFSMs • Can handle infinite state space • Disadvantage • Cannot determine the executability of test sequences

  35. Test generation from Statecharts (II) • Basic Idea • Test generation can be reduced into model checking • TG: for each state s, is there a path leading to s • MC: a temporal logic formula EFs express this property • Advantage • Fully automatic • Generates only executable test sequences • Disadvantages • Can handle only finite state space

  36. Another perspective • Testing classes in object-oriented programs • Can we show an CLASS implementation conforms to a Statechart specification? • Variables of Statecharts: data members of classes • Events of Statecharts: member functions of classes • Testing inheritance in object-oriented programs • Can we reuse the test sequences of a super-class for testing its subclasses ?

  37. Vending Machine off power-on/ light-on; m:=0 power-off/ light-off on coffee coffee[m>0]/start;dec Coffee Vending Machine Cocoa Vending Machine idle busy tm(coffee,[3,5])/stop cocoa cocoa[m>0]/start;dec idle busy tm(coffee,[3,5])/stop Coffee- Cocoa Vending Machine money inc/m:=m+1 inc/m:=0 not empty empty dec[m=1]/m:=0 dec[m>1]/m:=m-1 A coffee-cocoa vending machine

More Related