1 / 37

Modeling for Testing

Modeling for Testing. Margus Veanes Microsoft Research Redmond. What is the correct model of the application to be tested?. Testers are faced with this question on a daily basis Let us consider creating a behavioral model of a “Notepad application”. Notepad scenario.

dusan
Download Presentation

Modeling for Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Modeling for Testing Margus Veanes Microsoft Research Redmond Dagstuhl Seminar: Methods for Modelling Software Systems

  2. What is the correct model of the application to be tested? • Testers are faced with this question on a daily basis • Let us consider creating a behavioral model of a “Notepad application” Dagstuhl Seminar: Methods for Modelling Software Systems

  3. Notepad scenario • Start from a blank notepad application • Consider the following sequence of actions: • Type “hello” • Change the font size to 72 • Select “Undo” from the Edit menu • What is the expected behavior? A) 2 is undone (i.e. the text “hello” remains) B) 1 and 2 are undone (i.e. the page goes blank) C) None of the above! Dagstuhl Seminar: Methods for Modelling Software Systems

  4. Why are models needed for testing? • Unit testing is insufficient • Single unit may work properly in isolation • Incorrect interaction between units may cause serious security/reliability failures • System-level testing • Requires model of system behavior • Behavior is often reactive/nondeterministic • Implementation is multi-threaded or distributed • Thread scheduling hard to control • State space is typically infinite • Traditional FSM-based testing does not scale Dagstuhl Seminar: Methods for Modelling Software Systems

  5. Our approach • Behavior is described by model programs • Spec Explorer tool (developed at MSR/FSE) is used for • Model program exploration (finitization) • Model validation • Model-based testing • Test case generation • Online testing • … Dagstuhl Seminar: Methods for Modelling Software Systems

  6. The Big picture Using models for testing Model Provides expected results for Generates PassFail User Info Test Cases Test Oracle Provides actual results for Are run by Implementation Dagstuhl Seminar: Methods for Modelling Software Systems

  7. Models • Behavior is described by model programs • For example in Spec#, AsmL, C# or VB • Internal (intra-step) semantics is abstract state machines (ASMs) with rich background • One step of a model program is an ASM step that may change some dynamic functions (state variables) • External (inter-step) semantics is interface automata (IA) orlabeled transition systems • Execution of a model program produces an action trace (of input and output actions) • Model refinementis refinement of IA Dagstuhl Seminar: Methods for Modelling Software Systems

  8. Model program (formal def.) • A model program has the following components • A state space States (of f.o. structures with a common universe V of values and signature ) • An initial state s0 in States • A finite vocabulary   of action symbols partitioned into two disjoint subsets • c of controllable action symbols • o of observable action symbols • A reset action symbol Reset  c • A family (f [x1,…,xn], pf[x1,…,xn])f   of guarded update rules • f is a guard (a -formula), pf is a parameterized update rulei.e. pf denotes a function pf:States Vn Statesn isthe arity of f • Reset has arity 0 and pReset(s) = s0 for all states s s.t. s⊧ Reset Dagstuhl Seminar: Methods for Modelling Software Systems

  9. Actions • An action has the form f(v1,…,vn) where f is an n-ary action symbol and viV. • All action symbols have term interpretation • i.e. two actions a and b are equal iff they have identical action symbols and equal parameters Dagstuhl Seminar: Methods for Modelling Software Systems

  10. Empty coffee machine example (in AsmL) State: type Coin = Integer var cancelled as Boolean = false var coins as Bag of Coin = Bag{} Controllable actions: Insert(coin), Cancel Insert(coin as Coin) requirenot cancelled coins := coins + Bag{coin} Cancel() requirenot cancelled require coins.Count > 0 cancelled := true Dagstuhl Seminar: Methods for Modelling Software Systems

  11. Empty coffee machine example (cont.) Observable actions: Return(coin) Return(c as Coin) require cancelled require c in coins coins := coins – Bag{c} cancelled := false Dagstuhl Seminar: Methods for Modelling Software Systems

  12. Interface automaton (formal def.) • A set S of states • A nonempty subset S0 S called initial states • Disjoint sets of controllable actions Ac and observable actions Ao. • Enabling functions • Ec: S  (Ac) • Eo : S  (Ao) (let E = Ec Eo, A = Ac Ao) • Partial transition function  : S  A  S, s.t. (s,a) is defined for all a  E(s) Dagstuhl Seminar: Methods for Modelling Software Systems

  13. Model program M as an interface automaton M • SM = States, S0M = {s0} • E⎕M(s) = {f(v): f  ⎕,v  V, s ⊧ f(v)} (for ⎕ = c , ⎕ = o) • A⎕M = s E⎕M(s) • M(s,f(v)) = pf(s,v) ( for f(v)  EM(s) ) Dagstuhl Seminar: Methods for Modelling Software Systems

  14. Refinement • Alternating simulation  from MtoIis a relation  : SM  SIs.t. for all (s,t)  , • EcM(s)  EcI(t) and EoI(t)  EoM(s) • (M(s,a), I(t,a))  , for all a  EcM(s) EoI(t) • I refinesM if there exists an alternating simulation  from M to I, s.t. S0M S0I   Dagstuhl Seminar: Methods for Modelling Software Systems

  15. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  16. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  17. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  18. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  19. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  20. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  21. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  22. Example of refinement M: I: Return(10) 0 {} Insert(10) Return(5) Insert(5) Insert(5) 10 Return(5) {5} 5 Cancel Return(10) Insert(10) Insert(10) Cancel {5} 15 {5,10} Insert(5) Return(10) Cancel 5 Return(10) 15 10 Cancel Return(10) {5,10} {10} Return(5) I refines M Cancel {10} Dagstuhl Seminar: Methods for Modelling Software Systems

  23. Conformance Testing • Given a model Mand implementation I (as IA) • I conforms to M iff I refines M • Testing can be viewed as a game between M and I • M making controllable steps • I making observable steps • (I is assumed to behave as “nature”) • The “rules of the game” are dictated by M • Test cases are generated from M. • A test case tells what move M should make in a state where its actions are enabled • A test purpose may be • achieve transition coverage • reach a certain goal state • In online testing test cases are generated on-the-fly Dagstuhl Seminar: Methods for Modelling Software Systems

  24. Spec Explorer demo Dagstuhl Seminar: Methods for Modelling Software Systems

  25. Current Users (in MS) •  100 Testers and test architects in MS product groups • Avalon • Windows (various groups) • MSN • Excel Test • VS Team Division • Distributed Transaction Coordinator (Windows Vista Communication Foundation) • Windows Driver Framework Dagstuhl Seminar: Methods for Modelling Software Systems

  26. Users outside MS • Modeling courses in several universities in Europe • Semantic Anchoring of Domain Specific Modeling Languages in the MIC tool suite • Headed by Janos Sztipanovits Inst. for Software Integrated Systems, Vanderbilt Univ. • Research pgm funded by GM, Boeing and Raytheon • Quote from Janos: We have evaluated several frameworks, such as TLA+, the Tagged Signal Models, Z and others, and finally selected Abstract State Machines (ASM) because of two reasons: (a) ASM-s provide excellent foundation for specifying operational semantics, and (b) availability of Microsoft’s AsmL tool suite. Dagstuhl Seminar: Methods for Modelling Software Systems

  27. Experiences • Style of modeling depends on application • FSM style (deterministic Mealy machines) • LTS style (often nondeterministic using input (controllable) and output (observable) actions) • Integration with existing (informal) specification practices is important • Most tool features driven by demands of internal users at MS (mostly testers) • Models help discover more bugs during modeling (design bugs) than testing • Testers do not get credit for finding those bugs !!! • During testing models help discover deep system-level bugs where manual test scenarios don’t • Such bugs are hard to understand and fix Dagstuhl Seminar: Methods for Modelling Software Systems

  28. Experiences (cont.) • Bugs appear in both models and implementations (the ratio is roughly 50-50) • Code coverage is a poor measure for testing concurrent software, often a single execution thread provides the same coverage. • New versions of implementation usually require only local changes to models, whereas manual tests often need to be rewritten completely • The tool is built on .NET but has been used to test distributed C++ applications. Most test harnesses at MS are built in managed code. Dagstuhl Seminar: Methods for Modelling Software Systems

  29. User wish list (within MS) • Language independence for modeling • Use C#, VB, … • Better control over exploration • Scenario-oriented models • Model composition • Model features separately and compose the models • Model validation • Validate properties on models Dagstuhl Seminar: Methods for Modelling Software Systems

  30. Vision addresses the wish list of Spec Explorer users • Behavior is described by model programs • Written in C# for modeling (or another .NET language of users choice) • Preferably with graphical domain-specific front-end notations • Tools for model composition • For modeling one aspect at a time • Bigger models are compositions of smaller ones • For scenario control • Smaller models are compositions of bigger ones • Tools for model validation • Showing that desired conditions hold in the model • Compositional reasoning is used for composed models • Tools for model refinement • Showing that one model is an refinement of another model Dagstuhl Seminar: Methods for Modelling Software Systems

  31. Model program composition • Parallel composition P||Q • P and Q synchronize on shared actions • State spaces of P and Q are disjoint • Traces of P||Q are the intersection of P traces and Q traces • Other forms of compositions • serial, interleaving, … Dagstuhl Seminar: Methods for Modelling Software Systems

  32. Shared action composition • Action pgm for f(x1…xn)in M1 :  x1…xn(1, p1) • Action pgm for f(x1…xn)in M2 :  x1…xn(2, p2) • Action pgm for f(x1…xn)inM1 || M2 :  x1…xn(1 2, p1||p2) Dagstuhl Seminar: Methods for Modelling Software Systems

  33. Example: P p0 SelectFontStart(“Times”) p1 SetFontSize(10) p2 SetFontName(“SansSerif”) p3 OK Cancel p4 SelectFontFinish(_) p5 Dagstuhl Seminar: Methods for Modelling Software Systems

  34. Example: Q q0 SelectFontFinish(y) y == SavedFont SelectFontFinish(y) y == DialogFont SelectFontStart(x) DialogFont = x SavedFont = x q2 q3 OK Cancel q1 SetFontSize(z) DialogFont.Size = z SetFontName(y) DialogFont.Name = y Dagstuhl Seminar: Methods for Modelling Software Systems

  35. Example: P||Q p0,q0 SelectFontStart(Times(12)) DialogFont = Times(12); SavedFont = Times(12) p1,q1 SetFontSize(10) DialogFont.Size = 10 p2,q1 SetFontName(“SanSerif”) DialogFont.Name = “SanSerif” p3,q1 OK Cancel p4,q2 p4,q3 SelectFontFinish(SanSerif(10)) SelectFontFinish(Times(12)) p5,q0 Dagstuhl Seminar: Methods for Modelling Software Systems

  36. Related work (regarding foundations) • Abstract state machines • Interface automata • Input-Output COnformance (IOCO) theory • Markov Decision Process (MDP) theory Dagstuhl Seminar: Methods for Modelling Software Systems

  37. Thank you! • FSE web site: • http://research.microsoft.com/fse • Downloading Spec Explorer: • http://research.microsoft.com/specexplorer Questions? Dagstuhl Seminar: Methods for Modelling Software Systems

More Related