Model-Based Testing Using Spec Explorer Aditya Mathur Purdue University CS 49000 Software Testing Spring 2011

1. Model-Based Testing UsingSpec Explorer Aditya MathurPurdue UniversityCS 49000 Software TestingSpring 2011 Material extracted mostly from: “Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer”, MargusVeanes, Colin Campbell, Wolfgang Grieskamp, Wolfram Schulte, Nikolai Tillmann, and Lev Nachmanson, Publishedby: Springer Verlag, Lecture Notes in Computer Science, Volume 4949, Pages 39-76, 2007.

2. Objective • The purpose of this presentation is to introduce modeling using the Spec Explorer tool from Microsoft. • The example presented here is from a paper cited in the title slide. • Familiarity with Chapter 3 of the textbook is assumed. Spec Explorer

3. Model Based Conformance Testing Requirements Implementation Generate (manual) Model Generate (automated) Tests Test harness (send inputs, receive outputs, and compare) Test outcome Spec Explorer

4. Example: Chat Room Client Client Client Client Client Chat Room Spec Explorer

5. Chat Room: Operation • Each client may post text messages. • Each message is delivered to all clients logged into the chat room. • Pending messages from a client are delivered in the order sent. • Messages from multiple senders are interleaved arbitrarily. Spec Explorer

6. Client status // Client entered the chat room or not boolentered; // Queue of messages sent by other clients but not received by this client Map<Client,Seq<string>> unreceivedMsgs; Spec Explorer

7. Client Actions: Creator • Create a new instance of a client. • State changes so that Empty message queues between the new client and the previously created clients. Spec Explorer

8. Client Actions: Creator Denotes an action in the abstract state machine. // Create a client [Action] Client() { this.unreceivedMsgs= Map; foreach(Client c in enumof(Client), c != this){ c.unreceivedMsgs[this] = Seq{}; // Empty sequence this.unreceivedMsgs[c] = Seq{}; }entered = false; } enumof(T): set of instances of type T in the current state. Spec Explorer

9. Client Actions: Enter • Changes the state of a client to indicate that it has entered the chat room. Spec Explorer

10. Client Actions: Enter // Model client entry into the chat room [Action] voidEnter() requires !entered; { entered = true; } Method pre-condition Spec Explorer

11. Client Actions: Send • Appends a new message to the queue of unreceived messages in all clients in the chat room. Spec Explorer

12. Client Actions: Send message // Send a message [Action] voidSend(stringmessage) requires entered; { foreach(Client c in enumof(Client), c != this, c.entered) c.unreceivedMsgs[this] += Seq{message}; } Spec Explorer

13. Client Actions: Receive • Extracts a message sent from a given sender from the sender’s queue in the client. Spec Explorer

14. Client Actions: Receive [Action(Kind=ActionAttributeKind.Observable)] void Receive(Client sender, string message) requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail; } Spec Explorer

15. Client Model Program class Client { boolentered; Map<Client,Seq<string>> unreceivedMsgs; [Action] Client() [Action] voidEnter() [Action] voidSend(stringmessage) [Action(Kind=ActionAttributeKind.Observable)] } Spec Explorer

16. Action types Controllable: Input by the user client, send, enter Observable: Output from the system receive Spec Explorer

17. Client Model Scenario Start Passive state: Active state: Timeout Spec Explorer

18. Model Programs in Spec Explorer Finite set of actions or update rules (Acts); e.g. client, send , enter, receive Vocabulary S: function symbols State variables: V in S; (e.g., entered, unreceivedMsgs) State: values, or interpretations, of state vocabulary symbols Execution of an action method in a given state leads to the next state where some state variables may have changed. Each action is associated with a pre- and a post-condition. Spec Explorer

19. FSM and Model Automaton FSM=(X, Y, Q, q0, δ, O), where X is a set of input symbols, Y a set of output symbols, q0 in Q, δ is transition function and O the output function Spec Explorer uses the notion of Model Automata: Model automaton=(Q, Q0, Qf, δ, A ), where Q is a set of states, Q0 is a set of initial states, Qf is a set of final states, and A is a set of actions, A=Ctrl U Obs, Ctrl is a set of control actions and Obs is a set of observable actions, Ctrl Obs = empty U Spec Explorer

20. Model program and model automaton A model automaton is a complete unwinding of the model program. Exploration: Unlike an FSM with a given sets of nodes and arcs, the states and transitions of a model program must be deduced by executing sequences of atomic actions starting in the initial state. unwind Model program Model automaton Spec Explorer

21. Exploration for the Chat Example In state s0, the precondition is true and hence Client constructor is invoked. The dynamic universe Client is updated by the addition of client c0. This is denoted by enumof(Client). The new state is denoted as s1. The transition explored is δ(s0, Client/c0)=s1 Spec Explorer

22. Accepting state A state is considered an accepting state if the accepting condition is true in that state. A test is allowed to terminate in an accepting state. Needed particularly in testing distributed and multithreaded programs. Why? An action the execution of which takes the implementation to a state where no actions are enabled is known as a succeed action. It forces the system into an accepting state. Spec Explorer

23. Accepting condition example enumof(Client).Size > 0 && // Exclude the initial state, and Forall{ c in enumof(Client), s in c.unreceivedMsgs.Keys; c.unreceivedMsgs[s].Length == 0 // states where pending messages have not been received. } A state that satisfies the above condition has no observable actions enabled. Spec Explorer

24. Scenario Control A model program may correspond to a large, or infinite state, automaton. Techniques are available to control the size of a model automata for a specific test purpose. Techniques: Parameter selection, method restriction, state filtering, directed search, state grouping Spec Explorer

25. Scenario Control: Parameter selection Select (s, m, v) for each state s, action m, and v sets of tuples. Restrictions by triples may lead to reduction in the number of transitions and hence a smaller automaton. • In Chat example: • send has an implicit parameter this and explicit parameter message. • These can be restricted using the pair: Set {(c in enumof(Client)); <c, “hi”>} Spec Explorer

26. Scenario Control: Method restriction An action m is enabled in state s if all pre-conditions associated with m are satisfied. Strengthening the pre-conditions can be used to limit the scenarios. In Chat example: restriction: Clients send messages only after all configured clients are created and have entered the system enum Mode { Creating, Entering, Sending }; Mode CurrentMode{ get { if (enumof(Client).Size < 2) return Mode.Creating; if(Set{cinenumof(Client),!c.entered;c}.Size<2) return Mode.Entering; return Mode.Sending } } Spec Explorer

27. Scenario Control: Method restriction In Chat example: restriction: Clients send messages only after all configured clients are created and have entered the system enum Mode { Creating, Entering, Sending }; Mode CurrentMode{ get { if (enumof(Client).Size < 2) return Mode.Creating; if(Set{c in enumof(Client),!c.entered;c}.Size<2) return Mode.Entering; return Mode.Sending } } Enabling of actions can now be restricted using expressions such as currentMode==Mode.Creating; Spec Explorer

28. Scenario Control: State filtering A state filter is a set Sf, where Sinit is in Sf. [The subscript f stands for filter, and not for final.] A transition from state s to state t is included in the automaton if t is in Sf. Sfis specified using a state based expression. • In Chat example: Using state filter avoid states in which the same message is posted more than once before it is received. • Forall{c in enumof(Client), s in c.unreceivedMsgs.Keys, • m1 in c.unreceivedMsgs[s], • m2 in c.unreceivedMsgs[s]; • m1 != m2} Spec Explorer

29. Test Generation Offline: Generate tests in advance from the model. Online: Generate tests on the fly as testing progresses. Spec Explorer

30. Test Suite Test suite T: Is another automaton generated from an automaton M. • States ST in T may use new state variables (test variables). • Test variables make it possible to record test history; e.g., which states have been traversed so far. • It contains two new methods called test actions: Observe and Timeout. • Transitions corresponding to test actions are called test transitions. • The Observe action encodes a decision to wait for an observable action. • The Timeout action indicates that no other observable action happened. An accepting state is reachable from every state in ST . Spec Explorer

31. Test Suite: Example Consider the following model program P: enumMode = {A,B,C} Mode mode = A; void F() requires mode == A {mode = B;} void G() requires mode == B {mode = C;} void H() requires mode == B {mode = C;} void I() requires mode == C {mode = A;} // Added to P to create P’. Accepting state: Where mode is C. Exploration: M generated from P and M’ from P’. Spec Explorer

32. Test Suite: Generate Test automaton Add a test variable n to indicate test number. T: (F, G) and (F, H) Spec Explorer

33. Automaton Traversal algorithms • Algorithm(s) used in Spec Explorer: • T covers all states in M • T covers all transitions in M • Each action is associated with a weight and cost using a state-based expression. Tests are generated to optimize the expected cost of a test. Spec Explorer

34. Summary • Spec Explorer : • Allows the creation of a model program P that captures the expected behavior(s) of the implementation under test (IUT). • Generates one or more model automaton (M) from P using exploration subjected to scenario restrictions. • Generates a test suite T from M either offline or online. Spec Explorer