Model Program Based Black-Box Testing

1. Model Program Based Black-Box Testing Margus Veanes Foundations of Software Engineering Microsoft Research, Redmond Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

2. System-level black-box testing • Requires model of system behavior • Behavior is often reactive/nondeterministic • Implementation is multi-threaded or distributed • Thread scheduling hard to control • State space is typically infinite • Objects, unbounded value types • Traditional FSM-based testing does often not scale Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

3. Model based testing Model Provides expected results for Generates PassFail User Info Test Cases Test Oracle Provides actual results for Are run by Implementation Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

4. Our approach • Behavior is described by model programs • Written in Spec# or AsmL • Describe reactive system behavior as model automata • Use alternating simulation for conformance • Model exploration, validation and model-based testing are provided by the Spec Explorer tool developed at MSR • Supports scenario control • Provides offline as well as online testing • Views test cases as game strategies • Checks conformance violations Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

5. The big picture Modeling Model program Scenario control Online testing Constrained model Test generation Test cases Test execution Coding Implementation Pass/Fail Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

6. IUT Conformance checkingcomponent view Controllable actions Controllable actions Constrained model Refinement checker IUT wrapper Object mapping Observableactions Observable actions Model c1 d1 … … Pass/Fail Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

7. Constrained chat model IUT wrapper Controllable actions: … Login Send IUT Refinement checker Object mapping b.Send(“hi”) Chat server b Bob m Mary m.Send(“hello”) b.Send(“bye”) b.Rcv(“hello”) m.Rcv(“hi”) Observable actions: Receive b.Rcv(“hello”) m.Rcv(“hi”) m.Rcv(“bye”) m.Rcv(“bye”) b.Rcv(“hello”) Example: Chat Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

8. b.Send(“hi”) b.Send(“hi”) m.Send(“hello”) m.Send(“hello”) m.Receive(b,“hi”) b.Receive(m,“hello”) m.Receive(b,“hi”) b.Receive(m,“hello”) b.Receive(m,“hello”) Refinement relation • Conformance between the model and the system under test is alternating refinement Spec System Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

9. B.Send(“bye”) B.Send(“hi”) M.Send(“hello”) M.Receive(B,“hi”) M.Receive(B,“bye”) B.Receive(M,“hello”) Example: Valid chat scenario Service should guarantee local consistency of message delivery (FIFO wrt sender) Send is a controllable action Receive is an observable action Bob Server Mary Time Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

10. Model programs and automata • States are mappings of variables to values • first-order structures • Initial state is given by initial assignment to variables • Actions are defined by method invocations • Actions are either controllable or observable • An action m(v) is enabled is a state Sif the precondition of m(v) is true in S • An action m(v)transitions from state S1 to stateS2 if m(v) is enabled in S1and its execution in S1 yields S2 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

11. Model program : Chat service State: class Client { bool entered = false; Map<Client, Seq<string>> unreceivedMsgs = Map{}; } Actions: class Client { void Send(string message) requires entered; { foreach (Client c inenumof(Client), c != this, c.entered) c.unreceivedMsgs[this] += Seq{message};} void Receive(Client sender, string message) requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;} … } Controllable: Observable: Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

12. Online testing • For large reactive systems deriving an exhaustive test-suite is infeasible • Merge test generation and test execution into one process • Online testing is a form of model-based stress testing • Purely random exploration is wasteful • Use scenario control to direct online testing Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

13. Exploration & Scenario control • Exploration is the process of unfolding a rule-based model program into a transition system • Actions move the system from state to state • State comes from variables containing values • Compound data types are possible (sets, maps, sequences, etc…) • Objects are identities • Scenario control • Imposes limits on exploration • Using filters, state groupings, scenario actions , … Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

14. Example Demo: exploration of a coffee machine Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

15. Scenario control • In general the transition system is infinite, but we can impose limits • Goal • Create a state space of manageable size that satisfies a given testing goal • Two main tasks • Restrict action parameter domains to interesting values • Restrict the state space to interesting states • Note: The two tasks are not necessarily independent! Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

16. Scenario control (cont.) • The following techniques are used in Spec Explorer • State filters • Stopping conditions • State groupings • Search order • Enabling conditions on actions • Scenario actions • Usually a combination of all (or some) of the techniques is needed to achieve desired effect Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

17. Example of scenario actions: Chat initialization • Use a parameterized scenario action Start. • Invoking Start(n) produces a sequence of nCreate() invocations followed by nEnter() invocations [Action(Kind=Scenario)] void Start(int nrOfMembers) requires enumof(Client) == Set{}; { Seq<Client> clients = Seq{}; // 1-- The given number of // clients are created for (int i = 0; i < nrOfMembers; i++) clients += Seq{Create()}; // 2-- all clients enter the session for (int i = 0; i < nrOfMembers; i++) clients[i].Enter(); } Create()/c0; Create()/c1; c0.Enter(); c1.Enter(); Start(2) Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

18. Test generation • The result of exploration is a finite transition system with observable and controllable actions : a model automaton A. • Testing can be viewed as a game between two players • Testing tool (TT) • Making controllable transitions • Implementation under test (IUT) • Making observable transitions • The “rules of the game” are dictated by A • Tests are generated from A. Test cases are strategies • A strategy tells what move TT should make in a state where its actions are enabled • The purpose of a strategy may be to achieve transition coverage or to reach a certain goal state Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

19. Demo: chat Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

20. Experiences • Most tool features driven by demands of internal users at MS (mostly testers) • Models help discover more bugs during modeling (design bugs) than testing • Testers do not get credit for finding those bugs !!! • During testing models help discover deep system-level bugs where manual test scenarios don’t • Such bugs are hard to understand and fix Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

21. Experiences (cont.) • Bugs appear in both models and implementations (the ratio is roughly 50-50) • Code coverage is a poor measure for testing concurrent software, often a single execution thread provides the same coverage. • New versions of implementation usually require only local changes to models, whereas manual tests often need to be rewritten completely • The tool is built on .NET but has been used to test distributed C++ applications. Most test harnesses at MS are built in managed code. Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

22. Outlook • User wish list: • Language independence for modeling • Use C#, VB, … • Better control over exploration • Scenario-oriented models • Model composition • Model features separately and compose the models • Model validation • Validate properties on models Drives the focus of current research at FSE Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

23. Related foundational work • Abstract state machines • Interface automata • Input-Output COnformance (IOCO) theory • Markov Decision Process (MDP) theory Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands

24. Thank you! • FSE web site: • http://research.microsoft.com/foundations • Downloading Spec Explorer: • http://research.microsoft.com/specexplorer Questions? Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands