Some Testing Techniques (enhanced 6 + 4)

1. Some Testing Techniques(enhanced 6 + 4) Course Software Testing & Verification 2013/14 WishnuPrasetya

2. Plan • Unit testing  tool • Mocking (Sec. 6.2.1)  enhanced • Black box testing  Input Partitioning (Ch 4). • Regression (6.1)  enhanced

3. Unit Testing • Making sure that the units are correct. • Insufficient unit testing can be costly: debugging an error at the system-test level is muchmore costly than at the unit level (some reports suggest more than 10x). • Don’t confuse the concept of “unit testing” and “unit testing tool”; the later can often also be used to facilitate integration and system testing.

4. But what is a “unit” ? • In principle, you decide. “Units” are just something you can compose to build something bigger. Possibilities: function, method, or class. • But be aware that different types of units may have types of interactions and complexity, thus requiring different testing approach • a function’s behavior depends only on its parameters; does not do any side effect. • procedure depends-on/affects params and global variables • method: static vars, params, instance vars • class: is a collection of interacting methods

5. Unit testing in C# • You need at least Visual Studio Professional; and for code coverage feedback you need at least Premium. • Check these tutorials/docs (Visual Studio 2010): • Walkthrough: Creating and Running Unit Tests • Walkthrough: Run Tests and View Code Coverage • Walkthrough: Using the Command-line Test Utility • API Reference for Testing Tools for Visual Studio, in particular Microsoft.VisualStudio.TestTools.UnitTesting, containg classes like • Assert, CollectionAssert, ... • In this lecture we will just go through the concepts

6. The structure of a “test project” class Thermometer private double val private double scale private double offset public Thermometer(double s, double o) public double value() public double warmUp(double v) public double coolDown(double v) A test project is a just a project in your solution that contains your test-classes.

7. The structure of a “test project” • A solution may contain multiple projects; it may thus contain multiple test projects. • A test project is used to group related test classes. • You decide what “related” means; e.g. you may want to put all test-cases for the class Thermometer in its own test project. • A test class is used to group related test method. • A test method does the actual testing work. It may encode a single test-case, or multiple test-cases. You decide.

8. Test Class and Test Method [TestClass()] public class ThermometerTest { private TestContexttestContextInstance; //[ClassInitialize()] //public static void MyClassInitialize(...) ... //[ClassCleanup()] //public static void MyClassCleanup() ... //[TestInitialize()] //public void MyTestInitialize() ... //[TestCleanup()] //public void MyTestCleanup() ... [TestMethod()] public void valueTest1() ... [TestMethod()] public void valueTest2() .... } public void valueTest1() { target = new Thermometer(1,0); double expected = - 273.15 ; double actual = target.value(); Assert.AreEqual(expected, actual); } Be careful when comparing floating numbers, you may have to take imprecision into account, e.g. use this instead: AreEqual(expected,actual,delta,”...”)

9. Positive and Negative Test • Positive test: test the program on its normal parameters’ range. • But can we afford to assume that the program is always called in its normal range? Else do Negative test: test that the program beyond its normal range. • E.g. when unit testing a method, to test if it throws the right kind of exceptions.

10. Test Oraclesome patterns  will return later • Full: Assert.IsTrue( T == -273.15) • Partial: Assert.IsTrue( T >= -273.15) • Property-based : Assert.isTrue(safe(T)) public void valueTest1() { target = new Thermometer(1,273.15); double expected = - 273.15 ; double actual = target.value(); Assert.AreEqual(expected, actual); } An oracle specifies your expectation on the program’s responses. More costly to maintain, e.g. if you change the intended behavior of the program. Usually partial, but allow re-use the predicate “safe” in other test-cases.

11. Discussion: propose test casesin particular the oracles... incomeTax(i) { if (i18218) return 0.023 * i t = 419 if (i32738) return t + 0.138 * (i – 18218) t += 1568 if (i54367) return t + 0.42 * (i – 32738) } reverse(a) { N = a.length if (N  1) return for (inti=0; i< N/2 ; i++) swap(a,i, N-1-i) } Property-based testing fits nicely for reverse, but not for incomeTax; for the latter we’ll have to fall back to conrete-value oracles, which unfortunately tend to be more costly to maintain.

12. Discussion: the Oracle Problem (6.5) • Every test-case needs an oracle; how to construct it!?  always a big problem! • Using concrete values as oracles is often powerful, but potentially expensive to maintain. • Using “properties” on the other hand has the problem that often it is hard to write a complete yet simple property capturing correctness. E.g. how to express an oracle for a sorting function? • Alternatively we can fall back to redundancy-based testing.

13. Inspecting Test Result

14. Inspecting Coverage

15. Finding the source of an error: use a debugger! • Add break points; execution is stopped at every BP. • You can proceed to the next BP, or execute one step at a time: step-into, step-over, step-out. • VisualStudio uses IntelliTrace logging  you can even inspect previous BPs.

16. The Debug class • Debug.Print(“therm. created”) • Debug.Assert(t.scale() > 0) to check for properties that should hold in your program. • Will be removed if you build with debug off (for release). • Check the doc of System.Diagnostics.Debug

17. (Unit) Testing a class • Many classes have methods that interact with each other (e.g. as in Stack). How to test these interactions? • How to specify and test these interactions? Options: • class invariant • Abstract Data Type (ADT) • Finite State Machine (FSM)

18. Specifying with class invariant • Regardless the interaction with other methods, each method of a class C has to keep the state of its target object consistent. • Express this with a class invariant, e.g. • this.T >= -273.15 • this.saldo >= 0 • forall x in persons, typeOf(x) is Employee • Class invariant cannot express a constraint over the interaction itself.

19. Formulate and test the class invariant Stack<T> private Object[] content private int top public Stack() push(T x) T pop() boolclassinv() { return 0top && top<content.length } } Example test-cases, check class-inv after : call to the constructor constructor ; push(x) constructor ; push(x) ; push() constructor ; push(x) ; pop() some random sequence of push and pop

20. Specifying a class as an ADT • An Abstract Data Type (ADT) is a model of a (stateful) data structure. The data structure is modeled abstractly by only describing a set of operations (without exposing the actual state). • The semantic is described in terms of “logical properties” (also called the ADT’s axioms) over those operations.

21. Example : stack • Stack axioms : • For all x,s:s.push(x) ; y = s.pop ; assert (y==x ) • For all x and s :s.push(x) ;assert( s.isEmpty()) • For all x and s.isEmpty():s.push(x) ; s.pop assert (s.isEmpty()) Stack<T> boolisEmpty() push(T x) T pop() Depending of the offered operations, it may be hard/not possible to get a complete axiomatization.

22. Test each ADT’s axiom • For all x,s:s.push(x) ; y = s.pop ; assert (y==x ) For example, three test cases : empty s non-empty s s already contains x

23. Specifying a class with a finite state machine (FSM) (2.5.1, 2.5.2) File: open() write() close() FSM can also come from your UML models. • good when the sequence with which the methods of the class is called matters • For example, test that after every valid sequence the class invariant hold. • relevant concepts to apply: node coverage, edge coverage, edge-pair coverage.

24. In a larger project.... • You want to test your class Heater ; it uses Thermometer which is not ready yet! • We can opt to use a mock Thermometer. A mock of a program P: • has the same interface as P • only implement a very small subset of P’s behavior • fully under your control • Analogously we have the concept of mock object. • Make mocks yourself e.g. exploiting inheritance, or use a mocking tool.

25. Mocking with Moq class Heater double limit bool active public check() { if(thermometer.value() >= limit) active = false } thermometer interfaceIThermometer double value() double warmUp(double v) test1() { Heater heater = new Heater() var mock = new Mock<IThermometer>() mock.Setup(t => t.value()).Returns(-275.15) heater.thermometer = mock.object heater.limit = 303.0 heater.check() Assert.IsFalse(heater.active) }

26. Mocking with Moq(google it for more info!) var mock = new Mock<IThermometer>() mock.Setup(t => t.value()).Returns(303.00001) mock.Setup(t => t.warmUp(0)).Returns(0) mock.Setup(t => t.warmUp(It.IsInRange <double>(-10, 10, Range.Inclusive)) .Returns(0) mock.Setup(t => t.warmUp (It.IsAny<double>()))          .Returns((double s) => s + 273.15) Many more mock-functionalities in Moq; but in general mocking can be tedious. E.g. what to do when your Heater wants to call warmUp in an iteration?

27. Beyond unit testing, what if we don’t have access to the source code ? • Or you deliberately want to abstract away • Or, you do have access, but you don’t have a tool to instrument the code • (Def 1.26) White box testing : common at the unit-testing level • (Def 1.25) Black box testing: common at the system-testing level. Approaches : • Model-based testing • Partition-based testing

28. Model-based testing • We already seen that  use an FSM as a model of the system you test • Such an FSM specifies : • sequences which should be valid and invalid • these tell you which sequences to test • furthermore, coverage can be defined over the FSM TextEditor: new(fname) type(c) save()

29. Model-based testing size() = 0 • Predicates can be defined on the FSM’s states to specify properties that should hold; but keep in mind that in black box testing you cannot just inspect objects state at will. • Useful concepts : mutators and inspectors. • Mutators : methods with side effect  arrows in the FSM. • Inspectors : methods with no side effect  used to express your state predicates, TextEditor: new(...) type(c) type(c) size() > 0 save()

30. Partitioning the inputs • Based on “your best understanding” of save’s semantic. • Terminology: characteristic, block. The domain of a characteristic is divided into disjoint blocks; the union of these blocks must cover the entire domain of the characteristic. • Assumption : values of the same block are “equivalent” save(String fname, Object o) o : (P) null (Q) non-null serializable (R) non-null non-serializable fname : (A) existing file (B) non-existing file

31. So, what input values to choose? • (C4.23, ALL) All combinations must be tested. |T| = (i: 0i<k: Bi) ; does not scale up. • (C4.24, EACH CHOICE) Each block must be tested. |T| = (maxi: 0i<k: Bi) ; usually too weak. save(String fname, Object o) o : (P) null (Q) non-null serializable (R) non-null non-serializable fname : (A) existing file (B) non-existing file

32. t-wise coverage • (C4.25, pair-wise coverage). Each pair of blocks (from different characteristics) must be tested. • (C4.26, t-wise coverage). Generalization of pair-wise. • Obviously stronger than EACH CHOICE, and still scalable. • Problem: we just blindly combine; no semantical awareness. T : (A,P,X) , (A,Q,Y) , ... more? A B P Q X Y

33. Adding a bit of semantic • (C4.27, Base Choice Coverage, BCC) Decide a single base test t0. Make more tests by each time removing one block from t0, and forming combinations with all remaining blocks (of the same characteristics). |T| = 1 + (i : 0i<k : Bi - 1) Example: t0 = (A,P,X), generates these additional test requirements : (B,P,X) (A,Q,X) (A,P,Y) (A,P,Z) A B P Q X Y Z

34. Or, more bits of semantics • (C4.28, Multiple Base Choices). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remainingnon-base blocks.|T| at most M + (i : 0i<k : M*(Bi - mi)) Example, base tests = (A,P,X), (A,P,Y) A B P Q X Y Z (A,P,X) generates (B,P,X) (A,Q,X) (A,P,Z) (A,P,Y) generates (B,P,Y) (A,Q,Y) (A,P,Z)

35. Example-2, MBCC A BC P QR X Y Z • (C4.28, Multiple Base Choices). For each characteristic we decide at least one base block. Then decide a set of base tests; each only include base blocks. For each base test, generate more tests by each time removing one base block, and forming combinations with remainingnon-base blocks. Bold : base blocksChosen base tests = (A,P,X), (A,Q,Y) These produce these additional test requirements: (B,P,X)(C,P,X)(B,Q,Y)(C,Q,Y) (A,R,X) (A,R,Y) (A,P,Z) (A,Q,Z) • base-blocks are not cross-combined except as in the base tests. • non-base blocks are not cross-combined with each other.

36. Constraints, to exclude non-sensical cases • Example: • combo (A,P,Y) is not allowed. • if P is selected, then X must also be selected. • Solvable: pair-wise coverage + (A,P,Y) is not allowed. • Can be unsolvable, e.g. pair-wise coverage + (A,P) is not allowed. • General problem: given a coverage criterion C and a set of constraints, find a test set T satisfying both. • In general the problem is not trivial to solve.

37. Overview of partition-based coverage ALL Multiple Base Choice Coverage t-Wise Base Choice Coverage Pair-Wise EACH CHOICE

38. Regression Test • To test that a new modification in your program does not break old functionalities. To be efficient, people typically reuse existing test sets. • Usually applied for system-testing, where the problem is considered as more urgent. Challenge: very time consuming (hours/days!). • There are also research to apply it continuously at the unit level, as you edit your program; see it as extended type-checking. Result was positive! (Saff & Enrst, An experimental evaluation of continuous testing during development. ISSTA 2004)

39. Some concepts first... • Test Selection Problem: suppose P has been modified to P’. Let T be the test set used on P. Choose a subset T’T to test P’. • Obviously, exclude obsolete test cases: those that can’t execute or whose oracles no longer reflect P’ semantic. Let’s assume: we can identify them. • You want the selection to be safe : T’ includes all test-cases in T that will execute differently on P’. • Only attractive if the cost of calculating T’ + executing T’ is less than simply re-executing the whole T.

40. Idea: select those that pass through modified code • If m is the only method in P that changes, the obvious strategy is to select only test-cases that pass through m. • Better: only select test-cases that pass m’s “modified” branch. m(x) { if (d) y = x+1 else y=0 } (orginal) m(x) { if (d) y = x-1 else y=0 } (modified)

41. Corner case • The first if is modified by removing an else-branch. Using the previous strategy means that we have to select all test-cases that pass m. Yet we see that the paths [d, e, stmt] and [ d, e] present in both old and new m; so there is actually no need to select them. m(x) { if (d) { y = x+1 ; if (e) stmt ; u = 0 } elseif (e) stmt } m(x) { if (d) y = x+1 ; if (e) stmt}

42. Looking at it abstractly with CFG m(x) { if (d) { y = x+1 ; if (e) stmt ; u = 0 } elseif (e) stmt } m(x) { if (d) y = x+1 ; if (e) stmt} d y=x+1 e stmt d end y=x+1 e stmt e stmt Notice that [d, e, stmt, end] and [d, e, end] appear in both, and equivalent. u=0 end

43. Some concepts • We assume: P is deterministic.  each test-case always generate the same test path. • Let p and p’ be the test-paths of a test-case t when executed on P and P’; t is modification traversing if not(p p’).  let’s select modification traversing test-cases. • p p’ if they have the same length, and for each i, pi  p’i the latter means they contain the same sequence of instructions. • So far this is not helpful, because such a selection strategy requires us to first execute t on P’. Then it is not attractive anymore!

44. “Intersection” Graph a,a G’’ = G G’: a a • First, extend CFG so that branches are labelled by the corresponding decision value (e.g. T/F for if-branch). Label non-branch edge with some constant value. • Each node of G’’ is a pair (u,u’). Then G’’ is defined like this : • The pair of initial nodes (s0,s0’)  G’’. • If (u,u’)G’’, and uu’, and uv is an edge in G, and u’v’ and edge in G’ both with the same label, then (u,u’)  (v,v’) should be an edge in G’’. G : G’ : b,b b b c,c c c c,c c end,end end d end,d end

45. “Intersection” Graph a,a G’’ = G G’: a a • Each path p in G’’ describes how a path in G would be executed on G’ if the same decisions are taken along the way. Note that this is calculated without re-executing any test-case on P’. • Any path in G’’ ends either in a proper exit node (green), or in a pair (u,u’) where not uv (red). This would be the first time a test-case would hit a modified code when re-executed on P’. • The old test-cases are assumed to have been instrumented, so that we know which nodes/edges in G it traversed. G : G’ : b,b b b c,c c c c,c c end,end end d end,d end

46. Selection Algorithm a,a G’’ = G G’: a a • Select test-cases that pass [a,b,c,end] in G  expensive to check! • (Safe but not minimalistic) Select test-cases that pass a node u in G that is part of a red-node in G’’.  same problem as before, it will select also select [a,c,end] which is not modification traversal. • (Rothermel-Harold, 1997) Select test-cases that pass an edge e in G that in G’’ leads to a red-node in G’’.  actually the same problem. G : G’ : b,b b b c,c c c c,c c end,end end d end,d end

47. Selection Algorithm a,a G’’ = G G’: a a • (Ball algorithm,1998) Partition G’’ nodes to those can can reach a green-node (G partition), and those that cannot (NG partition). Look at edges in G’’ that cross these partitions (so, from G to NG). A test path p is modification traversing if and only if it passes through a crossing edge (as meant above).  use this as the selection criterion. G : G’ : b b b,b c,c c c c c,c end,end end d end,d G-partition end NG-partition