How do black- and white-box testing relate to one another in real-world testing projects?

Practical Software TestingTest (Code) Coverage Measurement andTest Enhancement using Test Coverage Information Vahid Garousihttp://maralsoft.garousi.net

How do black- and white-box testing relate to one another in real-world testing projects? Develop an initial Test suite using BB techniques Analyze the parts of the code uncovered by BB test suite Enhance the Test suite usingWB techniques Apply BB coverage criteria to enhance it Apply coverage code to enhance it

Outline • What is test coverage? • Line/Statement/Node Coverage • Block coverage • Decision coverage • Tools for the measurement of code coverage

What is test adequacy and test coverage? • Discussions between developers and testers: • “My JUnit test is covering 98% of my code” • “To improve your code coverage, you need to add JUnit test cases to cover the uncovered if-then-else’s!”

Measurement of test adequacy – A JUnit Example

Test Enhancement • A test set adequate with respect to some criterion does not guarantee an error-free program • But an inadequate test set is a cause for worry. • Identification of this deficiency helps in the enhancement of the inadequate test set. • Enhancement in turn is also likely to test the program in ways it has not been tested before Increase the coverage

Test Enhancement: Process (debug)

A Real Example of Test Enhancement in Eclipse and JUnit – Let’s Compare

Demo of Statement coverage - CoffeeMaker Example • src: let’s run the program first • You need to “Add a recipe” if you want to start making coffee • unittests: and then its unit tests and observe how statement coverage works

Outline • What is test coverage? • Line/Statement/Node Coverage • Block coverage • Decision coverage • Tools for the measurement of code coverage

Line/Statement/Node Coverage Hypothesis: Faults cannot be discovered if the parts containing them are not executed x  y x = y x<=y x > y

Example: Measurement of Statement Coverage • Coverage domain: {statements in lines 2…13}. • Let T1={t1:<x=-1, y=-1>,t 2:<x=1, y=1>} • In-class exercise: Find: • Number of statements covered by each test case • Number of unreachable statements • The statement coverage % for T

Tools for measuring Line Coverage • A lot of tools, depending on your programming language. • Java • CodeCover, Emma, Clover, Cobertura, jCoverage, … • C# • NCover, PartCover, TestMatrix • PHP • phpCoverage, Xdebug • …

Outline • What is test adequacy and test coverage? • Line/Statement/Node Coverage • Block coverage • Decision coverage • Tools for the measurement of code coverage

Example: block coverage Coverage domain: Be={1, 2, 3, 4, 5} • In-class exercise: Find: • Number of blocks covered by each test case • Number of unreachable blocks • The blocks coverage % for T

Tools for measuring Block coverageEclipse, JUnit and Emma • The CodeCover coverage tool does not provide Block coverage, but another tool called Emma does. • http://www.eclemma.org • Exercise for outside the class: Install Emma on your PC and try it on an example SUT

The need for Decision coverage:Incompleteness of Statement Coverage An example: begin int x,z; input(x); if (x<0) x=-x; z=foo1(x); output(z); end; Requirements • Read integer x. • If x is –ve, make it +ve • Call foo1 with x and store the return value in z. • Else, call foo2 when the input x>=0. • < The program we have written: • A negative x would result in the coverage of all statements (100% statementcoverage). • But still, the fault is not defected!

Decision coverage (also called Edge coverage) • A decision is considered covered if all its outcomes (outgoing paths) decision have been taken.

Decision coverage: Computation • The decision coverage of T is computed as follows. • Dc is the contribution (coverage) of each decision: • 2: it is fully covered (both T and F cases) • 1: partially covered (its T or F case), • or 0: not covered • Di is the number of infeasible decisions • De is the total number of decisions in the program, i.e., the size of the decision coverage domain. • T is adequate if the decision coverage of T =1.

begin int x,z; input(x); if (x<0) x=-x; z=foo1(x); output(z); end; T={t:<x=-2>} Calculate the decision coverage Which test case we should add to make decision coverage 100%? Decision coverage: Example

Tools for the measurement of code coverage • See the large list of tools @ http://java-source.net/open-source/code-coverage http://en.wikipedia.org/wiki/Code_coverage

Tools for the measurement of code coverage CodeCover

Tools for the measurement of code coverage Emma

Tools for the measurement of code coverage CoverLipse

Tools for the measurement of code coverage IBM Rational PurifyPlus.

Tools for the measurement of code coverage NCover - Code Coverage for .NET Developers

From NCover website

LCOV: a graphical front-end for coverage testing tool gcov, for PHP

A Nice Article http://www.stickyminds.com/s.asp?F=S14774_ART_2

The Linux Test Project • The Linux Test Project (usually referred to as LTP) is a body of regression test suites for the Linux OS kernel. • The project was started by SGI and is now maintained by IBM. • The Oct. 2009 version of the test suite contains 20,000+ test cases for the Linux OS. • I strongly recommend you to visit and “learn” from its website. • http://ltp.sourceforge.net

Looking at a few test cases in Linux Test Project

Outline • What is test adequacy and test coverage? • Line/Statement/Node Coverage • Block coverage • Decision coverage • Tools for the measurement of code coverage • Other code coverage criteria (for future study)

Other code coverage criteria (for future study) We covered From http://en.wikipedia.org/wiki/Code_coverage

Outline • What is test adequacy and test coverage? • Line/Statement/Node Coverage • Block coverage • Decision coverage • Tools for the measurement of code coverage • Other code coverage criteria (for future study)

We will now start the JUnit CodeCover Lab (#2)

How do black- and white-box testing relate to one another in real-world testing projects?