Coverage tools

1. Coverage tools • Program is typically compiled with special options, to add extra source or object code. • Additional data structures, such as a flow graph, may also be created. • Program is run, possibly via test cases • During execution, information is accumulated and written to an output file. • Post-processing phase: • User report is generated from output file.

2. How to measure coverage? • Possibilities: • Instrument the source code before compilation • Instrument the virtual machine or object code. • The Apache Byte Code Engineering Library (BCEL) is used by several Java coverage tools • http://jakarta.apache.org/bcel • Use a customized virtual machine.

3. Coverage filters • Normally, a filtering function is provided to determine for which classes coverage is to be measured: • Include or exclude specified packages • Filter out calls to provided class libraries or framework. • Exclude certain source code statements • Example: When a Java class is not intended to have instances created, the default constructor is often set to be private. This would show up as uncovered code.

4. Source code instrumentation • Create a copy of the source code, instrument the copy, and compile the instrumented copy to .class files. • This is the approach used by Clover. • Typical method: • Set up an array of counters. • Insert source code at points of interest that increment specific counters. • Coverage tool calls main() method • After main() method returns, dump values of counters.

5. Source code before instrumentation public final void add( final MatrixRow<T> newValues ) { if ( getNumColumns( ) == 0 ) { contents = newValues.copy( ).getContents( ); } else { contents.addAll( newValues.copy( ).getContents( ) ); } } • A counter is inserted for each method, statement, and branch. • Statement counters are before each statement, in case the statement throws an exception. • Branch counters need to identify true and false branches taken.

6. Same code after instrumentation public final void add( final MatrixRow<T> newValues ) { CLOVER.M[348]++; CLOVER.S[1814]++; if ( ( getNumColumns( )==0 && ++CLOVER.CT[338]!=0 | true ) || ( ++CLOVER.CF[338] == 0 & false ) ) { CLOVER.S[1815]++; contents = newValues.copy( ).getContents( ); } else { CLOVER.S[1816]++; contents.addAll( newValues.copy( ).getContents( ) ); } } statement counters branch counters method counter

7. Counting branches • Count number of times that a branch condition evaluates to true and false. if ( ( aBoolean && ++CT[338]!=0 | true ) || ( ++CF[338] == 0 & false ) ) • If aBoolean is true, then the && is also evaluated, which will increment the CT counter. We want the expression’s overall evaluation to be true. • If aBoolean is false, then execution switches to the right side of the || clause. Increment the CF counter, but then ensure that the entire expression still evaluates to false to take the else branch. • Short-circuit evaluation means that as soon as the expression result can be deterimined, evaluation stops. This avoids: • Incrementing the true counter if aBoolean is false. • Incrementing the false counter if aBoolean is true.

8. Analysis • Advantages: • Results are directly mapped to source code. • If a statement throws an exception, it will be counted as having been executed. • Disadvantages: • Requires keeping a second version of source code, and ensuring that original source is not over-written. • Instrumented code must not affect original code execution. • Management of .class files from compiling alternate source code. • Memory usage: for statement coverage alone, every statement requires one integer as a counter.

9. Byte code instrumentation • Can be done in two ways: • Instrument object files in advance to produce a alternate version. • Example: create instrumented .class files for Java. • Instrument object files as they are used. • Example: create customized .class file loader that inserts the instrumentation at run time.

10. Analysis • Pre-instrumentation: • Advantages: • source code is not modified. • can be done as a batch process before execution • Disadvantages: • management of two .class file versions. • mapping of source code statements to/from object code • counting source code statements can be difficult • object code may need extra information such as special compiler options

11. Clover • http://www.atlassian.com/software/clover • Commercial tool: requires license • Supports method, statement, and branch coverage • Reports its findings in multiple formats • From project level down to individual lines of source code • Shows list of test cases that hit each line of course code • Historical charting of code coverage and other metrics • Plug-in versions available for Eclipse, NetBeans

12. Clover plugin for Eclipse • Image source: atlassian.com

13. Clover coverage filters • One can choose not to instrument certain types of blocks in the code (e.g. assertions and exception catching), in order to focus on the coverage of interest.

14. Emma • Open-source tool • Supports class, method, “basic block”, and line coverage. • “Fractional” line coverage supported, but not branch coverage. • Standalone version works with Ant builds • http://emma.sourceforge.net • Eclipse plugin EclEmma also available • http://www.eclemma.org

15. Emma • Uses byte code instrumentation. • Classes can be instrumented in advance, or during class loading. • Tool keeps a metadata file to associate byte code with source code • A regular compile will allow for only limited metadata. • A compile with debug option will provide more information to support branch coverage.

16. Block Coverage • Block coverage assumes that if a block of statements without branches is entered and exited, all statements in the block were executed. • That is, the counter is at the end of the block, instead of before the source code statement. • Result: If an exception occurs in the block, the entire block is not recorded as having executed. • This may be fine for application source code, but it does not work well with JUnit test source code or in code for which exceptions are commonplace. • JUnit throws exceptions internally when tests fail, so the test may not have appeared to be executed.

17. Emma coverage report

18. Emma source code annotations

19. Fractional line coverage Only part of conditional executed Loop increment not executed

20. Anomalies with byte code coverage • There are cases where the compiler may emit multiple copies of object code for a single instance of source code. • In the above example, there is an instance variable initialized for each of two constructors. However, only one constructor is called. The result is the yellow annotation at initialization statement 17.

21. Coverlipse • Open source Eclipse plug-in • http://coverlipse.sourceforge.net • Provides block coverage, all-uses data flow coverage • No branch coverage • Uses Apache BCEL to instrument bytecode • Appears to be dormant, and not compatible with Java 6 or Eclipse 3.3.

22. Line coverage report

23. All-uses coverage report

24. All-uses coverage report

25. CodeCover • Open source Eclipse plug in • Web site: http://codecover.org • Version 1.0.0.0 available Nov. 2007 • Performs source-code instrumentation to obtain: • Statement coverage • Branch coverage • Loop coverage: loop executed zero/once/many times. • Multiple-condition coverage (called “strict condition” coverage by CodeCover)

26. CodeCover summary report

27. CodeCover detailed report

28. CodeCover features • Boolean value analyzer: shows how many Boolean combinations in conditions have been covered. • Code “hot spots”: highlighting of code that is executed more frequently than most. • Test correlation matrix: for each pair of test cases, the overlap in coverage for the two test cases is shown. • Source instrumentation is done following a statement. • The following statements do not appear as coverable items: • throw statements • return statements • Methods that throw exceptions are marked as not covered.

29. CodeCover Boolean analyzer • For the compound condition shown, combinations of atomic conditions that have occurred are shown. • The x shows a short-circuit evaluation.

30. CodeCover Correlation view • The colours give an indication of the overlap for pairs of test cases. • The selected square shows that for these two test cases, they have 24 and 25 coverable items, and that 19 are shared, for an overlap of 88.1%

31. Cobertura • Open source tool for Java code coverage • http://cobertura.sourceforge.net • Measures line coverage, branch coverage, and McCabe cyclomatic complexity metric for methods. • Branch coverage requires compilation with debug option. • Must be run as an Ant task, with four stages • Compile source code • Instrument classes • Run code • Generate report

32. Cobertura summary report

33. Cobertura detail report

34. Code Coverage: Benefits • If you haven’t exercised part of the code, it could contain a bug. • Can be used to locate “dead” or unexecutable code. • WARNING: Be very careful when removing code, if the purpose is to test for some exceptional condition that is difficult to trigger.

35. Code Coverage: Caveats • Coverage tests that the code has been exercised, and not that you built what the customer wanted. • If the logical structure of the code contains errors such as a missed case in a switch statement, the absence of code will not be detected. • 100% coverage is often unrealistic, especially for fine-grained coverage measures • Effort to achieve high coverage may be unjustified in terms of: • time spent • producing unreliable test cases • 80% coverage is a “rule of thumb” often used in industry

36. References • Clover: http://www.atlassian.com/software/clover • Emma: • http://emma.sourceforge.net • http://www.eclemma.org • Coverlipse: http://coverlipse.sourceforge.net • CodeCover: http://codecover.org • Cobertura: http://cobertura.sourceforge.net • A. Glover, “Don’t be fooled by the Coverage Report”, IBM developer works article http://www-128.ibm.com/developerworks/java/library/j-cq01316 • S. Gornett, “Code Coverage Analysis” • http://www.bullseye.com/coverage.html