Code Coverage Testing Using Hardware Performance Monitoring Support

1. Code Coverage Testing Using Hardware Performance Monitoring Support Alex Shye, Matthew Iyer, Vijay Janapa Reddi and Daniel A. Connors University of Colorado at Boulder Department of Electrical and Computer Engineering DRACO Architecture Research Group

2. Introduction • Code coverage is simple but useful for software testing • Common method of code coverage analysis is through program instrumentation • Insertion of software probes statically or dynamically • Incurs a high overhead! (~50-200% overhead) • Modern processors support contain a hardware Performance Monitoring Unit (PMU) • Itanium, Pentium 4, Power PC • Allow for low overhead sampling of low level information • PMU represents a low-overhead alternative to full instrumentation

3. PMU Itanium-2 PMU Features • PMUs are becoming more advanced • Coarse-grained and fine-grained features • DCPI, Oprofile- PC sampling • But PMU can do more… • For example, branch vectors on Itanium • Obstacles to PMU profiling • Non-deterministic (sampling) • Sample aliasing • Sampling = Less information • Offline analysis can extend PMU information! Goal: Explore PMU-based code coverage by sampling branch vectors and performing offline compiler analysis

4. Code Coverage Framework Configured to sample only taken branches Online Annotated Binary Terminology Branch Vector: Series of addresses from BTB Partial Path: Path of ops in compiler IR PMU Branch Vectors Offline Kernel Buffer Branch Vectors … Address Map Interrupt on kernel buffer overflow Dominator Analysis Partial Paths Branch Vector Hash Table Intermediate File Code Coverage

5. Dominator Analysis • Dominator Analysis • Finds all blocks guaranteed to execute • Cannot be performed effectively online • But is standard in any compiler infrastructure BTB Branch Vector 1-2-3-4 1 Partial Path from Branch Vector 2 Basic Blocks added with Dom. Analysis 3 4 Terminology Dominator: u dominates v if all paths from Entry to v include u Post Dominator: u post-dominates v if all paths from v to Exit include u

6. Methodology • Experiments run on Itanium-2 with 2.6.10 kernel • Developed tool using perfmon kernel interface and libpfm-3.1 to interface with PMU • Only sample taken branches to elongate branch vectors • Set of SPEC2000 benchmarks • Compiled with the OpenIMPACT Research Compiler • With annotations • OpenIMPACT module for offline analysis • Compared to full code coverage information from a Pin code coverage tool Number of Instructions and Actual Code Covered Coverage percentage is the percent of actually covered code discovered with PMU sampling and offline analysis

7. Effect of Sampling Period • Sampling Overhead due to: • Copy BTB to kernel buffer, interrupt on kernel buffer overflow, copy from kernel buffer into hash table

8. PMU vs Actual Instruction Distribution • Kullback-Leibler Divergence • Relative entropy of p with respect to q • d = k=0 pk log2(pk/qk)

9. Code Coverage

10. Multiple Runs • Regular Sampling: 1) gzip, parser, twolf improve greatly • Randomized Sampling may discover code regular sampling cannot

11. Conclusion • Motivates and presents initial results and rational for PMU-based code coverage • An example of using advanced PMU feature with branch vectors • Illustrates how simple offline analysis can extend PMU information • Indicates PMU could be very useful for low overhead profiling and program understanding • Could be promising for profiling of released software Questions?