Unit Testing

1. Unit Testing What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se

2. My View on Unit Testing • Part of something bigger – testing on many levels • Fast and early feedback! • Tight feedback loop, results within seconds • Early implies lower cost • Confidence! • Less fear of refactoring • Better code! • Testability requires decoupling and usable APIs • Reliability! • Possible to remove all non-determinism

3. Definitions and Terminology Unit test, functional test, system test – what does it mean?

4. Definitions and Terminology • There are no definitive answers • Different dimensions of automated tests: • Level – Unit, component, integration, system • API – Developer test, external test • Characteristic – Functional, non-functional • Acceptance testing, regression testing

5. What Level Are We Testing On? • Unit – The smallest unit of program code, normally a single function, class with methods or data structure with functions • Component – A set of units integrated together, normally a module or subsystem comprising an isolated set of classes or files. • Integration – Several layers of a stack integrated, normally a stand-alone application (e.g. client or server) • System – Several stand-alone applications integrated (e.g. clients and servers), normally the full system

6. What API Are We Using For Testing? • Developer tests – Function calls to an internal or external interface, normally test and production code running in the same process • External tests – Invokation of a non-programming interface, normally running in separate processes or machines. For example, UI (simulated user input), network (HTTP etc), IPC.

7. What Aspect Are We Testing? • Functional – The behavior of the system, normally expressed as a step-by-step instruction with some alternative paths • Non-functional – The characteristics of the system, normally expressed as measurements and numbers. For example, latency, round-trip time, memory/CPU/bandwidth consumption, scalability, availability.

8. Functional “Component test” – personal favorite. Provides stability just like a “functional test” (test code less brittle than unit tests). Provides flexibility, reliability and fast feedback just like a “unit test” (quick to write, easy to mock out problematic parts, results within milliseconds).

9. Non-Functional

10. Choose How to Test • When choosing, consider • Cost (of writing, maintaining, execution resources etc.) • Feedback loop (time) • Reliability • Trade-off example: Mocking or using system x? • Mocking: cost of writing and maintaining, risk of modeling wrong behavior. • Using: cost/risk of non-determinism, cost of test resources and longer execution times.

11. Regression And Acceptance Testing • Definition: “regression” or “regression bug” • Something that was once considered “done” but does not fulfill requirements anymore • Acceptance testing: the use of a test to determine if a feature fulfills the requirements • Acceptance tests becomes part of the regression test suite once a feature is “done”

12. How to Unit Test Necessities of effective unit and component testing

13. Dependencies • Assumption: we want to achieve fast feedback and reliable tests • Note, there are other kinds of tests that also have merit • Remove non-determinism: we need to be in control • Randomness, time, file system, network, databases, multi-threading • Remove dependencies to non-deterministic components

14. Dependencies • What kind of problematic dependencies to you have in your code base?

15. Getting Rid of Dependencies, C++ • Example: Randomness • class A { public: void foo() {boolcoinflip = rand() % 2 == 0; if(coinflip) … }}; • Problem: dependency to non-deterministic component (randomness)

16. Getting Rid of Dependencies, C++ • C++ solution: introduce interface and use dependency injection • class IRandomness {virtual boolcoinflip() = 0;};

17. Getting Rid of Dependencies, C++ • class A { public: A(IRandomness &r) : randomness(r) {} // ctor inject void foo() { if(randomness.coinflip()) … }private:IRandomness &randomness;};

18. What Is a Mock Object? • Conforms to interface • Programmable behavior • For example, “if someone calls your send function with argument ‘hello’, return 7” • Can verify function invocations • Mock frameworks available • E.g. gmock (Google) • Much more elegant in reflective languages

19. Getting Rid of Dependencies, C++ • Test code: • void testFoo() {RandomnessMock random; // inherits IRandomness A a(random); a.foo(); …}

20. Getting Rid of Dependencies, C++ • Production code: • void main() {Randomness random; // inherits IRandomness A a(random); …}

21. Bi-Directional Dependencies Application (inherits) sendData INetworkReceiver INetworkSender onDataReceived (inherits) Network

22. Getting Rid of Dependencies, C • Function pointers = hard-to-read code • Link-time tricks • Link with other translation unit defining e.g. rand()

23. Getting Rid of Dependencies, C • RandMock m; // mixed C/C++ test codeRandMock &getRandMock() { return m; // used by test case to access mock}int rand() {m.rand(); // forward to mock object} • Drawbacks: hard to understand test case code, one test executable per component to test = harder to maintain • Or clean C solution

24. Code Coverage • Don’t focus on high levels of code coverage per se • Necessary, but not sufficient • Instead, focus on covering the most important user scenarios • After that, use code coverage to determine which code has not been covered • Analyze to understand what user scenarios are missing

25. Unit Test Frameworks Test runners/frameworks • GTest • UnitTest++ • CppUnit • C Unit Testing Framework Mock frameworks • GMock • Hippo • Mockpp • Amop

26. Vision Agree on what to aim for

27. Fast Feedback Example From a previous job: • Proprietary hardware/software/OS platform • Very expensive hardware = bottleneck in testing • Emulator to test software locally on Linux machine • Ran unit tests in emulator, turn-around time = 15 mins • Unacceptable

28. Fast Feedback Example • Main problem: emulator start-up time • Challenge: proprietary types used (e.g. string classes) • Solution: fake proprietary types (string with std::string, many others just empty implementations) • Ran unit tests on local machine,turn-around time < 15 seconds (from 15 mins)

29. Fast Feedback Example • Invest to speed up feedback • Invest to remove testability obstacles • Unit testing should be easy, fast and fun

30. Development Excellence • Defect prevention • Continuous improvement - feedback from bugs in order to improve software development process • Relentless testing • Regression detection • Continuous integration running all regression tests often • Focus on both functional and non-functional requirements • Always releasable • Keep CI builds green • Keep the product working

31. Continuous Integration • Staged • Stage 1 – build, unit tests, code coverage and functional smoke tests – 10 mins • Stage 2 – extensive functional tests, smoke non-functional tests – 1 hour • Stage 3 – full functional and non-functional regression tests , static and dynamic analysis – 12 hours • Run stages per • Commit (fine-grained tracability, expensive), or • Regularly (1. commit, 2. hourly, 3. nightly)

32. Code Analysis Tools • Static analysis (run on source code) • Lint – free, local analysis only, lots of false positives • Klocwork – commercial, global analysis • Coverity – expensive, global analysis, good precision • Dynamic analysis (need representative data) • IBM Rational Purify – commercial, memory leaks, double frees, buffer overflow, use of uninitialized memory. Instrumentation. • Valgrind – free, memory problems, heap, usage, cache usage, call graphs, threading problems. No instrumentation.