Presentation Transcript

1. COMPSCI 230 S2C 2012Software Design and Construction

   Software Testing, Part 7
2. Lecture Plan: Software Testing M 10/9: Goals of software testing; test cases. Myers Ch. 1, pp.1-4. T 11/9: Psych. and econ. of testing; black-box testing. Myers Ch. 2, pp. 5-11. Th 14/9: White-box testing; Myers' principled approach. Myers Ch2, pp. 11-15. M 17/9: “Software Testing: A real world view”. Guest lecture by Shelly Mutu-Grigg, BSc Computer Science Auckland 2003 T 18/9: Myers' principles 2 through 10. Myers Ch. 2, pp. 15-20. Th 20/9: Basics of Extreme Programming (XP). Myers Ch. 8, pp. 177-183. M 24/9: Basics of Extreme Testing (XT). Myers Ch. 8, pp. 183-186. T 25/9: Applied XT and JUnit. Myers Ch. 8, pp. 186-191. Th 27/9: Famous failures. Testing 7
3. Learning Goals for Today Develop black-box unit tests from a unit specification Input validation Boundary-value analysis Automate tests using JUnit This lecture: theory In tutorial and lab assignment: some practical experience Testing 7
4. Myers’ Example: Primality Testing “Develop a command-line application That accepts any positive integer, n, where 0 ≤ n ≤ 1,000, And determines whether it is a prime number. “If n is a prime number, then The application should return a message stating it is a prime number. “If n is not a prime number, then The application should return a message stating it is not a prime number. “If n is not a valid input, then The application should display a help message.” My concern: This is an unlikely example for XUT. It’s a complete application, not an OO unit i.e. a class or a method. A Java app is likely to take input from a GUI, not a command-line. Testing 7
5. Myers suggests writing all tests first This is a valid XP approach – for use when the unit is simple. A more “extreme” approach is to identify a subset of features that you’ll implement first. Write tests for these features Implement them Then add tests for the more difficult features Implement them Iterate until done. In “test-driven development” you start every coding project with a single unit test which is guaranteed to fail, then add one test at a time. If you write all tests before you code anything, you might order your tests by “easy feature first”. Let’s try this with Myers’ example. What’s the simplest feature? Testing 7
6. Colour-coded features “Develop a command-line application That accepts any positive integer, n, where 0 ≤ n ≤ 1,000, And determines whether it is a prime number. “If n is a prime number, then The application should return a message stating it is a prime number. “If n is not a prime number, then The application should return a message stating it is not a prime number. “If n is not a valid input, then The application should display a help message.” My release planning: Red, blue, green. Reasoning: I like to validate inputs before computing with them. Not a good plan: the “coding distance” from blue to green is small, just an output statement. Getting from nothing to red, and from red to blue, is much harder. What feature-set is “halfway” between red and blue? Testing 7
7. A purple feature Test whether the input is divisible by 2. In a primality test on n, we must test divisibility by all primes ≤ . Planning Test-development tasks: Red, purple, blue/green. Coding tasks: Red, purple, blue/green. Scheduling option 1: develop all tests before coding red Myers encourages you to do this. Scheduling option 2: alternate testing & coding. Red test, red code, debug, refactor; purple test, purple code, debug, refactor; blue/green test, blue/green code, debug, refactor. “Test-first developers” take even smaller steps than this! “The development progresses in small steps, where testing and coding alternate. Such a ‘micro-iteration’ does not take more than 10 minutes.” [J. Link, Unit Testing in Java, Morgan-Kaufmann 2003] Testing 7
8. Developing the Red Tests There are many ways to generate tests. Here’s one way to slice-and-dice the problem… Is there more than one input argument? (input: “1 3”; output: “Invalid”) Is there less than one input argument? (input: null; output: “Invalid”) Is there exactly one input argument? (Refinement needed) Is it an integer? (Refinement needed) (input: “-1”, output: “Invalid”) (input: “1001”, output: “Invalid”) (input: “0”; output: “valid” on stderr, refinement needed) (input: “1000”; output: “valid” on stderr, refinement needed) (input: “3”; output: “valid” on stderr, refinement needed) (input: “0xA”; output: “Invalid”) – don’t allow hex input (input: “1.000”; output: “Invalid”) – http://en.wikipedia.org/wiki/Decimal_mark (input: “1,000”; output: “Invalid”) (input: “99999…9” with 1 million digits; output: “Invalid”, with no uncaught exceptions) Testing 7
9. Where should I send my testing output? I’m sending my red diagnostic output to stderr (= System.err) This is convenient during testing, but is hazardous in a release build. Performance may be degraded. System reliability will be degraded, unless stderris routed to a bottomless “bit-bucket” (for example, to System.IO.Stream.Null in .NET). Reverse-engineering attacks (on closed-source software) are facilitated. Some future product (your own, or a competitor’s) may rely on the unadvertised stderr “feature” of your release. If the product has a defined stderrstream (e.g. for debugging in the field), then you should use another stream, write to a file, or write to a method in a test harness. JUnit is a convenient test harness for unit-testing Java code. JUnit’s output-generating code is loaded only when you’re unit-testing. You should not use JUnit during an acceptance test (unless your test cases and the JUnit framework can be included safely in your release code). import org.junit.*; // should we acceptance-test this code??
10. Back to the example… Purple Tests Test whether the input is divisible by 2. In a primality test on n, we must test divisibility by all primes ≤ . Refine the existing decision tree. Is there exactly one input argument? Is it an integer? (Refinement needed) … … (input: 0; output: “valid, even” on stderr, refinement needed) (input: 1,000; output: “valid, even” on stderr, refinement needed) (input: 3; output: “valid, odd” on stderr, refinement needed) This should suffice – we’re testing both “even” and “odd”.
11. Blue/Green Tests In a primality test on n, we must test divisibility by all primes ≤ . Refine the existing decision tree… Is there exactly one input argument? Is it an integer? (Refinement needed) (input: 0; output: “valid, even” on stderr, “Not prime”) (input: 1000; output: “valid, even” on stderr, “Not prime”) (input: 3; output: “valid, odd” on stderr, “Prime”) (input: 1; output: “valid, odd” on stderr, “Not prime”) (input: 2; output: “valid, odd” on stderr, “Not prime”) (input: 25; output: “valid, odd” on stderr, “Not prime”) (input: 961; output: “valid, odd” on stderr, “Not prime”) – note that 961 = 312 (input: 997; output: “valid, odd” on stderr, “Prime”) – a boundary for primes ≤1000 Did I write too many cases? Too few? Myers wrote only 8 cases. I wrote 10 red + 5 blue/green = 15. Testing 7
12. Myers’ advice: validate all inputs! Myers decomposes his XUT example into “two discrete tasks: validating inputs and determining prime numbers”. Advantage: it’s much safer to reuse a unit that validates its input. In its current use, a unit might “always” be invoked on a path where this validation is unnecessary, but this may not be true in all future uses. Disadvantage: “wasted” time on duplicative coding and testing. “Gold-plated” code is heavily validated, and is suitable for re-use. “Marginally safe” code has unvalidated inputs, and is dangerous for re-use. My advice: You should safety-check the inputs of internal units (e.g. for null pointers in C, values that would cause an endless loop), but don’t validity-check them. Units which accept input from I/O devices (e.g. keyboards, storage devices, networks) should carefully validate their input. This validation is called “sanitizing”, when there are specific security requirements for “what the unit should not do” (e.g. to accept input from a SQL-injection attacker). Testing 7
13. Input validation could save $$$ “… Leo Gao [who ran a filling station in Rotorua] and his girlfriend Kara were like millions of couples around the world as they struggled to pay their bills and keep their business afloat. “Today [31 May 2009], they are the subject of an international hue and cry, leaving lawsuits, huffing and puffing private detectives and puzzled police on two continents in their wake. “And all because of a mark on a computer screen one fortieth of an inch across. The desktop in question was in the Christchurch offices of Westpac, a leading New Zealand bank.  “At its keyboard sat a woman with 30 years' experience who was entering the amount [$10,000.00] on a loan clearance form. Every digit, including the two zeroes for the cents, was put in. But one thing wasn't: the decimal point. And its absence gave Mr Gao and friend not $100,000, but 100 times that amount. ” Source: The Independent, 31 May 2009. Testing 7
14. Input validation is an important defense against code injections http://imgs.xkcd.com/comics/exploits_of_a_mom.png Testing 7
15. A Very Brief Introduction to JUnit JUnit is a set of “software tools” for unit testing. Kent Beck adapted it from his earlier SUnit [K. Beck, “Simple Smalltalk Testing: with Patterns”, chapter 30 of Kent Beck’s Guide to Better Smalltalk, 1998]. The syntax and semantics of JUnit are variable, depending on the release version. Old tests must be ported and re-validated, if you’re using a new version of Java or a new version of JUnit. “Write Once, Run Anywhere” does not imply “Write Once, Run At Any Time in the Future”. A cynical joke: “Write Once, Debug Everywhere”. It is possible to write very portable Java, and very portable JUnit tests. You should use only basic features and standard libraries. Testing 7
16. A Test Fixture in JUnit By convention, your extension should be called XxxTest if you’re testing Xxx. import org.junit.*; public class YourClassTest { @Before public void setUp(){ \\ allocate some objects for use during test } \\ put your test cases here @Test public void testCheckPrime() { assertFalse(check4prime.primeCheck(0)); assertTrue(check4prime.primeCheck(3)); } @Test(expected=IllegalArgumentException.class,timeout=100) public void testCheckPrimeRed() { assertTrue(check4prime.inputValidator(“1,000”)); } @After public void cleanUp() { \\ de-allocate your test setup } } Testing 7
17. Learning Goals for Today Develop black-box unit tests from a unit specification Input validation Boundary-value analysis Automate tests using JUnit This lecture: theory In tutorial and lab assignment: some practical experience Testing 7