Testing Computer Software

1. Testing Computer Software Mike Stewart Microsoft

2. Who Am I and Why Should You Care? • Software Test Lead for the Visual FoxPro team • Numerous Knowledge Base articles and white papers for Microsoft • Consultant developing custom applications using VFP, C++, Delphi and VB • Unix sysadmin and systems programmer using C++

3. What this Session is Not • A lot of code examples – no example can fit every situation • Testing the “Microsoft way” – this is all based on software QA industry methodologies • One size fits all – everyone’s testing needs are different • For people with aversions to blue Power Point slides

4. Why Test? • Customer satisfaction – testers are customer advocates • Cost – the later a bug is found, the more it costs to fix • Quality – the earlier a bug is found, the more likely it is to be fixed

5. What is the goal of testing? • Find bugs – more specifically, get bugs fixed • Verify that it works • Prevent errors before they get into the product • Provide information on risk – how solid is the product?

6. Important Things to Remember • “A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.” - Kaner • You will not find all of the bugs • Myers’ 20 line program: 100 trillion paths • Program takes two inputs of four-digit number: almost 400 million possibilities • Test something 10,000 iterations: what if it fails at 10K + 1?

7. Exhaustive Testing • “The only exhaustive testingthere is, is so much testing that the tester is exhausted!” — Hetzel

8. “White Box” vs. “Black Box” Testing • White box testing treats the software as a box that you can see into. • You are looking into the box to see how it works • Assumes you have source code • Black box testing treats the software as a box you cannot see into. • Put in data (input) and see what comes out of the box (output). • No source code needed. • Black box testing on finds about 55% of the bugs

9. Why White Box Testing isn’t Covered • Doesn’t test the software in the manner in which it will be used • It doesn’t do spec testing – it has no way of knowing if it is functioning as designed or not • It does not find missing code – some estimate this accounts for 30% of bugs • Not enough time! It’s a discipline of its own, and we’d be here all day.

10. Steps in testing • Test planning • Write test plans • Spec and code reviews • Build verification – is it even stable enough to test? • Function and system testing – run the full suite of test cases • Media verification • No viruses • Bits on the media are the release bits • The product can be installed from the media

11. Essential Testing Issues • Establish a bug tracking system • ATS Web – free at http://msdn.microsoft.com/vstudio • No sticky notes • Establish processes • Use cases • Test plans • Test cases • Bug entry and resolution • Automate, automate, automate!

12. Bug Tracking • Accountability • Assures that the bug does not fall through the cracks • When closed, we are assured that it is fixed. • Metrics • How great is the severity of bugs found? • How fast are we closing bugs? • Are we ready to ship? • Liability • CYA – cover yourself against legal liability (I am not a lawyer, however)

13. Test Plans • A test plan is a general “plan of attack”, not a detailed list of test cases. • List areas to test, and how they will be tested. • Notepad File/Open example “Notepad’s File/Open functionality will be tested for various file types, file size limits, file name limits and regional encoding. Various media need to be considered, such as removable media and networks.”

14. Test Cases • Each task can be compartmentalized into separate automation programs, or conglomerated into one big program. • Keep test cases atomized: test one particular feature at a time. • Create a matrix, and build test cases from that • Could be something as simple as an Excel spreadsheet • Example: the Notepad File/Open dialog

15. Test Cases for File/Open/Filename • Long file names – boundary testing • Single letter file names – boundary testing • Long extensions – boundary testing • Edit box text length – boundary testing • Invalid characters in file names – functional • Valid file names – functional • Wild card characters – functional • Edit keys (Ctrl-C, etc.) – functional or spec • Paste in binary from clipboard - pathological • MRU dropdown – functional • Path completion – functional • Large fonts – functional • Gain and lose focus – functional • High ASCII chars - functional

16. Bug Entry and Resolution • Minimum to track: • Severity • Priority • The tester who found it • The dev who fixed it • Enter good bugs: • The easier to reproduce, the more likely it is to get fixed • Check on a second machine – could be a configuration/OS issue • If it only happens on one OS, say so

17. Bug Entry and Resolution • If you’re org is tracking metrics, close resolved bugs as quickly as possible • Set severity appropriately, and leave the priority to the project manager(s) • Your org will have to decide how to set severity and priority – everyone has different needs

18. Testing Methodologies • Smoke testing – basic functionality • Exploratory testing • Testing while learning the product • Misses large areas of the product • Boundary conditions • Cannot test every value, so test for the most important values • Hex boundaries and the boundary of field length • Boundary -1, boundary, and boundary +1 • Also test largest value allowed, if it is not a hex value

19. Testing Methodologies • Stress • What happens when a large number of users hit it? • Low memory, low disk space, large amounts of data, repetitive • Bugs can often not be reproduced • Web Application Stress Tool is great for web apps.

20. Testing Methodologies • Functional • Does it do what the spec says it should do? • How does it handle error conditions? • Performance • A discipline unto itself, worthy of an entire session • Regression testing • Make sure nothing got broken • Does not find a lot of bugs

21. Testing Methodologies • Code Coverage – for VFP, use the Coverage Profiler • Code coverage testing tells you what code has not been hit, and therefore needs to have test cases that hit this code. • You are unlikely to get 100% coverage • The VFP Coverage Profiler can also reveal bottlenecks.

22. Testing Methodologies • Race conditions • Two pieces of code are colliding • Especially important in multi-threaded and multi-user apps • Error conditions - does it fail gracefully? • UI • Tab order is easy to miss if you are a mouse-addicted developer • When testing as a dev, do not use the mouse, because you know the mouse will work.

23. Testing Methodologies • Ad Hoc testing – “guerrilla testing” • Great for checking basic functionality • Can give broader testing than more formal means • Hard to verify coverage

24. Automation • The Active Accessibility test harness included with VFP 7.0 • Other automation products: • Rational – Visual Test • Mercury Interactive – Testing suite with various products for web and thick client applications

25. Automation Tips • The test should be self-documenting • Do not use “pass” or “fail” as the test result • Test should say what it expected and what it received, and whether it passed or failed. • Write “drivers” or hooks into your software if needed – for example, VFP’s Active Accessibility or Project Manager hooks • Tests should be self-sustaining – create your own data, and clean up. Expect nothing.

26. When Do You Ship? • You will ship with bugs if your testing is adequate. • Define show-stoppers • Will customers find the bug? • Does it crash your product? • Will only pathologic testers find it? • Define risk • Fixing a bug has a 60% risk of introducing new bugs • When the show-stoppers are fixed, ship

27. Selling Testing to Client • Bulk contracts – sell the job, not the hours to develop it, and the hours to test it. • Once the software is in the customer’s hands, bugs are much more expensive to fix • How much does it cost to have 30 users find a bug vs. a qualified, dedicated test effort at $x/hour, that assures quality? • Other ideas?

28. “Things They Should Teach in the Testing Stuff Class, but Don’t” • Run a high-color scheme like Marine High-Color • Use large fonts (Display/Settings/Advanced) • Throw away your mouse

29. More information • Testing Computer Software – Kaner • The Art of Software Testing - Myer • Software Quality Testing Engineering magazine: http://www.stickyminds.com • STAR Conference – the DevCon for professional software testers (www.stqe.com)

30. ADVISOR DEVCON Web Update Page http://www.Advisor.com/CMF0109update This session WILL have updates.

31. Thank you! Please remember to fill out your evaluation.