Context Driven Testing

See...Think…Respond Context Driven Testing Adam White Founder Manager Test Engineering and Escalations

Agenda – Context Driven Testing • Why did this topic come up? • What exactly is context driven testing? • Why should you care about context driven testing • How can you apply it at your organization?

Background • Who started this topic • When was it started? • Why talk about it now?

Across the portfolio –How would we test these products? ScriptLogic Desktop Authority Comprehensively manages the Windows desktop throughout the desktop lifecycle, reducing the total cost of desktop and application ownership by enabling administrators to proactively manage, inventory, secure and support desktops from a central location. Security Explorer A powerful, graphical solution for real-time, interactive management of access controls and Windows NT/2000/XP/2003/Vista security on Windows file servers and workstations across the enterprise Acronis True Image Enterprise Server Allows you to 1) create an exact server disk image, including the operating system, applications, and configurations and 2) backup mission critical databases. Eyeblaster eb.campaign manager An all encompassing ad server that is designed specifically for the generation and delivery of campaigns/creative units from rich media and standard banners to search bid management and more. Awareness Optimization Migration

Feedback From the Portfolio • Every major feature should have, at a minimum, a couple bugs found and assigned in the defect tracking system. • No bugs almost always means the feature has not been well tested. • A tester’s primary role is to find bugs and not to simply pass test cases. • Limiting the testing cycle, by extending development time, results in a product with a high number of defects. • All features presented by the product should function as intended.

Feedback From the Portfolio - continued • A quality product depends on adequate and accurate requirements. • Requirements are the basis for adequate test coverage planning. • Automated test scripts should be maintained for core product features. • Automated test scripts allows QA maximize test coverage and maintain staff size • The number of automated scripts should increase from release to release in order to extend the amount of functionality covered by automated testing.

The Seven Basic Principles of Context Driven Testing 1) The value of any practice depends on its context. 2) There are good practices in context, but there are no best practices. 3) People, working together, are the most important part of any project's context. 4) Projects unfold over time in ways that are often not predictable.

The Seven Basic Principles - continued 5) The product is a solution. If the problem isn't solved, the product doesn't work. 6) Good software testing is a challenging intellectual process. 7) Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

What people tend to forget about testing • Testing groups exist to provide testing-related services. They do not run the development project; they serve the project. • Testing is done on behalf of stakeholders in the service of developing, qualifying, debugging, investigating, or selling a product. Entirely different testing strategies could be appropriate for these different objectives. • It is entirely proper for different test groups to have different missions. A core practice in the service of one mission might be irrelevant or counter-productive in the service of another. • Metrics that are not valid are dangerous. • The essential value of any test case lies in its ability to provide information (i.e. to reduce uncertainty).

What people tend to forget about testing • All oracles are fallible. Even if the product appears to pass your test, it might well have failed it in ways that you (or the automated test program) were not monitoring. • Automated testing is not automatic manual testing: it's nonsensical to talk about automated tests as if they were automated human testing. • Different types of defects will be revealed by different types of test • Tests should become more challenging or should focus on different risks as the program becomes more stable. • Test artifacts are worthwhile to the degree that they satisfy their stakeholders' relevant requirements.

What people tend to forget about testing • Article • Aligning Testing with its Context. • http://www.stpmag.com/retrieve/stp-0612.htm. • Resources • www.context-driven-testing.com • www.developsense.com • www.satisfice.com • www.testingeducation.org • Training • Rapid Software Testing by Michael Bolton and/or James Bach • Highly Recommended!

Questions? What else is important to you?

Context Driven Testing