CS 5150 Software Engineering

1. CS 5150 Software Engineering Lecture 17 Testing Large Systems By Stephen Purpura October 27, 2010

2. Testing Begins on Day 1 A professional project team begins with: (a) An idea (usually mused over for a while between friends) A program/product manager A development manager (d) A test manager Today, we’re going to focus on how the Test Manager thinks

3. Good Test Managers • Before the project starts, they build the first draft of the Test Exit Criteria which answers:“How do we know when we’re done?” • The Test Manager: • Drives the testing framework and approach • Is a great communicator • Understands the value of defect-fix-build-retest cycles that work like clockwork. THINK BOTTLENECKS.

4. Example of a large scale system test: a browser At one point, the second largest automated software testing infrastructure in commercial software development was used to test Internet Explorer. The largest tested Windows. Why so many computers?

5. Different Testing Methodologies • In large projects there are many stakeholders. • Usually, they believe in different schools or methodologies. • You need to balance this from the start. • An example of a methodology: Agilehttp://pettichord.com/http://www.io.com/~wazmo/papers/four_schools.pdf • Five different schools (from pdf above): • Analytic, Standard, Quality, Context-Driven, Agile

6. A Test Pass A Test Pass: all of the actions involved in assessing whether a software system meets its requirements. This definition implies: • All of the requirements are expressed as “tests” (usually with instrumented indicators) • The team documents all of the steps required to run the tests, so that the test pass can be replicated. • The team runs the test pass multiple times. Else why write it all down?

7. Automated Testing Methods • Two different types of • Scripted testing • Programs that manipulate internal or external interfaces • “Screen Scrapers” • Webload (http://www.webload.org) can automate tests by capturing mouse movements, keyboard events, etc. • Both methods can be “fed” into scripted load testing solutions like WebLoad. • Note: UserTesting.com, while useful, is not ‘automated testing’.

8. The Test Harness • Test Harness: the program that controls the automated testing scripts. • Usually first built for “smoke testing” • Expands to replicate the maximum required load (simultaneous transactions, MTBF, more scenarios) • Smoke Testing: the minimal set of tests one runs to make certain that daily check-ins to the source control system didn’t break critical functionality in the daily build. • I integrate the smoke test into the “make” command

9. Scenarios Make Great Test Cases • Success and failure scenarios and use cases are the foundation of great testing systems. • Over the course of product development, you usually uncover additional scenarios by watching people break the system. • How does this happen …

10. Eating your own Dog Food • “Eating your own dogfood” refers to a situation when a team decides that it will use the software it is developing to conduct its daily business and operations. • If people use the system every day as part of their work, they find problems with it. These problems frequently become new scenarios for the test infrastructure. • With operating systems and browsers, this is easy. With IT systems (like human resource systems), this is less possible. But get creative.

11. Additional Sources of Test Scenarios • Ergonomics • Usability labs • In-home testing • Customer Support • Consulting • Sales Engineers • “Mail a PC” to family members program • Drive to Fry’s and buy one of everything • Executive dogfood • Executive Mission Statements • Other

12. Dealing with Mixed User Interface Feedback • Market segmentation (try dividing the feedback by demographic groups to look for patterns) • Example 1: Men over age 65; Income < Poverty Line • Example 2: Ages 20 – 24; college educated; • “Interface is not cool” • Ask questions like: • Why is this group buying the product? • Which demos like the product and why? • Is their issue related to the core “buying story”?

13. Beta Testing • Beta Testing: early release of a product for the purpose of accumulating feedback that can be integrated into the product. • Validates requirements • Uncovers buying issues • Generates testing scenarios • Users have a lot of diversity in both their uses and their environments. Exploit the diversity to get better data about the coverage of your test pass.

14. Permutations and Combinations • How many versions of Microsoft Windows are there? • 7? 35? 1100? • A version is a build number plus an install set: • Windows XP SP 2 with 7 of the 49 security patches • Win64 vs. Win32 • Office installed vs. not installed • IDE v SCSI v SATA drivers • Video adapter driver • Etc.

15. Durability • How long should you make the tests run? • Once? • Multiple times? • A number calculated based on analysis? • Determining MTBF is important because it is a measure of resilience (frailty). These tests frequently insert random events in the system as a part of the test harness.

16. Acting on the Results • Once you run the test pass, what do you do? • One bug for each failure • Bug counts go up because • Test coverage increased • Code quality went down • Major regression • Code base change that requires change in the test pass • Bug counts go down because: • Test team is not testing or losing creativity

17. End Game • The End Game: the process of balancing feedback from sales, marketing, executives, customers, engineering, testing, etc. • Usually involves: • A stabilization of bug find/fix ratios • A code freeze • A full test pass and subsequent analysis • Discussions among the stakeholders that begin with: Who would it effect if small corporate customers couldn’t …

18. How to Decide When to Release Imagine this scenario. What would you do? The browser has a bug which causes rendering to be off by 3 pixels for certain combinations of tags within <DIV>s The code works correctly on Windows 10 but not Windows 9. Fixing the bug would require adding 2,000 lines of code to the rendering engine Failure to fix will cause W3C standards advocates to complain.

19. How to Decide When to Release Imagine this scenario. What would you do? 3 months prior to the scheduled release, you are in a code freeze. Privacy advocates push Congress to pass legislation that would disable all cookies in the browser. The legislation passes. The legislation makes it illegal to sell consumer products or publish web sites that use cookies.

20. How to Decide When to Release Imagine this scenario. What would you do? The Final Four is in two weeks. You run ESPN’s web site development and the bracket downloader fails on the version of Windows 10 systems that was released yesterday.

21. Hiring a Test Manager Test Managers are different Some are good at hiring testers Some are good at thinking through business process Some have deep technical background in important areas Some have relationships with important players Test Managers need to be able to articulate arguments in terms of the requirements. They also need to be able to argue for the importance of their employees during personnel reviews.

22. Thanks and Questions • Thanks to the following people for assistance: • Joel Maher (a great test engineer that I’ve worked with) • Bill Walrond (a great PM who provided feedback) • Questions?