Traditional Software Testing is a Failure!

1. Traditional Software Testing is a Failure! Linda Shafer Director, Software Quality Institute (SQI) The University of Texas at Austin www.utexas.edu lshafer@mail.utexas.edu Don Shafer Chief Technology Officer, Athens Group, Inc. www.athensgroup.com donshafer@athensgroup.com

2. Seminar Dedication Enrico Fermi referred to these brilliant Hungarian scientists as “the Martians,” based on speculation that a spaceship from Mars dropped them all off in Budapest in the early 1900’s. Traditional Software Testing is a Failure!

3. And let us not forget…. 1994John Harsányi (b. 5/29/1920, Budapest, Hungary, d. 2000)  "For his pioneering analysis of equilibrium in the theory of non cooperative games.“ Shared prize with A Beautiful Mind‘’s John Nash Traditional Software Testing is a Failure!

4. Process Steps Process Gates Prototypes PROTOTYPE 1 REQUIREMENTS DEFINITION REVIEW HIGH LEVEL DESIGN PROTOTYPE 2 REVIEW DETAIL DESIGN PROTOTYPE 3 REVIEW SYSTEM CONSTRUCTION REVIEW VERIFICATION & VALIDATION REVIEW SYSTEM DELIVERY POST IMPLEMENTATION REVIEW REVIEW Project Management Support Processes Risk Reduction Training Planning Configuration Management Estimating Metrics Quality Assurance Development Life Cycle Model Traditional Software Testing is a Failure!

5. What is Test Planning? • What is test planning? • A phase during which the test manager or lead works with the project team to discover the challenges, dependencies, strategies, and tactics underlying the test effort • What is a test plan? • A written record of this process of discovery, useful for communicating the test effort to testers, peers, managers • Early drafts of the test plan can promote discussion, feedback, concurrence, and commitment • Possibly multiple documents when tests have… • Different time periods • Different methodologies and tools • Different objectives • Different audiences Traditional Software Testing is a Failure!

6. Test Planning • Once the scope, budget, and schedule are decided, the strategies and tactics are generally documented in a test plan • IEEE test plan template includes sections on… • Introduction (including cross-references) • Scope, including what is and isn’t to be tested • Test tools, techniques, and tasks required to configure the environment, develop the tests, and execute the tests • The test execution process, including ascertaining test case pass/fail status and determining readiness for test execution • Test documentation, status reporting, and other deliverables • The hardware, software, infrastructure, and other external dependencies for the test environment, including processes • Test team staffing and training considerations • Test milestones, especially dates for test deliverables • Risks to the test effort itself Traditional Software Testing is a Failure!

7. Why Do Organizations Test? • Testing doesn’t happen for its own sake, in a vacuum, but is part of a project or operation • Organizations invest in testing for benefits • Improved reputation for quality • Lower post-release maintenance costs • Smoother release cycles • Increased confidence • Protection from lawsuits • To reduce the risk of lost missions or even lives Traditional Software Testing is a Failure!

8. Testing: A Risk Management Activity • Testing, tactically • The search for defects in software • Testing, strategically • A risk management activity that reduces the likelihood of the organization having to bear certain unanticipated future costs • Like insurance, but without the same level of actuarial (statistical) certainty • Testing requires careful planning, followed by crisp execution by a talented team with significant resources • Testing is critical to long-term success • Good testing groups provide timely, accurate assessments of quality to the organization • The alternative: “quality roulette” Software project managers juggle four areas of risk: features, budget, schedule, and quality. Smart trade-offs require careful management Traditional Software Testing is a Failure!

9. Test Phases and Objectives • Unit/Component Test • Find bugs in the individual pieces of the system under test before the pieces are fully integrated into the system • Integration/String Test • Find bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come together • System Test • Find bugs in the overall and particular behaviors, functions, and responses of the system under test as a whole • Acceptance/Pilot Test • Demonstrate that the product is ready for deployment/release Proper definition of entry and exit criteria in test plans ensures the smooth transitions between phases and hand-offs between responsible parties that are essential to effective and efficient testing Traditional Software Testing is a Failure!

10. Testing Throughout the Organization Execution activities for test phases Selecting the right phases, defining proper criteria, and abiding by those criteria (only waiving them through mutual stakeholder consent) is key to fitting the testing effort into the project context Traditional Software Testing is a Failure!

11. The “V” Model (Modified Waterfall) • Usually schedule- and budget-risk-driven • Plan test phases while designing and implementing system • Plan in reverse order of execution • Intuitive and familiar model • Long-range plans risky and inaccurate • When plans fail, test execution--at the end--suffers Traditional Software Testing is a Failure!

12. The Spiral Model • Feature-risk-driven • Must deliver right product • Must manage risky areas • Discover the needed features and resolve risks through prototyping • Testing, redesign, and prototyping continuous once first prototype is released for experimentation • Do not confuse with waterfall • Do not deliver unready prototypes Iterations continue until the right feature set is decided upon, then the final product is developed Traditional Software Testing is a Failure!

13. Evolutionary and Incremental Models • Schedule-risk-driven • Must hit market window or delivery date • Feature set grown around core functionality • Can ship any time once the core functionality is ready--only question is what features • This is becoming a popular approach • Varies across a spectrum of formality from Extreme Programming to Rapid Application Development • Still a temptation to ship a destabilized or buggy product, especially late-arriving features • With agile methods, the full role of testing just gradually becoming recognized Traditional Software Testing is a Failure!

14. Code-and-fix: A (preventable) leading cause of project death Regrettably Common Model: Code-and-Fix • The process—if you can call it that! • Developers write and debug code (no unit tests) • Developers release code to an informal test team—often junior developers • Even with unskilled testing, testers find many bugs, which are often fixed live on the test environment by developers (little/no CM) • Repeat all the above steps at the same time • Code-and-fix arises from organizational unawareness or willful denial of software realities • The process continues until money, time, and/or patience run out…then you release, cancel, or go out of business Traditional Software Testing is a Failure!

15. Testing Maintenance Releases • Regression is the big risk for maintenance releases • Automated or manual test repetition (total or partial) is often used • Alternatives: release less frequently, use beta/pilot customers • Some organizations try to put a major release worth of features into a short maintenance release • Scarce time for new tests, but change effects are nonlinear Traditional Software Testing is a Failure!

16. System Integration • Some projects involve primarily acquiring, integrating, and customizing existing software • The amount of testing increases with… • The extent of the customization • Unusual or unique workflows/use cases • The number of supported environments • A smaller installed customer base • A “younger” release (e.g., X.0 vs. X.1) • Coupling between multiple components • Poor vendor testing • Try to leverage vendor testing and test tools Traditional Software Testing is a Failure!

17. Hardware/Software Development • Projects sometimes include custom hardware development or integration • This can create complications, such as… • Skills mismatches if software testers must test hardware • Insufficient engineering prototypes for test use • Hardware may not be “testable”; i.e., internal states—esp. error states—may not be visible w/o tools or special skills • Through thoughtful advance planning, hiring of an appropriate test team, and careful execution, these issues can be managed • Don’t forget hardware (reliability) testing! Traditional Software Testing is a Failure!

18. InterWavetm Testing Continuum Traditional Software Testing is a Failure!

19. Test Planning in the Testing Process The tester’s role focuses on some of these activities, not all, depending on how roles are defined. However, test planning requires an understanding how the test process works and how it fits into the overall project from a big picture perspective. Traditional Software Testing is a Failure!

20. High Fidelity Test System Traditional Software Testing is a Failure!

21. Low Fidelity Test System Traditional Software Testing is a Failure!

22. Identifying the Risks to System Quality You should test the critical, untested quality risks Traditional Software Testing is a Failure!

23. Aligning Mission, Strategies, and Tactics The more aligned all these elements are, the more effective and efficient the test effort will be…including alignment with the project Traditional Software Testing is a Failure!

24. A Constellation of Test Strategies • Different test teams use different strategies • Depending on context, any of the following strategies can succeed—or fail • Let’s survey some common ones • The categories are points along a spectrum • Blend and use multiple strategies Traditional Software Testing is a Failure!

25. Analytical Test Strategies • Risk-based. Use formal (FMEA, ISO 9126) or informal techniques to assess and prioritize quality risks. Test accordingly. • Object-guided. Look at requirements, design, or implementation objects to determine testing focus. Often relies on documentation. • Fully-informed. Study the system, usage profiles, target configurations, and any other data. Test based on broad, deep knowledge. Traditional Software Testing is a Failure!

26. Model-based Test Strategies • Scenario-based. Test according to real-world scenarios across the system’s functionality. • Use-case-based. Translate use cases into test cases. Cover all the use cases. • Domain-based. Analyze different domains of data and processing in the system. Pick the best test cases in each domain, determined by likelihood of bugs or prevalent usage. Traditional Software Testing is a Failure!

27. Methodical Test Strategies • Learning-based. Use checklists developed over time to guide testing. • Function-based. Test each and every function of the system, one at a time. • State-based. Test every state and, if possible, state-transitions that can occur. • Quality-based. Test the important “-ilities” for the system; e.g., functionality, usability, reliability, performance, and scalability. Traditional Software Testing is a Failure!

28. Process-Oriented Test Strategies • Standard. Test by standards, (e.g., IEEE 829), including standards-based processes. • Agile. Test following lightweight processes, looking for likely bugs, and focus on being able to respond to late changes. • Automated random. Test with enormous random data sets using an automated tool. • Automated functional. Test the functionality using repeatable scripts. Traditional Software Testing is a Failure!

29. Dynamic Test Strategies • Bug hunting. Use bug profiles, taxonomies, and hunches to focus testing where bugs are. • Intuitive. Test according to what the collective experience, wisdom, and gut instincts of the test team say needs testing. • Exploratory. Simultaneously learn about the system, test, find bugs, and refine the test approach and focus. Traditional Software Testing is a Failure!

30. Philosophical Test Strategies • Exhaustive. Test extensively, assuming that everything and anything can and will break. • Shotgun. Test wherever and whatever, randomly distributing the test effort, within resource and schedule boundaries. • Externally-guided. Test according to external direction, including deciding on correctness. Common guides include programmers, users, support, analysts, sales and marketing. Traditional Software Testing is a Failure!

31. Regression Test Strategies • Rerun every single test for every change • Can we afford it? • Will it take to long • Do we need to do so? • Automation can make retesting cheap and fast, when effectively applied • Structural coverage analysis is also used • Bundling changes in fewer releases helps • Staged releases to select customers; e.g., beta testing, phased rollouts, pilots, etc. Traditional Software Testing is a Failure!

32. Resources Money, people (testers and stakeholders), time Test environment Availability, quality, support Test tools Availability, skills, time Stakeholder (test, project) expectations, proclivities Coverage, accuracy, reports Risk-taking, value of quality Risks Likelihood of failures Failure effects Legal, ethical concerns System under test Incoming level of quality Application domain Development, build, and integration processes Test team mission Test system design Strategic Considerations Traditional Software Testing is a Failure!

33. We are often asked to estimate when testing will finish What makes an estimate a good one? Accurately predicts and guides the project’s future Realistic: All tasks included, accurately estimated, risks understood and mitigated Actionable: Clear ownership by committed individual contributors, assigned resources, known dependencies Some Estimation Terminology Project: Temporary endeavor undertaken to create or provide service or product Test subproject: The subset of the project performed by the test team to provide test services to the project Project schedule (a.k.a., work-breakdown-structure): Hierarchical decomposition of project into phases, activities, and tasks with resources and dependencies “How Long Will Testing Take!?!” Traditional Software Testing is a Failure!

34. Developing a Work-Breakdown-Structure • What are the major stages of a testing subproject? • Planning • Staffing (if applicable) • Test environment acquisition and configuration • Test development • Test execution • Break down to discrete tasks within each stage • What test suites are required for the critical risks? • E.g., functionality, performance, error handling, etc. • For each suite, what tasks are required? • Tasks should be short in duration (e.g., a few days) • “Inch-pebbles” as well as “milestones” Traditional Software Testing is a Failure!

35. How Can We Measure Task Completeness? • You can (truly) measure when tasks are done when each task produces a key deliverable • Internal: Test cases, test tools, test data • Incoming: Feature-complete dates, unit test results, test releases, test environments • Outgoing: Test plans, bug reporting systems, results reporting (dashboard) • Contextual: Metrics, assessments to customers, release notes… • These deliverables are often inputs to subsequent tasks Traditional Software Testing is a Failure!

36. Delphic Oracle Technique Gather estimates from each team member Ask high and low estimators to explain estimates Repeat twice, then use average Three-Point Technique Have team estimate best-case, worst-case, and expected-case Use expected-case Variances between best and worst useful to gauge estimate accuracy Tapping the Team’s Wisdom for Estimation These techniques can be combined (which is called the Wideband technique) and also used with other sources of estimation like historical project data, expert advice, and rules of thumb Traditional Software Testing is a Failure!

37. Assign Dependencies Identify tasks with no predecessors Identify tasks dependent only on previously-identified tasks Repeat step two until dependencies all identified Small projects: Can use project management tools (e.g., MS Project) Medium to Large projects: Use index cards or sticky notes and whiteboard Analyze Critical Paths A set of dependent tasks where delay in any task delays the project end Near-critical path exist where significant delays will delay project end What affects phase entry and exit criteria? During test execution: How many test passes, releases, and cycles? (More later) External dependencies: a frequent cause of delay Understanding Dependencies Traditional Software Testing is a Failure!

38. People Test engineers and test technicians, contractors and employees, outside test resources Using less-skilled people increases task effort, duration People’s skill with a given task or tool determines estimate accuracy Test environments Hardware, software, networks, facilities, etc. Especially important to include expensive or long-lead-time items like large servers, test tools, lab space Test tools Custom Commercial off-the-shelf Assigning Resources • Assume two people can get the job done in half the time • Overload tools or the test environment • Forget setup and support tasks for environments and tools Traditional Software Testing is a Failure!

39. How Can We Predict Test Execution Time? • When will you be done executing the tests? • Part of the answer is when you’ll have run all the planned tests once • Total estimated test time (sum for all planned tests) • Total person-hours of tester time available per week • Time spent testing by each tester • The other part of the answer is when you’ll have found the important bugs and confirmed the fixes • Estimate total bugs to find, bug find rate, bug fix rate, and closure period (time from find to close) for bugs • Historical data really helps • Formal defect removal models are even more accurate Traditional Software Testing is a Failure!

40. How Long to Run the Tests? • It depends a lot on how you run tests • Scripted vs. exploratory • Regression testing strategy (repeat tests or just run once?) • What I often do • Plan for consistent test cycles (tests run per test release) and passes (running each test once) • Realize that buggy deliverables and uninstallable builds slow test execution…and plan accordingly • Try to understand the amount of confirmation testing, as a large number of bugs leads to lots of confirmation testing • Check number of cycles with bug prediction • Testers spend less than 100% of their time testing • E-mail, meetings, reviewing bugs and tests, etc. • I plan six hours of testing in a nine-to-ten hour day (contractor) • Four hours of testing in an eight hour day is common (employee) Traditional Software Testing is a Failure!

41. Traditional Software Testing is a Failure!

42. How Long to Get the Bugs Out? • Using historical data and test cycle model, you can develop a simple model for bugs • Predicting the total number bugs is hard • Subject to many factors and nonlinear effects • Common techniques: bugs per developer-day, KSLOC, or function point (FP) • Project size is usually the easiest factor to estimate • Bug injection rates, fix rates, and closure periods are beyond the test team’s control • You can predict, but document assumptions • The more historical data you have for similar (size, staff, technology, etc.) projects, the more accurate and confident you can be Traditional Software Testing is a Failure!

43. This chart was created in about one hour using historical data from a couple projects and some simplified models of bug find/fix rates. The more data you have, the more accurate your model can be, and the more you can predict the accuracy. Traditional Software Testing is a Failure!

44. Objective: Realism, Not Optimism • Effort, dependencies, resources accurately forecast • Equal likelihood for each task to be early as late • Best-case and worst-case scenarios known • Plan allows for corrections to early estimates • Risks identified and mitigated, especially… • Gaps in skills • Risky technology • Logistical issues and other tight critical paths • Excessive optimism “[Optimism is the] false assumption that… all will go well, i.e., that each task will take only as long as it ‘ought’ to take.” --Fred Brooks, in The Mythical Man-month, 1975. Traditional Software Testing is a Failure!

45. How About those “Rules of Thumb”? • Various rules of thumb exist for effort, team size, etc., e.g., developer-to-tester ratios, Jones’ Estimating Software Costs • How accurate are these rules? • Jones’ test development rule: ~ 1 hr/case • Across four projects, I averaged 9 hrs./case: almost an order of magnitude difference • Imagine if I’d used Jones’ rule on IA project • Even on my own projects, lots of variation • However, historical data from previous projects with similar teams, technologies, and techniques is useful for estimation • You can adjust data from dissimilar projects if you understand the influential factors Person-hours per test case developed varied widely across four projects. Factors included test precision, number of conditions per case, automation, custom tool development, test data needs, and testware re-use Traditional Software Testing is a Failure!

46. Refining the Estimate • For each key deliverable, look at size and complexity • Small, medium, large and simple, moderate, complex • Do you have data from past projects that can help you confirm the effort and time to develop deliverables of similar size and complexity? • Iterate until it works • Must fit overall project schedule… • …but not disconnected from reality • Check “bottom-up” (work-breakdown-structure) estimates against “top-down” rules of thumb, industry benchmarks, models relating metrics like FP or KSLOC to test size • Estimates will change (by as much as 200%) as the project proceeds (knowledge is gained, requirements evolve) Traditional Software Testing is a Failure!

47. Overview Bounds Scope Definitions Setting Quality Risks Schedule of Milestones Transitions Entry Criteria Continuation Criteria Exit Criteria Test Configurations and Environments Test Development Test Execution Key Participants Test Case and Bug Tracking Bug Isolation and Classification Release Management Test Cycles Test Hours Risks and Contingencies Change History Referenced Documents Frequently Asked Questions A Lightweight Test Plan Template You can find this template, a case study using it, and three other test plan templates at www.rexblackconsulting.com Traditional Software Testing is a Failure!

48. Test Execution (Test Release Management) • Release schedule (i.e., weekly? daily? hourly?) • Update apply (process to install new build) • Update unapply (process to remove bad build) • Build naming (revision level); e.g., X.01.017 • Interrogation (process to determine rev. level) • Synchronizing with databases, other systems, etc. • Roles and responsibilitiesfor eachstep Traditional Software Testing is a Failure!

49. Test Execution • Tracking and managing test cases and bugs • Define your approach for keeping track of tests and bugs • Bug isolation and characterization • What is the extent of research done on each bug reported? • What is the information to be gathered on each bug reported? • Test cycles • What tests against what builds • Test selection and prioritization process • Test hours • When will testing be taking place • Will you use multiple shifts Traditional Software Testing is a Failure!

50. Other Issues for the Plan • Risks and contingencies • What could go wrong to delay/invalidate plan effort? • How would you prevent or manage these risks? • Change history • Track revisions to the plan, including why you made the changes • Referenced documents • Requirements, design specs, other project docs • Corporate or technical standards, processes, etc. • Frequently Asked Questions (if applicable) Traditional Software Testing is a Failure!