Optimize Quality for Business Outcomes Charlie Li, Andreas Golze, Shel Prince

1. Optimize Quality for Business OutcomesCharlie Li, Andreas Golze, Shel Prince

2. First Published Book from Mercury!

3. Understanding Defects Introduction of Defects • The majority of defects are introduced during the requirements and design phase • However, the majority of defects are actually detected during user acceptance testing and in production Detection of Defects Source: NIST 2002 RTI Project 7007.011

4. Defect Correction Costs 1000x in Production! This industry average is used as a baseline for arriving at cost savings Industry References: 3 B. Boehm and V. Basili, "Software Defect Reduction Top 10 List," IEEE Computer, IEEE Computer Society, Vol. 34, No. 1, January 2001, pp. 135-137.

5. TraditionalCost QualityEffortandCost Risk Mitigated Approach Time A Risk-Mitigated Quality Approach • Align testing with business requirements • Mitigate implementation risk with effective quality process Test Project Prep Blueprint Design Build Go-Live Support Assess Risk Test Test Manage Quality Process

6. The Missing link • Who’s testing for the business outcomes? • How do you structure requirements? • Requirements -> behavioral model -> Test Cases • What to test? How to Test? How much to test? • What test strategies to use and how can automation be leveraged? • What do you measure, assess, and continuously improve?

7. Behavioral Modeling

8. What is it? • A graphical representation… • ...of the functional behavior… • …of the software article.

9. What does it buy us? • Allows the behavior to be specified at any level of detail. • Allows different people to work on different sections at the same time. • Limited “vocabulary” enforces a high level of consistency in the models. • Allows understanding (and review) by people with a wide variety of experience.

10. Vocabulary Beginning or End Process Decision Case Result Displayed Result Representative Sample Manual Action Straight Line Connector Note Curved Line Connector

11. How do we do it? There are only two questions -- • What does it do? • “Do” in the active sense • What influences it?

12. Sample Problem 1 • Externals only • Example – Bigger: Two numbers in One number out +1 if 1st number bigger than 2nd -1 if 1st number smaller than 2nd 0 if the numbers are equal

15. Sample Problem 2 • FAST • The Functionally Advanced Sidewalk Teller • Deposit • Withdraw • Transfer • Check Balance

24. Attributes of Good Test Suites • Effective • Efficient

28. Test Case Design Sample

29. Business Impact Testing

30. The Specification

31. The User Acceptance Test

32. Improvements

33. The Retest

34. Risk Ratio Analysis Design Build Analyze Test Deploy 1 R I S K R A T I O 0 Test Activities Tracking Risk Ratio • Baseline the risk ratio and set the risk value to 1; current risk/initial risk • Adding new requirements post-analysis increases the risk ratio; enables risk-based discussions • Risk analysis acts as a verification of the requirements • Risk hopefully reduces over the lifecycle

35. Test Requirements Structure Organization of Test Requirements Applications support business activities/processes and functions Structure is uniform and enforced, and naming convention is informative Example 1 Example 2

36. Business Impact Driven Test Strategy Assess the business impact of each activity Assess the Probability for failure Determine the Risk based on Impact and probability Analysing Probability Determining the Risk

37. The Risk determines the Test Strategy Procedure for High Risk A Procedure for Medium Risk B Procedure for Low Risk C Systematic testing using business component testing strategy, and root-cause analysis Procedure Approach Automated: 30% Manual: 70% Systematic testing with or without business component testing strategy, and root-cause analysis Procedure Approach Automated: 20% Manual: 80% Procedure Systematic or Ad-hoc testing Approach Automated: 5% Manual: 95%

38. Functional Complexity determines Test effort Structuring Test Requirements

39. Prioritizing Automation Effort Prioritization Strategy Automation ROI Matrix • Assuming test cases for all risks and complexities are in-scope for each test cycle. • Testing Scope • Defined by risk-levels to be tested (ie. Higher risk items tested first) • Prioritization Order for Automation: • Test cases with lowest # of test cycles for ROI • Test cases with highest risk levels first • Test cases with lowest complexity first 1 4 3 5 6 8 2 7 9 - Priority # / Order #

40. Utilize the best testing strategies Start Project Application Available Map Requirements Test Plans Define Use Cases Create Test Lab Run Tests Modify Tests Analyze Tests Run Tests Modify Tests Manual Testing Map Requirements Test Plans Define Use Cases Automate Tests Run Tests Analyze Tests Modify Tests Run Tests Test Automation Customize Pre-DefinedContent Map Requirements Test Plans Define Use Cases Run Tests Analyze Tests Modify Tests Run Tests Test Acceleration Customize Pre-DefinedContent Map Requirements Test Plans Define Use Cases Run Tests Analyze Tests Modify Tests Run Tests Test What Matters!!! Change Impact Testing Time Line

41. Wrapping up

42. 10 SUCCESSFUL Steps to Measure

43. More Book information • Books will be available for purchase by 12/15 from Mercury’s Education site • www.merc-training.com • Send Questions to getanswers@mercury.com

44. Questions?