Risk Analysis for Testing

1. Risk Analysis for Testing • Based on Chapter 9 of Text • Based on the article “ A Test Manager’s Guide to Risks Analysis and Management” by Rex Black published in Software Quality Professional, Vol. 7, No1, pp 22-30. • Will follow the flow of the article more than chapter 9.

2. The question is - - - - • What are you testing? • What are areas of the system do you think is high risk? • How do you know which ones are the most important one? • How do you know which ones are high risk? • What is a risk ? --- • it is A Potential Undesirable Outcome • So for system’s quality, a potential undesirable outcome is when the system fails to meet its expected behavior. • The higher this potential of undesirable outcome is, the more we want it not to occur. • We test to find these high potentially undesirable outcomes and get them fixed before release • We test to show that the areas tested do not harbor these high potentially undesirable outcomes.

3. Quality Risks • For most systems, it is not probable to test all possibilities; so we know there are errors shipped. (There are risks!) • Two major concerns about risks are • How likely is it that certain risks (the potential problems) exist in the system? • What is the impact of the risk to: • Customer/user environment • Support personnel workload • Future sales of the product

4. Quality Risk Analysis • In quality risk analysis for testing we perform 3 steps: • Identify the risks (in the system) • Analyze the risks • Prioritize the risks

5. 5 Different Techniques for Quality Risk Analysis • All 5 techniques involve the three steps of identifying, analyzing and prioritizing based on a variety of sources for data and inputs. • Informal Analysis • ISO 9126 Based Analysis • Cost of Exposure Analysis • Failure Mode and Effect Analysis • Hazard Analysis • One may also use combinations of these

6. Informal Quality Analysis • Identified by a “cross-section” of people • Requirements analyst • Designers • Coders • Testers • Integration and Builders • Support personnel • Possibly users/customers • Risks are identification, analysis and prioritization based on: • Past experiences • Available data from inspections/reviews of requirements and design • Data from early testing results • Mission-critical and safety-critical system tests would need to “formalize” the analysis of the past inspection/review results and tests results

7. ISO 9126 Quality Risk Analysis Functionality Efficiency ISO 9126 • ISO 9126 defines on 6 Characteristics of Quality. • Functionality: suitability, accurateness, interoperability, compliance, security • Reliability: maturity, fault tolerance, recoverability • Usability: understandability, learnability, operability • Efficiency: time behavior, resource behavior • Maintainability: analyzability, changeability, stability, testability • Portability: adaptability, installability, conformance, replaceability • These 6 characteristics form a convenient list of prompters for the identification, analysis , and prioritization of risks (potential problems) • Good to follow an industry standard, especially for large software systems which require a broad coverage of testing. Reliability Maintainability Usability Portability

8. Cost of Exposure Analysis • Based on two factors: • The probability of the risk turning into reality or the probability that there is a problem • The cost of loss if there is a problem • The cost of loss may be the viewed as cost of impact tousers in (work hours delayed, material loss, etc.) • The cost of loss may also be viewed from a reverse perspective --- cost of fixing what the problem caused. • Based on the probability of occurrence and the cost of impact, one can prioritize the test cases. • Consider three risks • Risk 1: prob. of occurrence (.3) x cost ($20k) = $6k • Risk 2: prob. of occurrence (.25) x cost ($10k) = $2.5k • Risk 3: prob. of occurrence (.1) x cost ($100k) = $10k • In this case, you will prioritize in the order: • Risk 3, • Risk 1, and • Risk 2 • Both factors are hard to estimate for normal testers; need experiences.

9. Failure Mode/Effect Analysis (FMEA) • This methodology was originally developed by the US military (MIL-P-1629) for detecting and determining the effect of system and equipment failures, where failures were classified according to their impact to: • a) the mission success and • b) personnel/equipment safety. • The steps are ( for each mechanical-electrical component ): • Identify potential design and process failures • Find the effect of the failures • Find root cause of failures • Prioritize the recommended action based on risk priority number which is computed by (probability of risk occurrence x severity of the failure x probability of detection of the problem) --- • Recommend actions • For software ( for each function): • Same steps apply, but the reasons for failure are quite different from the physical component . • There is no aging and wear/tear • Mechanical stress needs to be replaced by performance stress similar to Cost of Exposure Analysis

10. Hazard Analysis • This is similar to Failure Mode and Effect Analysis except we look at the effects first and work backward and trace at potential cause(s). • In order to list the effects (or anticipateproblems) , the testers will need to be fairly experienced in the application discipline and software technology

11. Ranking of Problems (Hutcheson example) • Rank = 1 (Highest priority) • Failure would be critical • Required item by service level agreement (SLA) • Rank = 2 (High Priority) • Failure would be unacceptable • Failure would border the SLS limit • Rank = 3 (Medium priority) • Failure would be survivable • Failure would not endanger the SLA • Rank = 4 (Low priority) • Failure would be trivial and has small impact • Failure would not be applicable to SLA

12. Risk Ranking Criteria (Hutcheson Example & Yours) • (Hutcheson) Similar in concept to the ISO 9126’s 6 characteristics • Is this risk a required item: (1= must have - - - 4 = nice to have) • What is the severity of risk: ( 1 = most serious - - - 4 = least serious) • What’s the probability of occurrence: (1= most likely - - - 4= least likely) • Cost of fixing problem if it occurs: (1 = most expensive - - - 4 = least exp.) • What is the failure visibility: (1 = most visible - - - 4 least visible) • How tolerable will users be: (1 = most tolerable - - - 4 least tolerable) • How likely will people succeed: (1= fail - - - 4 = succeed) Which from the above would you keep and what new ones would you add? e.g.: - How difficult is it to test for or discover this problem (1 hard - - - 4 =easy)