Software Quality Engineering Roadmap

1. Software Quality Engineering Roadmap

2. Business Objectives • Maximize Revenue • Minimize Costs

3. Maximize Revenue • Improve market share: • Improve product quality • Increase customer satisfaction • Better marketing • Improve cycletime • Increase product value: • Increase product quality • Define value proposition and match product capabilities and quality to customer needs

4. Improve Market Share • Improve product quality • Increase customer satisfaction • Identify and address key customer concerns • Continuous improvement process • Practices for measuring and improving satisfaction • Expectation management practices • Better marketing • Match product to marketing needs • Improved requirements practices • Practices for measuring and improving marketing effectiveness • Improve cycletime • Cycletime measurement and continuous improvement • Increase product value • Define value proposition and match product capabilities and quality to customer needs

5. Minimize Costs – Improve Productivity • Productivity measurement and continuous improvement • Practices for improving productivity: • Tools usage • Technology selection • Reuse and common architecture development • Streamlining and automation • Workflow, templates, tools • Task elimination • Developer capability improvement through training

6. Improve Project Management– Activity Management • Practices: planning, estimation, progress tracking • Metrics: estimation accuracy, progress metrics • Estimation practices: • Maintain database of past projects • Estimate based on history of comparable projects • COCOMO (constructive cost model) provides questionnaire and formula for estimation based on organizational and project profile

7. Improve Project Management – Risk Management • Identify and track risks • Metrics examples: • Percentage of unidentified high-impact risks among those that actually occurred • Percentage of unmitigated high-impact risks among those that actually occurred • Percentage of effort spent on risk management and mitigation

8. Improve Project Management – People Management • Practices to measure and improve employee satisfaction • Teaming practices and effectiveness: • Metrics: Turnover %

9. Improve Project Management: In-Process Metrics • Slippage, progress • Inspection effectiveness (evolve as project proceeds) • Defect rates • Test progress • Reliability growth curves

10. Product Quality- Features • Requirements quality & effectiveness • Requirements prioritization & release planning

11. Product Quality - Performance • Defining performance objectives • Practices: designing for performance, performance analysis, performance testing • Primarily measurements and not metrics

12. Product Quality - Dependability • Reliability engineering: objective setting, practices and metrics • Availability: objective setting, practices and metrics • Security: • Practices for identifying threats (defining objectives) • Security engineering practices • Security testing: threat injection • Safety (similar to security and reliability)

13. Product Quality - Usability • Specifying usability objectives • Usability engineering practices • Usability measurements: surveys, avg. learning time, keystrokes for frequent operations etc. • Usability metrics: SUMI

14. Product Quality - Evolvability • Evolvability goals • Practices for improving evolvability

15. Development Effectiveness • Requirements • Design • Testing & Inspections • Configuration Management • Quality Engineering Creative Research Systems

16. Development Effectiveness - Requirements • Requirements effectiveness practices • Elicitation practices, prototyping, requirements analysis practices, traceability • Requirements metrics • Activity metrics, also requirements defect injection rate

17. Development Effectiveness - Design • Practices for design effectiveness • Modularity practices, design patterns for achieving various quality attributes, design diagramming, design analysis techniques • Design metrics • Activity metrics, injection rates, product quality metrics, reuse percentage

18. Development Effectiveness – Testing & Inspections • Practices for testing effectiveness • Multistage testing, test strategy & planning, test case identification through equiv classes, automation of test generation, randomized parameters, automated testing • Reliability engineering • Test metrics • DRE charts, defect rates and densities, reliability … • Inspection practices and metrics

19. Development Effectiveness – Configuration Management • Configuration management practices • Metrics for configuration management effectiveness • Configuration management defect injection rate

20. Development Effectiveness – Quality Engineering Practices • Processes • Process Improvement / Continuous Improvement • Assessments

21. Quality Engineering Process • Initial process definition and tailoring • Process compliance metrics (certifications, assessments) • Process improvement • Improve effectiveness of practices • Improve breadth of process & practices (increased “maturity level”, CMMI-style) • Minimize volume of process, maximize effectiveness (agile development) • Define set of processes that address the most frequent causes of defects and achieve more of the business objectives: maximizing cost-benefit ratio for process practices • Requires deep understanding of process/quality relationships, high developer maturity and competence, and extensive customization to situational needs • A stage of evolution beyond simply adding process breadth and doing improvement.

22. Continuous Improvement • Quality tools and causal analysis, defect elimination • Identify problem area • Use fishbone diagrams to brainstorm and capture possible causes • Use results to design data collection scheme and gather data about actual causes of problem - This is an extremely important step. The fishbone itself is just a set of guesses. It does not tell you which causes need addressing • Use pareto charts to identify most significant causes of the problem • Use defect elimination techniques (checklists, templates etc) to try to prevent the problem in the future • Repeat continuously or until objective is achieved • Effectiveness measured through trend lines for various process metrics

23. Metrics Concepts • Defining Metrics • GQM paradigm • Metrics must have significant value: High impact (value provided to user of metric) compared to effort spent in collecting data. • Meaningfulness of metrics • Reliability • Validity • Importance of interpretation • Gathering Measurements

24. Goal-Question-Metric (GQM) Paradigm • What are the objectives for the metric? • Should identify the purpose of the metric – who wants to do what with it? • Includes “viewpoint”: whose viewpoint are we interested in for this? E.g. developer, project manager, organization manager, quality engineer, external assessor, customer. • What specific questions do we want answered? • Identify specific aspects of the problem that we care about • Includes operational definitions of the quantities we want to measure • Define measurements and metrics • What data will need to be gathered? Who will gather it and how? • What metric will be computed from the measurements? • Metrics must provide a basis for action • Metrics must have an interpretation: given a value, how will this be interpreted? • Metrics must have significant value: high impact (value provided to user of metric) compared to effort spent in collecting data • Meaningfulness of metrics • Reliability • Validity

25. Importance of Interpretation • Understanding the intrinsic limitations of the metric • Understanding the limitations of the data collection process • Understanding situational factors that provide context to the numbers

26. Gathering Measurements • Minimizing measurement effort: • Effort spent on providing data is “wasted” effort from project perspective – needs to be minimized! • Often, person who has to put in effort is different from the one who gets immediate value! Higher the effort, less the motivation for the provider to supply good data • Need for tools: • Tools improve accuracy of data, especially if collection is automated • May sometimes need to provide for manual adjustments to tool-generated data • Tools can automate processing of data and generate useful views (graphs) directly • Motivating measurement: • Set high bar on value: only ask for measurements that provide considerable value • Make value visible to people providing measurements (clear how the gathered data will be used) • Try to provide value to those providing measurements: • e.g. customer concerns actually get addressed • e.g. developers get useful feedback that helps them avoid problems and reduce effort

27. Metrics Perspective • Metrics are a small part of the overall quality engineering process. • Define objectives for metrics. • Metrics interpretation is key. • Actions and improvements based on metrics interpretation is the primary value achieved in the use of metrics.