Software Quality and Management: How the World’s Most Powerful Software Makers do it

1. Software Quality and Management: How the World’s Most Powerful Software Makers do it Dien D. Phan Professor BCIS Department St. Cloud State University St. Cloud, MN 56301. Information Systems Management, Vol. 18, No.1, 2001.

2. Software Quality Concepts Weinberg: Quality is relative. What is quality to one person may be lack of quality to another IEEE: Quality is the probability of failure-free operation of software. Fault is the cause of failure in software: • A failure is the departure of the external results of a program operation from requirements • A fault is the defect in the program that when executed under particular conditions, causes one or more failures.

3. Capability Maturity Integration Model (CMMI) • Level 1: Initial • Level 2: Repeatable. Config mgmt, Qual Assurance, Subcontract mgmt, Proj tracking, Proj planning, requirements mgt. • Level 3: Defined. Peer rev., Coordination, Prod. Engineering, Integr. Mgmt, Training, Org. process def., Org. process focus. • Level 4: Quantitatively Managed. Softw. Quality Mgmt., Quant. process mgmt. • Level 5: Optimized. Process change mgmt, Technology change mgmt, Defect prevention.

4. Impacts of CMM on Software Quality • Putnam (1994): For organizations that moved from level 1 to 3: • Time reduced by a factor of 1.7 • Peak staff reduced by a factor of 3.2 • Efforts reduced by a factor of 5.7 • Herbslef et al (1997):For organizations that moved from level 1 to 3: • 5 to 1 ROI at Hughes Aircraft • 7.7 to 1 ROI, 75% decrease on rework, 190% increase in productivity, and decrease in error rates per KLOC from 17.2 to 4.0 at Raytheon

5. Common Quality Measurements • Mean time between failure • Mean time to repair • Defect rate by time • Defect rate by size • Defect backlog size • Number of clean LOC passed first QA attempt • Cumulative defects per version • Timeliness to fix defect • Customer satisfaction level • Rate of defects per unit of software size discovered after first year of delivery

6. Quality Control Processes at IBM -OS/400 Development • During development: • Scope control: Requirements and resources control • Reviews: HiLvl Design, LoLvl Design, and Code reviews • Customer involvement • Tests and defect removal activities • Development and quality control tools: IDSS, DCR, PTR, and quality models • After delivery: Quality control for services, administration, and documentation.

7. Quantitative Model

8. Causes of bad fixes • Not thorough enough testing • Incorrect instructions to solve problems • Not all steps tested • Incomplete test bucket (data, programs, scenarios) • Inadequate reviews • Inability to test user hardware environment • Insufficient testing of special installations

9. Results • Defect rate • Level of user satisfaction and complaints • Numbers of service calls decreased by 50% • Cost of service calls decreased by 20-40% • System installation time reduced to 2 weeks • Documentation errors reduced by 50%

10. More Recent Studies at IBM • Kaplan et al. (1995): Secret of Software Quality: 40 Innovations from IBM • IBM Santa Theresa lab: defects decreased by 46%, service costs decreased by 20%, revenues per employees increased by 58%, and customer satisfaction increased by 14% • Mixed results on the cleanroom process

11. The Cleanroom Process Rather than developing a software and then working to remove defects, the cleanroom approach demands the discipline required to eliminate defects in specification and design then implement in a “clean” maner. The reasons that it is not widely used today are: • It’s too theoretical, mathematical, and radical • It abandons unit testing • It uses statistical (randomized testing) and thus cannot effectively test weak spots in software • It requires a high level of maturity in software development process

12. Quality Management at Microsoft Agile process using small teams: team decides how the work get done Process Model

13. Daily (nightly) build/synchcronization and stabilization process • Check out source code to work ->private copy • Tool: Diff to find the difference between master source since the time code is checked out • Submit private copy (merging) to “quick test” before daily deadline (daily/nightly-build, daily synchronization and stabilization) • Problem: “Breaking the build” • Solution: Use Build-Master in daily-build

14. Microsoft’s Quality Strategies • Grow rather than design and building software, • Change or replace 50% of code after each release, • Change in design must be justified by features that are at least twice as good, • Build multiple version simultaneously, • Develop software at single site, • Continuously test the products as software grows, and • Use Adaptive Software Development (ASD) process to build web-based software for clients

15. IBM, Microsoft Quality Capabilities measured by SEI’s CMM Legend: Y= Yes - = Not defined

16. Lessons learned • Development Processes • IBM focuses on requirements, Microsoft focuses on vision • IBM processes are development oriented, less trade off resources, features, and schedule. Microsoft’s are customers oriented, more trade-offs allowed at each milestones. • Microsoft half-life (50% replacement) reduced software reuse • Microsoft teams enjoy greater freedom in the planning and development phases

17. Lessons learned • CMMI: • IBM is at level 4 or 5 • Microsoft is at level 3 • Are there any Silver Bullets?

18. New Software Technology

19. Key success factors • Feedback, On-line monitoring, • Project management techniques, quality control standards, quality processes, • Team members are responsible of quality of components being built, • Test early, and test frequently: frequent synchronization, • Software reuse, • Quantitative Defect Removal Models, • Test cases should be ready before development. • Version Control, Configuration Management.

20. Conclusion • IBM relies on both process and quantitative defect models: traditional software development oriented • Microsoft relies on its process model: agile development oriented. • Agile methodology is popular in Web projects