420 likes | 432 Views
Software Quality Assurance. Introduction. What is software quality? How can it be measured? How can it be measured before the software is delivered? Some key quality factors Some measurable indicators of software quality. Introduction. Think of an everyday object e.g. a chair
E N D
Introduction • What is software quality? • How can it be measured? • How can it be measured before the software is delivered? • Some key quality factors • Some measurable indicators of software quality
Introduction • Think of an everyday object • e.g. a chair • How would you measure it’s “quality”? • construction quality? (e.g. strength of the joints,…) • aesthetic value? (e.g. elegance,…) • fit for purpose? (e.g. comfortable,…) • All quality measures are relative • there is no absolute scale • we can say A is better than B but it is usually hard to say how much better • For software: • construction quality(建造的品質)? • software is not manufactured • aesthetic value (美學上的價值)? • but most of the software is invisible • aesthetic value matters for the user interface, but is only a marginal concern • fit for purpose? • Need to understand the purpose
McCall’s Quality Factors Maintainability Flexibility Testability Portability Reusability Interoperability revision transition operation Correctness reliability usability integrity efficiency
Measuring Quality examples... The Quality Concepts (abstract notions of quality properties) reliability complexity usability information flow between modules? time taken to learn how to use? Measurable Quantities (define some metrics) mean time to failure? minutes taken for some user task??? Counts taken from Design Representations (realization of the metrics) run it and count crashes per hour??? count procedure calls???
SQA: What is SQA? • SQA is the process of assuring people that every effort has been made to ensure that software products have the desired level of reliability, maintainability, usability, and salability. • SQA是一種執行軟體評估與衡量的活動 (Baker and Fisher) • Set of systematic activities providing evidence of the ability of the software process to produce a software product that is fit to use (G. Schulmeyer and J. McManus, Software Quality Handbook, Prentice Hall, 1998.)
SQA and CMMI • Software (Process and Product ) quality assurance is the key process in Level 2, Managed. • The CMMI lists four goals that must be achieved to satisfy this key process area: • Goal 1: Objectively Evaluate Processes and Work Products • Adherence of the performed process and associated work products and services to applicable process descriptions, standards, and procedures is objectively evaluated. • Goal 2: Provide Objective Insight • Noncompliance issues are objectively tracked and communicated, and resolution is ensured. • Goal 3: Institutionalize a Managed Process • Goal 4: Institutionalize a Defined Process
Software Quality Assurance • How do you assure the quality of your software? • Good processes • Good documentation and artifacts • Accountability (可說明性) • Learn and improve
The Objectives of SQA • Monitoring processes and products throughout the software development lifecycle to ensure the quality of the delivered product(s) • Monitoring the processes • Provides management with objective feedback regarding process compliance to approved plans, procedures, standards, and analyses • Monitoring the products • Focus on the quality of product within each phase of the SDLC • e.g., requirements, test plan, architecture, etc. • Objective: identify and remove defects throughout the lifecycle, as early as possible
SQA: An SQA Program • Minimizing number of defects in delivered s/w • Creating mechanisms for controlling software development and maintenance so that costs and schedules can be met • Making certain that the delivered product can be used in its intended marketplace • Improving the quality of future products
Software defects • Mistakes made at any point in the software process • Requirements, design, coding, maintenance • Consequences • Inconvenience, loss of service, financial loss, equipment damage, injury, death
The truth of defects 1. The later in the life cycle that an error is detected the more expensive it is to repair. 2. Errors remain latent and are not detected until well after the stage at which they are made. • 54% of errors detected after coding and unit testing. • 45% of these errors were requirements and design errors. 3. There are numerous requirements errors. • Estimates indicate that 56% of all errors are errors during the requirements stage.
Relative Cost to Repair based on when it was found • Requirements - 1 time • Design - 3-6 times • Code - 10 times • Unit test - 15-40 times • System test - 30-70 times • Field operation - 40-1000 times
When should quality assurance be done? At every stage in the software process
Testing • Process of exercising or evaluating a system by manual or automatic means to verify that it satisfies specified requirements or to identity differences between expected and actual results.
Testing • Begins as soon as requirements and specifications are written • >= 40% of total project effort • Constructive activity • Includes formal reviews and walkthroughs
Testing Objectives • Testing is a process of executing a program with the intent of finding an error. • A good test case is one that has a high probability of finding an as yet undiscovered error. • A successful test is one that uncovers an as yet undiscovered error.
Testing Limitations • Cannot determine whether software meets user’s needs; only whether it appears to conform to user’s requirements • Cannot show that a system is free of defects; only that it contains defects • Does not correct defects; only finds defects
The Process of SQA 功能與完整性 需求評估 定義品質需求 制定SQA計畫 需求分析 評估客戶滿意需求程度 feedback 設計評估 設計 在需求分析階段完成 feedback 測試評估 測試 系統完整性與一致性 measurement 執行效率與正確性
SQA Planning • IEEE Std 730-2002 • Standard for Software Quality Assurance Plans • 12 pages • IEEE Guide for Software Quality Assurance Planning • draft P730.2 • 87 pages
SQA Planning • IEEE SQAP • Purpose (Section 1 of the SQAP) • Reference documents (Section 2 of the SQAP) • Management (Section 3 of the SQAP) • Documentation (Section 4 of the SQAP) • Standards, practices, conventions, and metrics (Section 5 of the SQAP) • Reviews and audits (Section 6 of the SQAP) • Test (Section 7 of the SQAP) • Problem reporting and corrective action (Section 8 of the SQAP) • Tools, techniques, and methodologies (Section 9 of the SQAP) • Code control (Section 10 of the SQAP) • Media control (Section 11 of the SQAP) • Supplier control (Section 12 of the SQAP) • Records collection, maintenance, and retention (Section 13 of the SQAP) • Training (Section 14 of the SQAP) • Risk management (Section 15 of the SQAP)
Contents of SQA Plan (sect 1&2) • Purpose • list software covered • state portion of software life cycle covered • Reference Documents • complete list of documents referenced elsewhere
Sect 3 - Management • organization - depict structure of org. • responsibilities • tasks • tasks to be performed • relationship between tasks and checkpoints • sequence of tasks • responsibilities • of each organizational unit
Sect 4 - Documentation • identifyrequired documents • state how documents will be evaluated • minimum documents • SRS - Software Requirements Specification • SDD - Software Design Description • SVVP – S. Verification and Validation Plan • SVVR - S. Verification and Validation Report • User documentation - manual, guide • SCMP – S. Configuration Management Plan
Verification and Validation • 驗證(Verification): "我們是否正確的開發了產品?" • 軟體應該與規格相符 • 確認(Validation): "我們是否開發了對的產品?" • 軟體應該執行使用者真實的需求 • 驗證與確認著重於讓程式的缺失得以浮現 • 除錯著重於找出缺失並加以改正
S. Verification and Validation Plan • 測試規劃與建立測試程序的標準有關,而非著重於描述產品如何測試
Sect 5- Standards, Practices, Conventions and Metrics • Identify S,P,C,and M to be applied • How compliance is to be monitored and assured • Minimum • documentation standards, logic structure standards, coding standards, testing standards • selected SQA product and process metrics
Sect 6 - Reviews and Audits • purpose • define what reviews/audits will be done • how they will be accomplished • what further actions are required • Minimum • Software Requirements Reviews • Preliminary Design Review • evaluate technical adequacy of top-level design
Min Set of Reviews/Audits (cont) • Detailed Design Review • acceptability of detailed designs • Software Verification and Validation Plan Review • adequacy of planned verification and validation • Functional Audit • all requirements in SRS have been met • Physical Audit • software and documents are consistent and ready • In-Process Audit • Managerial Reviews
Sect 7 - Test • All tests that are not included in SVVP
Sect 8 - Problem Reporting • Practices and Procedures for reporting, tracking, and resolving problems • Organizational responsibilities
Sect 9 - Tools, Techniques and Methodologies • identify the special software tools, techniques and methodologies • purpose • describe use
SQA: 6 SIGMA QUALITY • Sigma = “Standard Deviation” • Typical software has 3 to 4 defects per KLOC • 6 Sigma = 3 to 4 defects per million lines of code • Average companies accept 99.98% quality = 4S • 6 Sigma = 99.9999998% level of quality
SQA: 6 SIGMA QUALITY • Quality Improvements: • 3Sigma to 4Sigma = 10 fold • 4Sigma to 5Ssigma = 30 fold • 5Sigma to 6Sigma = 70 fold • Best-in-Class companies in some industries operate at 6Sigma (Airline = 6.4Sigma; 2.5M:1) • Software organizations need to assess this
SQA: Quality Software Process People Management Discipline SQA
SQA: Pursuing SQA What organizations are doing • Nothing (42%) • Slogans(口號) - “Quality is Job One!” (4%) • Improved testing (24%) • Focus on defect prevention (20%) • Process Improvements (9%) • Other... (1%)
SQA: Software Reliability (MTBF) Putnam’s Software Reliability Model Operational Capability N U M B E R E R R O R S O F 0 0 T I M E
Purpose includes improvement Quality philosophy Eliminate mass inspections Award business based on more than price Continuous improvement Institute OJT(On the job training) Institute Leadership SQA: Pursuing SQA the Deming Way • Drive out fear • Break down barriers • Eliminate slogans • Eliminate numerical quotas & goals • Remove barriers to pride of workmanship • Institute education and self-improvement • Get everyone involved
SQA: Strategy by Yourdon If management or customer says... • Speed up testing...just say NO! • Don’t worry about a few bugs...just say NO! • We’ll pin down the specs later...just say NO! • Don’t worry, its just a beta version...just say NO! • I don’t care if there are bugs, get it out the door...just say NO!
Summary • Software quality generally means fitness for purpose • need to know what that purpose is… • …what functions must it perform • …what other properties must it have (e.g. modifiability, reliability, usability…) • Not all quality attributes can be measured during design • because quality is not an attribute of software in isolation • but we can look for predictors • Reliability, efficiency, maintainability, usability • are usually the four most important quality factors • …although different authors give different lists • Modularity is often a good predictor of quality • measure it by looking at cohesion and coupling