Software Testing and QA

1. Software Testing and QA Borrowed from:Yongge Wang (yonwang@uncc.edu)

2. What is quality? • In Webster’s Dictionary, quality is defined as “the essential character of something, an inherent or distinguishing character, degree or grade of excellence” • In the computer literature, there are two generally accepted meanings of quality • Quality means “meeting requirements” • Quality is a binary state, that is, it is a quality product or it is not • For example, “is this program correct?” • From the customer’s point of view, is whether or not the product or service does what the customer needs • Another way of wording it is “fit for use” • Customer needs are generally contained in the description of the purpose of the product, typically documented in a customer’s “requirement specification”

3. Binary state quality • The first is that quality means “meeting requirements” • With this definition, quality is a binary state, that is, it is a quality product or it is not. • For example, “is this program correct?”

4. You can’t always get what you want ever Correctness properties are undecidable the halting problem can be embedded in almost every property of interest Property Decision Procedure Pass/Fail Program

5. Why undecidable? Example: Hello assignment • Students write a computer program: • Outputs “Hello!” and stops • Assignment gives full credit for ANY program that outputs “Hello!” and then halts • No credit otherwise • Can a program be written that automatically grades your homework?

6. The “Hello” Assignment • Can I write a program that automatically grades your homework? • Sample program Function GradeHELLO (Program P) Execute P, and store output in output If (output = “Hello!”) Then Return “Pass” Else Return “Fail” EndIf End Function

7. The “Hello” Assignment • Possible glitch • What if your homework program never stops? Program P Do While (1 > 0) Print “Hello!” End While End Program

8. The “Hello” Assignment Program P Do While (1 > 0) Print “Hello!” End While End Program • Student’s program runs forever! • Means the grading program runs forever • Never tells me that the student should fail…

9. The “Hello” Assignment I’m smarter than that: Stop the Program after 1minute Fail the student if the program hasn’t stopped yet

10. The “Hello” Assignment • What if student’s program stops, but takes a long time? Program P Initialize x = 0 Do While (x is not a proof of Fermat’s Last Theorem) x = next mathematical statement in a sequence End While Print “Hello!” End Program

11. The “Hello” Assignment • What if your program stops, but takes a long time? • Grading program might say to fail you • But technically you deserve to Pass • Unless you never reach a statement that is the proof Program P Initialize x = 0 Do While (x is not a proof of Fermat’s Last Theorem) x = next mathematical statement in a sequence End While Print “Hello!” End Program

12. The “Hello” Assignment • This is becoming a cat-and-mouse game… • Could it be that writing a computer program to grade such a simple assignment is impossible? • YES! Moreover we can prove that it is impossible.

13. The Halting problem • Can we design a program, that tells whether other programs ever halt on their own input? • Output of this program, called HALT, when applied to program P, is “halts” if P eventually halts. I.e., HALT(P)=“halts” • Output of this program is “never halts” if P never halts. I.e., HALT(P)=“never halts” • Theorem (Alan Turing 1937): No computer program cansolve the halting problem.

14. Simple unsolvable problem This sentence is false.

15. Proof by Contradiction • Assume the hypothesis to be proven is false, i.e. there is a computer program that can solve the halting problem • Show that this assumption leads to a contradiction

16. Proof by Contradiction • Suppose there was a computer program,called HALT, which solved the halting problem. • We can write a new program that uses HALT and leads to a contradiction: Program NonConformist (Program P) If ( HALT(P) = “never halts” ) Then Halt Else Do While (1 > 0) Print “Hello!” End While EndIf End Program

17. Proof by Contradiction Program NonConformist (Program P) If ( HALT(P) = “never halts” ) Then Halt Else Do While (1 > 0) Print “Hello!” End While EndIf End Program • Does NonConformist(NonConformist) halt? • Note: It calls HALT(NonConformist) • Yes? Means HALT(NonConformist) = never halts • No? That means HALT(NonConformist) = halts • Contradiction: There exists a program (NonConformist) for which the HALT program gives the wrong answer

18. Undecidability • We’ve shown that the Halting Problem is Undecidable – no computer program can ever solve it, no matter how powerful the computer is. • Many other problems are undecidable, too, including: • A software package is dependable? • A software package is correct? • …

19. Facts about testing • Question “does program P obey specification S” is undecidable • Every testing technique embodies some compromise between accuracy and computational cost • Facts: • Inaccuracy is not a limitation of the technique • It is theoretically impossible to devise a completely accurate technique • Every practical technique must sacrifice accuracy in some way

20. Customer’s viewpoint quality • Another definition of quality, from the customer’s point of view, is whether or not the product or service does what the customer needs • Another way of wording it is “fit for use” • Customer needs are generally contained in the description of the purpose of the product, typically documented in a customer’s “requirement specification”

21. Some Terminology • Safe: A safe analysis has no optimistic inaccuracy, i.e., it accepts only correct programs. • Sound: An analysis of a program P with respect to a formula F is sound if the analysis returns true only when the program does satisfy the formula. • Complete: An analysis of a program P with respect to a formula F is complete if the analysis always returns true when the program actually does satisfy the formula.

22. robustness .... correctness reliability safety maintainability reusability .... modularity .... usability Customer needs: Software Qualities Dependabilityproperties External properties (that can be verified) timeliness interoperability .... Process oriented (internal) properties External properties (that can be validated) user-friendliness

23. Resume 4/23

24. Dependability Qualities • Correctness: • A program is correct if it is consistent with its specification • seldom practical for non-trivial systems • Reliability: • likelihood of correct function for some ``unit'' of behavior • relative to a specification and usage profile • statistical approximation to correctness (100% reliable = correct) • Safety: • preventing hazards • Robustness: • acceptable (degraded) behavior under extreme conditions

25. Example of Dependability Qualities • Correctness, Reliability: let traffic pass according to correct pattern and central scheduling • Robustness, Safety: Provide degraded function when possible; never signal conflicting greens.

26. Relation among Dependability Qualities reliable but not correct: failures occur rarely robust but not safe: catastrofic failures can occur robust reliable Correct safe safe but not correct: annoying failures can occur correct but not safe or robust: the specification is inadequate

27. Validation vs. Verification • Validation: Does the software system meets the user's real needs? Are we building the right software? • Verification: Does the software system meets the requirements specifications? Are we building the software right?

28. Validation vs. Verification Formal descriptions Actual Requirements System Verification Validation Includes: - testing - inspections - static analysis Includes: - usability testing - user feedback

29. Verification or validation depend on the specification 1 2 3 4 5 6 7 8 Unverifiable (but validatable) spec: ... if a user press a request button at floor i, an available elevator must arrive at floor i soon... Example: elevator response Verifiable spec: ... if a user press a request button at floor i, an available elevator must arrive at flooriwithin 30 seconds...

30. Validation and Verification Activities validation verification

31. Quality Summary • Most interesting properties are undecidable, thus in general we cannot count on tools that work without human intevention • assessing program qualities comprises two complementary sets of activities: • validation (does the software do what it is supposed to do?) • verification (does the system behave as specificed?) • There is no single technique for all purposes: test designers need to select a suitable blend of techniques

32. Types of testing

33. Impact of the type of software on testing and analysis • Testing and analysis activities will be influenced by the type and characteristic of the system under test: • Different emphasis may be given to the same properties • Different (new) properties may be required • Different (new) testing and analysis techniques may be needed

34. Different emphasis to the same properties Dependability requirements • Differ radically between: • Safety-critical applications • flight control systems have strict safety requirements • telecommunication systems have strict robustness requirements • Mass-market products • dependability is less important than time to market • Vary within the same class of products: • reliability and robustness are • key issues for multi-user operating systems (e.g., UNIX) • less important for single users operating systems (e.g., Windows or MacOS)

35. Different type of software may require different properties • Timing properties • deadline satisfaction is a key issue for real time systems, but can be irrelevant for other systems • performance is important for many applications, but not the main issue for hard-real-time systems • Synchronization properties • absence of deadlock is important for concurrent or distributed systems, not an issue for other systems • External properties • user friendliness is an issue for GUI, irrelevant for embedded controllers

36. Different properties require different testing and analysis techniques • Performance can be analyzed using statistical techniques, but deadline satisfaction requires exact computation of execution times • Reliability can be checked with statistical based testing techniques • Correctnesscan be checked with test selection criteria based on structural coverage (to reveal failures) or weakest precondition computation (to prove the absence of faults)

37. Different T&A for checking the same properties for different software • Test selection criteria based on structural coverage are different for • procedural software (statement, branch, path,…) • object oriented software (coverage of combination of polymorphic calls and dynamic bindings,…) • concurrent software (coverage of concurrent execution sequences,…) • mobile software (?) • Absence of deadlock can be statically checked on some systems, require the construction of the reachability space for other systems

38. Principles • Principles underlying effective software testing and analysis techniques include: • Sensitivity: better to fail every time than sometimes • Redundancy: making intentions explicit • Partitioning: divide and conquer • Restriction: making the problem easier • Feedback: tuning the development process

39. Sensitivity:Better to fail every time than sometimes • Consistency helps: • a test selection criterion works better if every selected test provides the same result, i.e., if the program fails with one of the selected tests, it fails with all of them (reliable criteria) • run time deadlock analysis works better if it is machine independent, i.e., if the program deadlocks when analyzed on one machine, it deadlocks on every machine

40. Redundancy: making intentions explicit • Redundant checks can increase the capabilities of catching specific faults early or more efficiently. • Static type checking is redundant with respect to dynamic type checking, but it can reveal many type mismatches earlier and more efficiently. • Validation of requirement specifications is redundant with respect to validation of the final software, but can reveal errors earlier and more efficiently. • Testing and proof of properties are redundant, but are often used together to increase confidence

41. Partitioning: divide and conquer • Hard testing and verification problems can be handled by suitably partitioning the input space: • both structural and functional test selection criteria identify suitable partitions of code or specifications (partitions drive the sampling of the input space) • verification techniques fold the input space according to specific characteristics, thus grouping homogeneous data together and determining partitions

42. Restriction: making the problem easier • Suitable restrictions can reduce hard (unsolvable) problems to simpler (solvable) problems • A weaker spec may be easier to check: • It is impossible (in general) to show that pointers are used correctly • But the simple Java requirement that pointers are initialized before use is simple to enforce • A stronger spec may be easier to check: • It is impossible (in general) to show that type errors do not occur at run-time in a dynamically typed language • But statically typed languages impose stronger restrictions that are easily checkable

43. Feedback: tuning the development process • Learning from experience: • Checklists are built on the basis of errors revealed in the past • Error taxonomies can help in building better test selection criteria Taxonomy: the science of classification according to a pre-determined system. Enables the intelligent grouping and sub-grouping of “things”.

44. Software Quality Assurance (SQA) • Formal definition of software quality assurance: • “the systematic activities providing evidence of fitness for use of the total software product” ([Pezze and Young]) • Software quality assurance is achieved through the use of established guidelines for quality control to ensure the integrity and prolonged life of software. • The relationship among quality assurance, quality control, the auditing function, and software testing are often confused. • Indeed, they refer to different aspects of the quality assurance process.

45. SQA Plan: Steps • Document the plan. SQA plan should include the sections: • Purpose section • Reference document section • Management section • Documentation section. This section could include the following example items: the software requirement specification, system user guide, the installation guide, test results summary, software unit documentation, and translator software units • Standards, practices, conventions, and metric section • Reviews and inspection section • Software configuration management section • Problem reporting and corrective action section • Tools, techniques, and methodologies section • Code control section • Media control section • Supplier control section • Records collection, maintenance, and retention section • Testing methodology • Obtain management acceptance • Obtain development acceptance • Plan for implementation of the SQA plan • Execute the SQA plan

46. ISO9000 quality standards • ISO9000 is a quality series and comprises a set of five documents developed in 1987 by the International Standards Organization (ISO). ISO9000 standards and certification are usually associated with non-IS manufacturing processes. However, application development organizations can benefit from these standards and position themselves for certification if necessary. • ISO9000 consists of ISO9001, ISO9002, and ISO9003. • ISO9001: very comprehensive and defines all the quality elements required to demonstrate the supplier’s ability to design and deliver a quality product • ISO9002 covers quality considerations for the supplier to control the design and development activities • ISO9003 demonstrates the supplier’s ability to detect and control product nonconformity during inspection and testing • ISO9004 describes the quality standards associated with ISO9001, ISO9002, and ISO9003 and provides comprehensive quality checklist.

47. Capability Maturity Model (CMM) CMM: a model for judging the maturity of the software processes of an organization and for identifying the key practices that are required to increase the maturity of these processes. Five levels for the CMM: • Initial. The software process is characterized as ad hoc, and occasionally even chaotic • Repeatable. Basis project management processes are established to track cost, schedule, and functionality. • Defined. The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization • Managed. Detailed measures of the software process and product quality are collected. • Optimizing. Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

48. Quality through a Continuous Improvement Process and TQM • The best practice in the industry for quality assurance through a continuous improvement process is known as the Total Quality Management (TQM) principle which is essential for software quality assurance also. TQM was developed in the mid 1940s by Dr. Edward Deming who at the time was an advisor in sampling at the Bureau of Census and later became a professor of statistics at the New York University Graduate School of Business Administration. TQM has little success in US during that period but did gain success in Japan after World War II. In the 1970s and 1980s, many American companies began adopting TQM principles • TQM is defined as a quality-centered, customer-focused, fact-based, team-driven, senior-management-led process to achieve an organization’s strategic imperative through continuous process improvement

49. Fourteen points for TQM (1) • Create constancy of purpose for improvement of product and service. Organizations must allocate resources for long-term planning, research, and education, and for the constant improvement of the design of their products and services. • Adopt the new philosophy. Government regulations representing obstacles must be removed, transformation of companies is needed. • Cease dependence on mass inspections. Quality must be designed and built into the processes, preventing defects rather than attempting to detect and fix them after they have occurred • End the practice of awarding business on the basis of price tags alone. Organizations should establish long-term relationships with [single] suppliers.

50. Fourteen points for TQM (2) • Improve constantly and forever the system of production and service. Management and employees must search continuously for ways to improve quality and productivity. • Institute training. Training at all levels is a necessity, not optional. • Adopt and institute leadership. Managers should lead, not supervise. • Drive out fear. Make employees feel secure enough to express ideas and ask questions. • Break down barriers between staff areas. Working in teams will solve many problems and will improve quality and productivity