Software Testing Definition

1. Software Testing Definition Software Testing Module (0721430) Dr. Samer Odeh Hanna

2. Testing and Related Terms' Definitions • The testing literature is mired with confusing and inconsistent terminology because it has evolved over decades and by different writers (Jorgensen, 2002). The following slides will introduce a definition of testing and the related terms. The testing literature has the following main definitions of testing: • Hetzel (Hetzel, 1973) “The process establishing confidence that a program or system does what it suppose to”.

3. Testing Definition • Myers (Myers, 1979) “The process of executing a program or system with the intent of finding errors”. • Beizer (Beizer, 1990) “A process that is part of quality assurance and aims to show that a program has bugs (faults)”. • IEEE (IEEE, 1990) “An activity in which a system or component is executed under specified conditions, the results are observed or recorded, and evaluation is made of some aspect of the system or components”.

4. Testing Definition • Voas (Voas and McGraw, 1998) “the process of determining whether software meets its defined requirements”. • Harrold (Harrold, 2000) “One of the old forms or verification that is performed to support quality assurance”.

5. Testing Definition • Sommerville (Sommerville, 2004) defines testing as: “Software testing involves running an implementation of the software with test data. You examine the outputs of the software and its operational behavior to check that it is performing as required. Testing is a dynamic technique of verification and validation”

6. Testing related terms • The previous definitions introduce testing related terms such as quality assurance, fault, error, verification and validation. • The goal of quality assurance is to improve software quality and to determine the degree to which the actual behavior of the software is consistent with the intended behavior or quality of this software. • Testing related terms are defined to enable a better understand of testing definitions:

7. Fault, Errors, and Failure • As mentioned before, Fault, error, and failure are considered as a threat to the dependability and they are defined as follows: • Faultis defined as: “A defect in the system that may lead to an error” (Osterweil, 1996); another name of a fault is bug or defect. For a certain quality attribute there exist faults that affect this quality attribute. Examples of faults that may affect robustness quality attribute include: wrong input accepted, correct input rejected (IEEE, 1995). Some faults can affect more than one quality attribute, for example, wrong input accepted fault affects robustness, fault tolerance and security.

8. Fault, Errors, and Failure • Error is defined as “The part of the total state of the system that may lead to a failure” (Avizienis et al. 2004). • Failure is defined as “the deviation of the execution of a program from its intended behavior” (Osterweil, 1996) Another definition of failure is: “An event that occurs when the delivered service deviates from correct service” (Avizienis et al. 2004).

9. Verification and Validation • Verification and validation (V & V) is the process of checking that a program meets its specification and delivers the functionality expected by the people paying for the software (Sommerville, 2004). Verification and validation are defined as follows:

10. V & V • Verification is defined as: “Checking that the software conforms to its specification and meets its specified functional and non-functional requirements” (Sommerville, 2004) • Validation is defined as “Ensure that the software system meets the customer’s expectations” (Sommerville, 2004) • Another definition of validation is “Determination of the correctness of the final program or software produced from a development project with respect to the user needs and requirements” (Adrion , et al. 1982).

11. Roles of software testing • It is noticed that different researchers view software testing differently, however, the following roles or goals of software testing can be included from the definitions: • Testing involves running or executing the system under test with test data. • Testing is a performed to support quality assurance by assessing the quality attributes

12. Roles of software testing • Testing is performed to find faults before they cause an error and consequently a failure • Testing is a form of verification. • Testing is a form of validation.

13. Relation of roles of software testing definitions • Table 1. analyzes the roles in each definition of software testing in order to reach a definition that contains all the testing roles. The table indicates whether we can infer a role (column) based on a particular definition (row). The symbols shown in the table are: • The full circle (●) indicates that the definition explicitly states the role. • The symbol (≈) indicates that the definition does not explicitly express that specific role, but the context of the definition suggests it. • The empty circle (○) indicates that the role is not included in a specific definition.

14. Relation of roles of software testing definitions

15. General Defenition • It is noticed from table1 that Sommerville (Sommerville, 2004) definition contains more of the software testing roles than the other definitions. • After analyzing all the definitions, this module will use the following definition of software testing that includes all the roles mentioned in Table 1: • Software testing is a quality assurance process that is part of the verification and validation processes, and involves executing the system under test with test data for the purpose of detecting faults and assessing the quality attributes of that system or software component.

16. References • Adrion W., Branstad M. & Cherniavsky, J. (1982) Validation, Verification, and Testing of Computer Software. ACM Computing Surveys, (Vol. 14, No. 2), pp.159-192. • Avizienis, A., Laprie, J.-C., Randell, B. & Landwehr, C. (2004) Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Transactions on Dependable and Secure Computing, (Vol. 1, No. 1), China, pp. 11-33. • Boehm, B. W., Brown, J. R. & Lipow, M. (1976) Quantitative Evaluation of Software Quality. Proceedings of the 2nd International Conference on Software Engineering (ICSE), California, USA, 1976, pp. 592-605. • ISO 9126-1: 2001 (2001) Software Engineering – Product quality – Part 1: Quality Model, International Organization of Standardization, Geneva, Switzerland.

17. References • McCall, J. A., Richards, P. K. & Walters, G. F. (1977) Factors in Software Quality. Three volumes, US Department of Commerce, National Technical Information Service (NTIS). • Neumann, P. (2004). Principled assuredly trustworthy composable architecture. Emerging draft of the final report for DARPA’s composable high assurance trustworthy systems (CHATS) program. [Retrieved from http://www.csl.sri.com/users/neumann/chats4.pdf]. • Raghavan, G. (2002). Improving Software Quality in Product Families therough Systematic Reengineering. Proceedings of the Software Quality- ESSQ conference, Berlin, Germany, Springer-Verlag, pp. 90-99. • Zhang, J. & Zhang, L.-J. (2005) Web Services Quality Testing. International Journal for Web Services Research, (Vol. 2, No. 2), pp. 1-4.