Download
1 / 63
660 likes | 866 Views
Testing. Testing is a critical process within any software development and maintenance life cycle model.
E N D
Testing • Testing is a critical process within any software development and maintenance life cycle model. • Testing is becoming increasingly important to the software development industry because of its implications on the quality of the produced software and the reputation of the producing company. • Testing activities include testing specifications, designs and implementations. • Testing implementations is often referred to as software testing, or simply testing.
Testing • Testing was performed in ad hoc and informal manners • Testers were not properly trained and unmotivated • Testing often downplayed and little resources were allocated. • Now: quality of produced software is greatly affected by the quality of testing performed. • Reputation and marketability of the software is affected by the type and amount of errors discovered after release. • Errors should be resolved and fixed before the software is released in the market. • Revealing all failures and hence removing their resulting errors or faults is a very optimistic idea.
Test coverage • Only exhaustive testing can guarantee full coverage. • Exhaustive testing is impractical since it may take years to complete even for testing small size programs. • Consider testing the software when a first name text field of 20 characters maximum is filled by a user. Exhaustive testing requires that we test the software for all possible correct and incorrect field input values. • Assume 80 possible correct and incorrect characters, the number of possible combinations for a valid or invalid first name would exceed 8020. Using a computer that takes, say, 50 nanoseconds to process one combination, we would need about 1011 years to complete the testing for all possible combinations. It is indeed impractical!
The reality • With the absence of exhaustive testing, there are no formal guarantees that software is error-free. • In fact, it may happen in practice that even when all errors have been detected during testing, they may not all be fixed due to the lack of time. • Unfixed errors may be fixed in future releases of the software. • The difficulty in software testing is the inability to quantify the errors existing in software, and hence predict how many errors are left undiscovered.
Test coverage experiment • Random testing experiment: errors of different levels of complexity were inserted in the software under test • All errors except for one were discovered progressively. • No indication on how much more testing is needed • Finding errors is an asymptotic nonlinear function of time.
Ethics for software organizations • It is the ethical responsibility of the software development company to ensure that • adequate and appropriate resources are allocated to the testing activities, • employees are properly trained to perform their testing tasks, • state-of-the-art testing tools are available for use by trained employees, and • an environment emphasizing the importance of software quality and dependability is being promoted internally.
Ethics for testers • It is the ethical responsibility of the software tester to ensure that • they remain up-to-date in their testing knowledge, and • on the look for better tools and techniques, and • that they put their best efforts and mental capabilities to discover as many errors as possible. • After all, human well-being and physical safety may be at stake.
Dynamic and static testing • Dynamic testing • Test the software by executing it and applying test inputs • Observe the output and make a judgment to take a verdict • Static testing • Check code without executing it • Automatic static testing: Code as an input to a static analysis tool • Non-automatic static testing: review, walkthrough, inspection
Test case • Test plan includes a test suite • A test suite is a set of organized test cases • Organize around use cases for example • Template to describe a test case • Id, name, reference, priority, preconditions, preamble, input, expected output, observed output, verdict and remarks • Test cases are obtained using known testing techniques and strategies
Test case design vs execution • Test case is developed first – verdict and remarks are empty • Test case is executed when the actual testing is performed – the verdict is obtained • Typically test cases are developed (designed) before even writing any code • Acceptance test cases in the acceptance test plan (ATP) can be developed after requirements specification phase • Unit and integration test plans can be developed after the design is completed and prior to coding
White-box, black-box and grey-box testing techniques • White-box: module or unit testing • Black-box: module testing and system testing
Black-box unit/module testing techniques • Boundary value analysis • Equivalence class testing • Decision table / decision tree based testing • State based testing • Robustness testing
Boundary value analysis • BVA: test inputs obtained from the boundaries • Serious errors hide at the boundaries. • For example, simple boundary values for an input of integer data type would include -INT_MAX, -INT_MAX + 1, -1, 0, +1, INT_MAX - 1 and INT_MAX. • For complex and composite data types, boundary values become more difficult to generate and may require more ingenuity to obtain them. • Moreover, if interface specification includes more than one input, different test cases considering possible feasible combinations of BVs for each input must be considered
BVA Example • Suppose you are given an executable code including a function called digitCount whose functional interface specification states that digitCount takes an integer n as an input parameter, and returns an integer representing the number of digits in n. • Test cases whose test inputs are selected based on boundary value analysis are partially shown in Table 8.3. • Assume that for a 32-bit integer, INT_MAX is +2147483647.
Testing function digitCount based on boundary value analysis
Equivalence class testing • Partitioning input domain into disjoint subsets or partitions • Partition includes related values • Consider one test input from each partition • Equivalence relation among partition elements • ‘having the same number of digits’ • ‘all positive integers’ • ‘all even numbers’ • ‘all prime numbers’ • ‘all powe or 2 numbers’ • …..
Decision table based testing • Decision table: each column denoting a rule becomes the basis of a test case • Conditions produce inputs • Actions produce expected outputs • In table 4.1: input variables are age, history and speed, and output variables are penalty and points
Decision tree-based testing • Each distinct path from the root node to a leaf node is the basis of a test case • If along the path, a variable input value does not exist, the test input for that variable can be assigned any valid value. • For example, in fig 4.23, for the test case corresponding to the leftmost path, the speed value can be any illegal speed
State-based testing • An implemented finite state machine conforms to its specification, if the observed output sequence matches the expected output according to the specification • Test sequence must cover all states transitions • Used to test do black box testing of systems • GUI, protocols, web page connectivity, … • Transition tour: a sequence covering all transitions and returning back to the initial state • Can only cover output errors
State based testing • We need to cover state errors – reaching bad states • Use distinguishing sequences DS to detect if the correct state has been reached • DS is unique for every state – its is a state identification sequence • When DS is applied at the different states it produces different output sequences hence recognizing the state at which DS was applied • For a transition from s1 to s2 with i1/o1, assuming we know we are at s1, we apply i1 and must observe o1, then to make sure we reached s2 we apply the DS and if we observe the expected output for s2, we can confirm that we have reached s2
Black box-based robustness testing • Test a system using one or preferably more black box based testing method • Test for incorrect and illegal inputs also • In equivalence class testing, select one representative bad input from every partition of the bad input values • For FSM based systems, apply unexpected inputs at each state – the system should survive • For boundary value testing, test for illegal values beyond the boundaries – the system should survive
White-box testing techniques • Given the source code of the module to test • Control flow based testing techniques • Data flow based testing techniques • Path testing
Control flow based testing • Given source code of a module, transform it to a control flowgraph • Nodes and edges – decision nodes, joining nodes
Example: digitCount 1 int digitCount(int n) 2 { 3 int count = 0; 4 while (n != 0) 5 { n = n / 10; 6 count++; 7 } 8 return count; 9 }
Problem • When we have a multiple condition decision • Reduced number of test cases but … • A typo error may not be detected – because of short circuiting • Solution – break the decision into many nodes
Data flow testing • Need to test problems with the data variables and the way they are used • Produce a data flow graph by annotating the control flow graph • Can only consider critical variables • Actions on variables: definition (def), destroying or dereferencing (kill), use in a predicate (p-use), use in a computation (c-use), declaration (dcl)
Example 8.11 • Examine what is happening to variable x. • Check if there any anomalous pattern of use • For example, 2 contiguous x def, or x kill followed by p-use or c-use of x
x x dcl def x p-use x c-use x def x c-use x kill
Example 8.12 • Examining 3 variables • Consider the path for each variable separately • Any pattern of anomalous use?
Path expression • Representing in a generic way an execution sequence • Can represent the complete program • Use operators: + for choice . For a concatenation {} * for zero or more repetition {}+ for one or more repetition () for grouping
