Testing and Debugging

Testing and Debugging Testing, Bug Fixing and Debugging the Code YordanDimitrov Telerik Corporation www.telerik.com

Testing Testing is a means of detecting errors.

Developer Testing • Unit testing • Component testing • Integration testing • Regression testing • System testing

Additional testing • beta tests • customer-acceptance tests • performance tests • configuration tests • platform tests • stress tests • usability tests

Black and White • Black-box testing • White-box (or glass-box) testing.

Role of Developer Testing in Software Quality • Individual testing steps (unit test, component test, and integration test) typically find less than 50 percent of the errors present each. • The combination of testing steps often finds less than 60 percent of the errors present (Jones 1998).

Why is testing a hard activity? • Testing's goal runs counter to the goals of other development activities • Testing can never completely prove the absence of errors • Testing by itself does not improve software quality • Testing requires you to assume that you'll find errors in your code

How much time should be spent in developer testing ? Developer testing should probably take 8 to 25 percent of the total project time

Recommended Approach to Developer Testing • Test for each relevant requirement to make sure that the requirements have been implemented • Test for each relevant design concern to make sure that the design has been implemented • Use "basis testing" to add detailed test cases to those that test the requirements and the design • Use a checklist of the kinds of errors you've made on the project to date or have made on previous projects

Test First or Test Last? • Effort is the same • Detect defects earlier • Forces you to think at least a little bit • Exposes requirements problems sooner • Run it when you want source flickr

Limitations of Developer Testing • Developer tests tend to be "clean tests“ • Developer testing tends to have an optimistic view of test coverage • Developer testing tends to skip more sophisticated kinds of test coverage

Bag of Testing Tricks • Incomplete Testing • Structured Basis Testing • Data-Flow Testing • Equivalence Partitioning • Error Guessing • Boundary Analysis • Classes of Bad Data • Classes of Good Data • Use Test Cases That Make Hand-Checks Convenient

Structured Basis Testing Test each statement in a program at least once. Compute the minimum number of test cases: • Start with 1 for the straight path through the routine. • Add 1 for each of the following keywords, or their equivalents: if, while, repeat, for, and, and or. • Add 1 for each case in a case statement. If the case statement doesn't have a default case, add 1 more.

Sample Test Cases

Data-Flow Testing The normal combination of data states is that a variable is defined, used one or more times, and perhaps killed. The key to writing data-flow test cases is to exercise all possible defined-used paths: • All definitions. Test every definition of every variable—that is, every place at which any variable receives a value. • All defined-used combinations. Test every combination of defining a variable in one place and using it in another.

Sample Test Cases if ( Condition 1 ) { x = a; } else { x = b; } if ( Condition 2 ) { y = x + 1; } else { y = x - 1; }

Boundary Analysis

Compound Boundaries Minimum and Maximum allowable values

Classes of Bad Data • Too little data • Too much data • The wrong kind of data • The wrong size of data • Uninitialized data

Classes of Good Data • Nominal cases—middle-of-the-road, expected values • Minimum normal configuration • Maximum normal configuration • Compatibility with old data

Typical Errors • Which Classes Contain the Most Errors? • Errors by Classification • The scope of most errors is fairly limited • Many errors are outside the domain of construction • Most construction errors are the programmers' fault • Clerical errors (typos) are a surprisingly common source of problems • Misunderstanding the design is a recurring theme in studies of programmer errors • Most errors are easy to fix • It's a good idea to measure your own organization's experiences with errors

Improving Your Testing • Planning to Test • Retesting (Regression Testing) • Automated Testing

Debugging Debugging is a means of diagnosing and correcting the root causes of errors that have already been detected.

Finding a Defect • Stabilize the error • Locate the source of the error • Gather the data • Analyze the data and form hypothesis • Determine how to prove or disprove the hypothesis • Prove or disprove the hypothesis by 2c • Fix the defect • Test the fix • Look for similar errors

DEMO: Finding a Defect The program is supposed to print a list of employees and their income-tax withholdings in alphabetical order.

Tips for Finding Defects (1) • Use all available data • Refine the test cases • Check unit tests • Use available tools • Reproduce the error several different ways • Generate more data to generate more hypotheses • Use the results of negative tests • Brainstorm for possible hypotheses

Tips for Finding Defects (2) • Narrow the suspicious region of the code • Be suspicious of classes and routines that have had defects before • Check code that’s changed recently • Expand the suspicious region of the code • Integrate incrementally • Check for common defects • Talk to someone else about the problem • Take a break from the problem

Fixing a Defect • Understand the problem before you fix it • Understand the program, not just the problem • Confirm the defect diagnosis • Relax • Save the original source code • Fix the problem not the symptom • Make one change at a time • Add a unit test that expose the defect • Look for similar defects

Psychological Consideration • Your ego tells you that your code is good and doesn't have a defect even when you've seen that it has one. • How "Psychological Set" Contributes to Debugging Blindness

Psychological Distance • How "Psychological Distance" Can Help

Tools • Diff Tools • Compiler Warning Messages • Set your compiler’s warning level to the highest • Treat warnings as errors • Initiate project wide standards • Extended Syntax and Logic Checking • Profilers • Test Frameworks/Scaffolding • Debuggers

Testing and Debugging