Software Testing

1. Software Testing

2. Recap • Testing methods / Types • Black Box testing • White Box testing • Incremental / Thread testing • Testing levels Vs testing methods • Testing Strategy

3. Contents • Debugging Process • Test cases • What is a test case? • Writing better test cases • Examples

4. Debugging Process • Debugging occurs as a consequence of successful testing • It is still very much an art rather than a science • Good debugging ability may be an innate human trait • Large variances in debugging ability exist • The debugging process begins with the execution of a test case • Results are assessed and the difference between expected and actual performance is encountered • This difference is a symptom of an underlying cause that lies hidden • The debugging process attempts to match symptom with cause, thereby leading to error correction

5. Why is Debugging so Difficult? • The symptom and the cause may be geographically remote • The symptom may disappear (temporarily) when another error is corrected • The symptom may actually be caused by non-errors (e.g., round-off accuracies) • The symptom may be caused by human error that is not easily traced (continued on next slide)

6. Why is Debugging so Difficult?(continued) • The symptom may be a result of timing problems, rather than processing problems • It may be difficult to accurately reproduce input conditions, such as asynchronous real-time information • The symptom may be intermittent such as in embedded systems involving both hardware and software • The symptom may be due to causes that are distributed across a number of tasks running on different processes

7. Debugging Strategies • Objective of debugging is to find and correct the cause of a software error • Bugs are found by a combination of systematic evaluation, intuition, and luck • Debugging methods and tools are not a substitute for careful evaluation based on a complete design model and clear source code • There are three main debugging strategies • Brute force • Backtracking • Cause elimination

8. Strategy #1: Brute Force • Most commonly used and least efficient method • Used when all else fails • Involves the use of memory dumps, run-time traces, and output statements • Leads many times to wasted effort and time

9. Strategy #2: Backtracking • Can be used successfully in small programs • The method starts at the location where a symptom has been uncovered • The source code is then traced backward (manually) until the location of the cause is found • In large programs, the number of potential backward paths may become unmanageably large

10. Strategy #3: Cause Elimination • Involves the use of induction or deduction and introduces the concept of binary partitioning • Induction (specific to general): Prove that a specific starting value is true; then prove the general case is true • Deduction (general to specific): Show that a specific conclusion follows from a set of general premises • Data related to the error occurrence are organized to isolate potential causes • A cause hypothesis is devised, and the aforementioned data are used to prove or disprove the hypothesis • Alternatively, a list of all possible causes is developed, and tests are conducted to eliminate each cause • If initial tests indicate that a particular cause hypothesis shows promise, data are refined in an attempt to isolate the bug

11. Test Cases

12. Test case • A test case contains: • A sequence of Steps describing actions to be performed, • Test data to be used • An expected response for each action performed. • Test Cases are written based on Business and Functional/Technical requirements, use cases and Technical design documents.  • There can be 1:1 or 1:N or N:1 or N:N relationship between requirements and Test cases • The level of details specified in test cases will vary depending on Organizations, Projects and also on the Test Case Template used OR on the Test Management tool being used in the project. 

13. Test case (cont…) • Construction of Test Cases also helps in: • Finding issues or gaps in the requirements • Technical design itself. • As Test Case construction activity would make tester to think through different possible Positive and Negative scenarios. • It is test cases against which tester will verify the application is working as expected. • Number of test cases to be created depends on the size, complexity and type of testing being performed

14. Test case examples • Test case for ATM • TC 1 :- succesful card insertion. • TC 2 :- unsuccessful operation due to wrong angle card insertion. • TC 3:- unsuccesssful operation due to invalid account card. • TC 4:- successful entry of pin number. • TC 5:- unsuccessful operation due to wrong pin number entered 3 times. • TC 6:- successful selection of language. • TC 7:- successful selection of account type. • TC 8:- unsuccessful operation due to wrong account type selected w/r to that inserted card. • TC 9:- successful selection of withdrawl option. • TC 10:- successful selection of amount. • TC 11:- unsuccessful operation due to wrong denominations. • TC 12:- successful withdrawl operation. • TC 13:- unsuccessful withdrawl operation due to amount greater than possible balance. • TC 14:- unsucceful due to lack of amount in ATM. • TC 15:- un due to amount greater than the day limit. • TC 16:- un due to server down. • TC 17:- un due to click cancel after insert card. • TC 18:- un due to click cancel after insert card and pin no. • TC 19:- un due to click cancel after language selection,account type selection,withdrawl selection, enter amount

15. Test cases for a web page • Testing without entering any username and password • Test it only with Username • Test it only with password. • User name with wrong password • Password with wrong user name • Right username and right password • Cancel, after entering username and password. • Enter long username and password that exceeds the set limit of characters. • Try copy/paste in the password text box. • After successful sign-out, try “Back” option from your browser. Check whether it gets you to the “signed-in” page.

16. Some terminologies… • Documented Test Cases are referred to as Test Scripts. • Related test cases or test scripts are collectively referred to as Test Suite. 

17. Writing better test cases • Requirement ID(s) being covered in the Test Case. • Test Condition(s) and Expected Result(s) being exercised in the Test Case. • Initial setup required for executing the test script. This could be environment or data or configuration setup to be done before running the test case. • Post execution activities. For e.g.:- Delete the application user ‘WebAdmin’ after test execution is completed. • Priority (High, Medium and Low) of the Test Case. Priority will help the tester to decide which test case(s) have to be run earlier than others. • Complexity of the Test Case. It will help to identify and filter Test Cases based on complexity. This would help in assigning test cases to testers, before test execution. • Approximate time required for executing the test case. This entry is required from Project management perspective to track the productivity and also to ensure we can still meet the test execution deadlines. • Test Steps. This contains instruction on what actions to perform and what test data to use.

18. Writing better test cases (cont…) • Expected results. Each Test Step will have a corresponding Expected result field that would specify the expected response. • Actual result. Each Test Step will have a corresponding Actual result field. Tester would enter the details on the response he saw after executing the test step. • Test Step result. Typically this field would contain values Not Applicable, No Run, Passed, Failed or in progress etc • Test Case Version number • Test case creation timestamp. • Revision history. When and who wrote or modified the test case etc. • Test Case status (Draft, completed, reviewed, Not Valid etc.) • Test Case execution timestamps. • Associated Defects. This field will help to identify what are the existing defect(s) that are associated with the test case. • Project Name • Application Name

19. How to write test cases? • Requirements • Use cases • Design documents

20. Understanding requirements • First understand Business requirements get the big picture of what is being achieved from business perspective. • Read and understand how Business requirements are exploded into functional requirements. • Go through Technical specification and UI design document to gain understanding of how the business requirements get translated into application functionality. • Once you do this exercise obviously you will get at least few questions or you may even discover few gaps in the requirements. • Reach out to Business Analyst and Business User (as required) and get your questions answered. Now you are ready to go ahead with your Test Scenario identification. • Identify Test Scenarios, Test Conditions and Expected results. Once you have sign-off on Test Scenarios, Test Conditions and Expected results, start creating Test Cases.

21. Design documents • Also consider verifying the technical aspects mentioned in the Technical and look and feel aspects mentioned in UI design documents. • Some of the technical aspects like: • Field Length • Validation • Data structures will be documented in-detail in Technical specification. • So, during test case construction, one has to incorporate additional validations as well.

22. How to write test cases? • Get the test cases reviewed • Detail of each test case • Write test cases in plain English • Requirement traceability matrix (RTM) • Provide test data within the test cases • Test case version control • Prioritize test cases • Pre-requisites and clean up sections

24. Test case review! • Get your Test Cases reviewed by Business Analyst and Business Users. • It is always good to get inputs from Subject Matter Experts • It is better to get review comments from Business Analysts and Business Users before testing starts rather than they pointing out at the deficiencies of your test cases during acceptance testing or Post implementation.

25. How detailed should be a test case? • Test Cases steps should be as detailed as possible and should not be written at high level. • Writing detailed test steps is very important considering the fact that the same person who wrote the test cases may not always execute the same test cases. • If the test cases are not very detailed then the person who executes the test case for the first time will not be able to validate the system thoroughly as he/she might have not gone through the requirements and will test only as per the test case steps.

26. Example of detail • Example of a Test Case that is written at high level. • Step No: 1Step Description: Login to test application with valid user id/passwordExpected Result: Home page is displayed • Step No: 2Step Description: Click “Logout” link on home pageExpected Result: Login page is displayed • Example of a Test case written that is written in detail. • Step No: 1Step Description: Open URL https://www.example.com/login.aspExpected Result: Login page is displayed and contain the below fieldsa) “User Name” text fieldb) “Password” text fieldc) “Submit” button • Step No: 2Step Description: Once Login page is displayed. Enter valid userid/password.a) Enter “user1” in “User Name” text fieldb) Enter “abc123″ in “password” text fieldExpected Result: a) Verify “User Name” is populated with text “user1”b) Verify text entered in “password” field is masked and is not readable. • Step No: 3Step Description: Click “Submit” buttonExpected Result: Verify application “Home” page is displayeda) Verify “Home” page displays “Welcome user1” message on top of left navigation Menu.b) Verify Left Navigation menu contains links “Directory”, “Submission”, “Latest Links”, “Approve Links”, “Logout”. • Step No: 4Step Description: Click “Logout” link on the left menu.Expected Result: Verify user is successfully logged out of the application and application login page is displayed https://www.example.com/login.asp

27. Why do we write test cases? • Test case creation could have two broad goals: • Test Cases are supposed part of the deliverable to the customer. TC goal credibility in this case. Typically UAT (acceptance) level. • Test Cases are for team internal use only. Typically System level testing. Testing efficiency should be the goal in this case. The idea is to write test cases based on design while code is incomplete, so that we could test product quickly once the code is ready. • In case of agile development goal number one is not applicable • TC are used internally, but the goal is credibility, not efficiency. It also means that TC are dramatically reworked during test execution

28. Summary