Exploratory Testing – an improved approach

1. Exploratory Testing – an improved approach by Jayapradeep Jiothis Narendra Kumar

2. Agenda Introduction. Existing formalized methods. Improvements. Orthogonal arrays, Decision tables,State diagrams. Team Management. Applying in Corporate world.

3. Introduction What is Exploratory Testing ? How is it different from Ad hoc Testing ? Where does ET fit ? Characteristics of Exploratory tester. Learning styles and ET. Styles of Exploration. Pros and Cons of ET.

4. What is ET ? "Exploratory testing involves simultaneously learning, planning, running tests, and reporting / troubleshooting results." - Dr. Cem Kaner. "Exploratory testing is an interactive process of concurrent product exploration, test design and test execution. To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing.” - James Bach.

5. How is it different from Ad hoc Testing ? Ad hoc Testing: “It is an impromptu and unrehearsed testing to uncover the bugs with no mission or goal.”

6. Increasing degree of ET Ad hoc testing Pure Scripted testing 0 100Documentation(%) Ad Hoc Vs ET Charters Vague Scripts

7. Where does ET fit ? No specifications or little documentation available. No time to specify,script and test. Thread Pulling. When the application does not have any future releases. ET can be associated with scripted testing to improve the coverage. To help assess risks. When there is open communication between development and testing teams.

8. Characteristics of Exploratory Tester : Design and Excel : An exploratory tester is first and foremost a Test Designer. The excellent exploratory tester is able to craft tests that systematically explore the product. One way to approach it is to consider it a questioning process. To design a test is to craft a question for a product that will reveal vital information.

9. Characteristics of Exploratory Tester : …contd Hawk eyes : Excellent exploratory testers are excellent observers . The exploratory tester must watch for anything unusual mysterious, or otherwise relevant to the testing. Exploratory testers must be careful to distinguish observation from inference(even under pressure) and avoid to allow preconceived assumptions to blind them.

10. Characteristics of Exploratory Tester : …contd Critical Thinking : Excellent exploratory testers are able to review and explain their logic, looking for errors in their own thinking. Break the Rules : Exploratory tester clears their mind from the shackles of the past, thinks creatively and laterally, to produce better ideas.Use heuristics to accomplish this.

11. Characteristics of Exploratory Tester : …contd Race up the learning curve : Excellent exploratory testers climb learning curves more quickly than most. Intelligence helps but its more a matter of skill and practice that is build over a period of time. Rich Resources : Excellent exploratory testers build a deep inventory of tools,information sources, test data, and friends to draw upon. While testing, they remain alert for opportunities to apply those resources to the testing at hand.

12. Characteristics of Exploratory Tester : …contd Manage and Succeed : Excellent exploratory testers know the value of their precious time.They differentiate between a dead end and a promising lead. They follow their main mission while taking deviations to ensure the coverage. Keep Track : Excellent exploratory tester keeps track of what was tested, techniques used and risks the tests were intended to explore, and how they relate to the mission of testing.

13. Learning Styles and ET: A learning style is a characteristic strengths and preferences in the ways they take in and process information. Sensory/Intuitive : Preferred type of information. Visual/Verbal : Preferred mode of information received. Active/Reflective : Preferred organization of information. Inductive/Deductive : Information processing approach. Sequential/Global : Preferred way of understanding. • This is based on an informal survey of a small number of testers by Andy Tinkham and Cem Kaner. Reference “Learning Styles and Exploratory Testing”.

14. Learning Styles and ET: …contd Sensory/Intuitive A sensory learner preferentially perceives information by the external senses.Examples :Sights , Sounds , Physical sensations.etc. The sensory-based person focuses on the details gained on actual observations. Inclined to develop a standard pattern for approaching exploratory testing. An intuitive learner perceives information relies on the internal information (generated from memory, conjecture, and interpretation) and intuition. The intuitor focuses on the internal model , applies risk-based approach and different theories of error to the software.

15. Learning Styles and ET: …contd Visual/Verbal Visual learners perceive information from visual images such as pictures, movies, diagrams or demonstrations.They tend to work off an internal model that is picture-based.They make diagrams and pictures for their notes as they explore. Verbal learners perceive information hearing (or reading) to lectures, written words, and mathematical formulas.They use a textual model like descriptions for their testing.

16. Learning Styles and ET: …contd Inductive/Deductive An inductive learner prefers to work from specifics and derive the generalities.They gather as many specifics as possible along with the historical data and generalize them to the application. A deductive learner starts with the generalities and applies them to the specific situations they encounter.They apply guidelines and heuristics immensely in their approach.

17. Learning Styles and ET: …contd Active/Reflective : Active learners experiment with the information they receive and like to work in groups and find solutions. They do very rapid hands-on testing.They solicit the feedback from the teams. Reflective learners prefer to think about information before they use it and tend to work alone.They have a thought process that precedes each test.They make up for their lack of speed by executing the “good” tests that are most likely to find bugs.

18. Learning Styles and ET: …contd Sequential/Global Sequential learners learn in a logically structured incremental manner. They follow “linear reasoning processes”. The sequential tester will seem to get off to a faster start and test plans will grow in complexity over time as the understanding deepens. Global learners learn in chunks and have a holistic view of the information. A global tester will get off to a slower start. They need demonstrations on the application.Once the understanding is complete , he creates complex tests and can catch the bugs which are normally not seen.

19. Styles of Exploration : Hunches Error handling Troubleshooting Examples Group Insight Models Specifications Invariances Interference

20. Styles of Exploration : …contd Hunches : Intuitions  Interrogative  Analogous  Hearsay Trace Changes

21. Styles of Exploration : …contd Examples : Use Cases Simple Walkthroughs  Positive Thinking Scenarios Soap Operas

22. Styles of Exploration : …contd Models : Architecture diagrams Bubble diagrams       Relationships  Requirements definition  Functional relationship (for regression testing) Failure models

23. Styles of Exploration : …contd Invariances : Tests that change things that should have no impact Example: Printing a word document or a power point.

24. Styles of Exploration : …contd Interferences : Interrupt Alter Terminate Suspend Switch Contend.

25. Styles of Exploration : …contd Error Handling : Check if errors and exceptions are handled correctly. Specifications : Heuristics User Manual Active reading

26. Styles of Exploration : …contd Group Insights : Brainstormed test lists Group discussion of related components Fishbone analysis Paired Exploratory Testing Troubleshooting : Bug Analysis. Bug regression.

27. Pros and Cons of ET: Pros :- Does not require extensive documentation. Responsive to changing scenarios. Under tight schedules, testing can be more focused depending on the bug rate or risks. Improved coverage.

28. Pros and Cons of ET: Cons :- Dependent on the tester’s skills. Test tracking not concrete. More prone to human error. No contingency plan if the tester is unavailable.

29. Formalised Methods of ET What is Formalised ET ? Scripted Testing vs Formalised ET What does ET achieve ? Approaches used for Formalised ET

30. What is Formalised ET ? A structured and reasoned approach to exploratory testing is termed as Formalised ET. This approach consists of specific tasks, objectives, and deliverables that makes it a systematic process. Using the systematic approach, arrive at an outline of what to achieve what to attack what would be the scope what effort to spend Applying planning to Exploratory Testing, we have Exploratory Planning.

31. Increasing degree of formalised ET Ad hoc testing Pure Scripted testing 0 100Documentation(%) Scripted Testing Vs ET Charters Vague Scripts

32. What does Formalised ET achieve? How much planning is done with ET? How do you execute the tests with ET? What kind of documentation is produced with ET? How to use Pair Testing with ET?

33. Approaches used for Formalised ET Identify the application domain Identify the purpose Identify the primary and secondary functions Identify the workflows Identify the break points Identify the UI standards

34. Approaches used for Formalised ET …contd Identify the expected results Identify the the interfaces with other interfaces/external applications Record failures Document the issues and questions Decomposing the activites to smaller tasks Charter – states the goal and the tactics to be used

35. Approaches used for Formalised ET …contd Session Based Test Management (SBTM) It’s the concept of Testing in Sessions A session, is not a test case or bug report, is the basic testing work uninterrupted unit of reviewable, chartered test effort. It operates within a documented process structure and produces records that complement verification documentation. Reference :“Session-Based Test Management” by Jonathan Bach (first published in Software Testing and Quality Engineering magazine, 11/00)

36. Approaches used for Formalised ET …contd UnInterrupted : No significant disturbances during the testing( e.g mails, calls, meetings etc.). Reviewable : A report , typically called a Session Sheet is produced on the testing performed which can be comprehended by competent authority and client. Chartered : States the ‘What’ and ‘How’ of testing and the issues to look for.It is not suggested to make it a detailed plan.

37. Approaches used for Formalised ET …contd We can separate test sessions into broadly 3 tasks(called the TBS metrics) : Test design and Execution: Scanning the product and looking for problems. Bug investigation and Reporting: Begins when the tester stumbles on something what typically looks to be a fault. Session Setup : Setting up the test environment and improving the understanding of the product.

38. Approaches used for Formalised ET …contd SBTM uses a very generalized light weight report called the Session Sheet. A Session sheet consists of: Session charter Data files Tester name(s) Test notes Date and time started Issues Task breakdown (the TBS metrics) Bugs

39. Approaches used for Formalised ET …contd Each session is debriefed. During debriefing the manager reviews the session sheets, makes improvements to the charters, gets feedback from the testers and estimates and plans made for further sessions. Agenda for debriefing session is :PROOF Past :What happened during the session? Results :What was achieved during the session? Outlook :What still needs to be done? Obstacles :What got in the way of good testing? Feelings :How does the tester feel about all this?

40. Approaches used for Formalised ET …contd Defect Driven Exploratory Testing: Defect Driven Exploratory Testing (DDET) is a goal oriented approach focused on the critical areas identified on the Defect analysis study based on Procedural Testing results. Scripted Tests Bug Reports DDET *Reference: Defect Driven Exploratory Testing, Ananthalakshmi H.V,SISL

41. Approaches used for Formalised ET …contd DDET: The Pre-Requisites for productive DDET are : In-depth knowledge of the product. Procedural Testing has to be carried out. Defect Analysis based on Scripted Tests. Key advantages of DDET are : Tester has clear clues on the areas to be explored. No wastage of time. Goal oriented approach , hence better results.

42. Approaches used for Formalised ET …contd Scripted Testing DDET:

43. Approaches used for Formalised ET …contd Applying DDET DDET:

44. Approaches used for Formalised ET …contd Normal ET Formalized ET DDET Time Defects DDET:

45. Improvements Improvements Test Session Sheets Metrics Improved Approaches using ET Using DDET / RDET.

46. Improvements …contd Test Session Sheets Charter \ Mission Areas Covered Strategy Used Session Sheet Session and Tester details TBS metrics Test Notes Anomalies Questions Issues

47. Improvements …contd Why “Test Session Sheets” ? Documented record for regulatory purposes. Managers can have track of quantum of testing. Client can have track of billable hours. Standardized reporting mechanism across the organization. Prevents overlapping of the areas covered. Acts as a feedback medium. Provides the structure for organization and management.

48. Improvements …contd Metrics Helps determine Test Coverage details. Helps in debriefing sessions. Helps the manager to plan resource and work allocation. Provides a simple medium for traceability. Provides an insight into the quality of testing. Helps collect important data for organizational purposes. Test Metrics

49. Improvements …contd Session Name Date of session Time Duration S1 07/01/2004 09:00 Hrs 90 mins S2 S3 S4 Coverage Metrics Session Division :

50. Group User Group Acronym Testers Application Roles Description of the Role G1 Joe, Mary Improvements U1 Admin Administrator …contd G2 U2 User End user G3 G4 Group Division: User groups :