1 / 44

Just-In-Time Testing

Just-In-Time Testing. Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com. Overview. Some Philosophy Context Drivers Turbulence Skills Testing Ideas Test Triage. Just In Time Testing. Robert Sabourin , Software Evangelist President AmiBug.Com Inc.

everly
Download Presentation

Just-In-Time Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Just-In-Time Testing Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com © Robert Sabourin, 2008

  2. Overview Some Philosophy Context Drivers Turbulence Skills Testing Ideas Test Triage © Robert Sabourin, 2008

  3. Just In Time Testing • Robert Sabourin , Software Evangelist • President • AmiBug.Com Inc. • Montreal, Quebec, Canada • rsabourin@amibug.com © Robert Sabourin, 2008

  4. Fundamental Question • How do you know when you are finished? Some Philosophy © Robert Sabourin, 2008

  5. Gerald M. Weinberg “Quality is value to some person” Exploring Requirements Quality Before Design Dorset House Some Philosophy © Robert Sabourin, 2008

  6. Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” Some Philosophy © Robert Sabourin, 2008

  7. C. Northcote Parkinson Parkinson’s Law: “…work expands so as to fill the time available for its completion…” Some Philosophy © Robert Sabourin, 2008

  8. Steve Covey “…begin with the end in mind … “…first things first …" Some Philosophy © Robert Sabourin, 2008

  9. Purpose of Testing Common definition: To find bugs before our customers do! Broader definition: The role of testing is to provide objective input to facilitate business decisions! Keeps stakeholders aware of all issues or concerns that relate to shipping a product! Some Philosophy © Robert Sabourin, 2008

  10. JIT Core Principals • RRR – Keep it Real Recent & Relevant • Purposeful Testing – On Time On Quality On Budget are meaningless if you are not on Purpose • Fundamental Question of Software Engineering: How do you know you are finished? • First Things First: Begin with the end in mind. • BTO - Context Drivers: Business Technology Organization • Be Prepared – Be ready for anything • Testing is Intractable: Tests cannot show software is bug free • Collect new test Ideas from many sources always • Triage Ruthlessly • Decide what not to test on purpose • Bugs are concerns about the project • There is no try: Do or Do Not • It’s all about people – and the occasional bug! Core Values © Robert Sabourin, 2008

  11. Context Drivers - BTO • Business • Value • To whom? • Why? • Technology • Solutions • Organization • Corporate Structure • Team Structure • Roles and Responsibilities Context © Robert Sabourin, 2008

  12. Context © Robert Sabourin, 2008

  13. Just-In-Time Testing Turbulence Turbulence © Robert Sabourin, 2008

  14. Just-In-Time Testing Unprepared Test Skills © Robert Sabourin, 2008

  15. Just-In-Time Testing Sharpen Testing Skills Thinker Detective Reporter Diplomat Negotiator Cheer Leader Pragmatist Test Skills © Robert Sabourin, 2008

  16. Just-In-Time Testing So what exactly did they throw over the wall? Test Focus © Robert Sabourin, 2008

  17. First Things First • Begin with the end in mind • Gain Consensus • Goals • How do we know we are finished? • Purpose • Why are we doing this project? • How will be react to change? • Meaning • What is a bug? • What is a test? Test Focus © Robert Sabourin, 2008

  18. Be Prepared We have precious little time to run tests! We must always be prepared! Test Triage © Robert Sabourin, 2008

  19. Time Test Triage © Robert Sabourin, 2008

  20. Yoda "No! Try not, Do. Or do not. There is no try." Test Triage © Robert Sabourin, 2008

  21. Testing Ideas • Collect all testing ideas you can find! • List • Sort • Organize • Shuffle Test Triage © Robert Sabourin, 2008

  22. Testing Ideas • How to find them? • Does system do what it is suppose to do? • Does the system do things it is not supposed to? • How can the system break? • How does the system react to it’s environment? • What characteristics must the system have? • Why have similar systems failed? • How have previous projects failed? Test Triage © Robert Sabourin, 2008

  23. Testing Ideas • Collect testing ideas • From testing ideas build a series of testing objectives • Each can be assigned as work to testers • Each can include all, part of, or multiple testing ideas Capture testing ideas © Robert Sabourin, 2008

  24. Testing Ideas • I often use Index Cards • Unique id • One testing idea per card • Colour indicates source • Shuffled and reviewed • Organized and reorganized • Sorted, grouped, prioritized and collected Capture testing ideas © Robert Sabourin, 2008

  25. Test Idea Sources • Capabilities • Failure Modes • Quality Factors • Usage Scenarios • Creative Ideas • States • Data • Environments • White Box • Taxonomies Capture testing ideas © Robert Sabourin, 2008

  26. Testing Ideas • Investigative approaches • We become truffle snorting pigs and try to find useful information in all evidence we discover • We can even get good ideas from out of date sources Capture testing ideas © Robert Sabourin, 2008

  27. Testing Ideas • Requirements • Use cases • Functional requirements • Quality factors • Constraints • Written requirements • Implicit requirements Capture testing ideas © Robert Sabourin, 2008

  28. Testing Ideas • Failure Modes • What can break? • Reaction to invalid input? • How does software behave in constrained environment? • Memory • Disk Space • Network Bandwidth • CPU capacity • Shared resources • Stress, Load, Volume Capture testing ideas © Robert Sabourin, 2008

  29. Capture testing ideas Quality Factors Importance Different Application Types © Robert Sabourin, 2008

  30. Testing Ideas • Usage Scenarios • Identify classes of users • Identify how users will use system • Describe scenarios • Use Story board or similar approaches • Identify variations Capture testing ideas © Robert Sabourin, 2008

  31. Testing Ideas • Creative approaches • Action verbs • Mind Maps • Soap Operas • Lateral Thinking Capture testing ideas © Robert Sabourin, 2008

  32. State Models Testing Ideas power up service needed idle inserting coins reset button coin inserted coin return cup removed no cups OR no coffee OR sensor jam right amount entered coin return make coffee user choose Capture testing ideas button pushed © Robert Sabourin, 2008

  33. Testing Ideas • Data • Flow • Structure • Create • Update • Change Capture testing ideas © Robert Sabourin, 2008

  34. Testing Ideas • Environment • Hardware • Software • Operating systems • Locales • Browsers • Plug-ins • Co-dependent software Capture testing ideas © Robert Sabourin, 2008

  35. Testing Ideas • White Box • Design • Internal structure • Code Capture testing ideas © Robert Sabourin, 2008

  36. Testing Ideas • Bug taxonomies • Collections of possible bugs • Appendix A of Testing Computer Software, Kaner, Falk, Nguyen • Boris Biezer Taxonomy Otto Vinter manages • Shopping cart taxonomy Giri Vijayaraghavan Capture testing ideas © Robert Sabourin, 2008

  37. Which test? • Impact estimation • For each test idea guesstimate: • benefit of implementation • consequence of implementation • benefit for not implementing • consequence of not implementing • How credible is the information? Triage testing ideas © Robert Sabourin, 2008

  38. Which test? Test Idea Rejection – What If? • If the cost/benefit does not make business sense then consider implementing: • part of the test, could that lead to part of the benefit at a more reasonable cost? • more than the stated test, would that generate more benefit? • a different test than the stated idea, could that generate more benefit for less cost? Triage testing ideas © Robert Sabourin, 2008

  39. Which test is next? • Questions • Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next? • How much effort are we willing to spend continuing to test this project? • Can we ship yet? Triage testing ideas © Robert Sabourin, 2008

  40. Which test is next? • Magic crystal ball • If it existed how would you use it? • What question would you ask? • What question would it ask? Triage testing ideas © Robert Sabourin, 2008

  41. Deciding what not to test? • Time pressure • Should we skip a test? • If test failed could system still be of value to some stakeholder? • If test was skipped could important bugs have been otherwise found? Triage testing ideas © Robert Sabourin, 2008

  42. Test Triage • Test Triage Meeting • Review Context • Business • Technical • Information since last triage • Test results • Bug results • New testing ideas Triage testing ideas © Robert Sabourin, 2008

  43. Test Triage • Allocate Testing Assignments to Testers • Make sure testers know context • Best thing to test • Best person to test it • Best people to explore it • Best lead • Are subject matter experts required? • Blend scripted & exploratory testing Triage testing ideas © Robert Sabourin, 2008

  44. Bottom Line • My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it! • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now? Get Started Right © Robert Sabourin, 2008

More Related