1 / 32

Software Testing - a necessary evil

Software Testing - a necessary evil. SQNZ April 2002 Michael Osborne mike@osborne.gen.nz. Does this guy know what he’s talking about?. Testing is about bugs This guy knows about bugs In 25+ years of development he’s created thousands of them Now, I pay other people to do this. Simply.

MikeCarlo
Download Presentation

Software Testing - a necessary evil

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Software Testing - a necessary evil SQNZ April 2002 Michael Osborne mike@osborne.gen.nz

  2. Does this guy know what he’s talking about? • Testing is about bugs • This guy knows about bugs • In 25+ years of development he’s created thousands of them • Now, I pay other people to do this

  3. Simply • Quidquid latine dictum sit, altum sonatur. • Whatever is said in Latin sounds profound. • Sedulously eschew obfuscatory hyperverbosity and prolixity. • Let’s all keep it as simple as possible. • If you don’t understand - interrupt! From - How To Write Unmaintainable Code http://mindprod.com/unmain.html

  4. What’s going on here? • Why this presentation? • enquire vs. inform • tomorrow vs. yesterday • quality vs “quality” • continuous improvement • how to get and give good testing bang for buck From - Joel on Software http://www.joelonsoftware.com

  5. The Principles Quality Software Quality Quality & Testing The Necessity The Evil Techniques Types of testing Testing 2002 If time permits we’ll do some testing What we will cover

  6. Why it’s necessary • Analysts & developers are imperfect • Capers Jones - bug input is fps to power of 1.2 • there is no perfect prevention - yet • you can’t ship (successfully) software that doesn’t work • it’s a process measure • in a highly sophisticated environment • it goes from primarily being a QC tool • to a primarily being process measure

  7. Why it’s evil • You can’t test quality in • You can’t test it all • You can’t prove it works • It’s costly and inefficient • The point is to prove it doesn’t work • The more problems found the better

  8. Testing Explained • Testing is about finding bugs

  9. Which part don’t you understand? • If you’re not going to find any bugs why are you doing it? “A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.” • Testing Computer Software by Kaner, Falk, Nguyen

  10. It’s an Economics Problem • Cost of Quality = Cost of Conformance + Cost of Non-Conformance • Testing adds to Cost of Conformance • It must directly reduce Cost of Non-Conformance or else it is waste • Testing to do anything other than find bugs is a WOMBAT • a Waste of Money Brains and Time

  11. How it works - roughly

  12. When CoNC is high... • It’s appropriate to raise the CoC

  13. High “quality” software • It is possible to develop very high quality software • An example • NASA space shuttle on-board flight software • Approx. 3 million lines of code • <1 error per 10k LOC after release • Cost: the order of $1000 / LOC

  14. The “Q” word • quality vs. “quality” • it’s important we’re clear about this • it will show the necessity for Software Testing • it will expose some evils of Software Testing • So, what is quality? • And, what is “quality”?

  15. What is software development? • Putting bugs into software • ie. hiding bugs in software • Finding the bugs • Taking them out again • Putting (hiding) new / different bugs in when you take others out • Finding those bugs • Taking them out again • And so on ...

  16. What is better software development? • Putting fewer bugs into software • exposing bugs in software • Finding the important bugs faster • Taking important bugs out again • assessing the associated risk • Putting fewer new / different bugs in when you take others out • And so on … • Today’s focus is on Software Testing

  17. What is better software testing? • What is quality software testing? • Quality is … • conformance to requirements • Note: test cases are not test requirements • Finding the important bugs faster? • which leads to the critical question -

  18. When is testing done? • Typically - • when the testing schedule ends • when all the test cases have been exercised • when all the bugs have been found and fixed • If you don’t have a set of testing requirements • How do you answer the question?

  19. The Zero Defects Approach • Quality = conformance to requirements • Test exhaustively • Find all defects • Fix all defects • Result - quality software • a quick digression into Zero Defects • a personal view

  20. Direction vs. Destination • Current state - some level of defects • Desired state • more defects • same level of defects • fewer defects • An asymptote • Note “six sigma” is not zero • “Zero defects” is a pointer and an accelerator • use it that way

  21. Uh oh - there’s more than one requirement!! • Quality = conformance to requirements • There are generally three sets of requirements • Functionality • Schedule • Quality • The bad news • they compete

  22. The Good-Enough Software Triangle • Conformance to all requirements Schedule Delivery Date Functionality Features Quality Absence of Defects

  23. Software Testing - A Wicked Problem • A “mess” is a complex problem with a “simple” solution • A “wicked problem” is a “simple” problem with a complex solution • usually because of stakeholder differences • Each project (testing problem) potentially has a different solution

  24. So? • How do we do it?

  25. Software Testing 2002 • Context-driven testing • all at www.satisfice.com • Exploratory testing • heuristics • short bursts • focus on results • purposeful wandering • you must think

  26. white box - glass box, structural path testing - coverage black box top-down, bottom-up unit function system integration regression performance alpha beta compatability quicky port usability exploratory hallway others ... Testing Types

  27. Later Test Phase Middle Phase ? ? Frequent Annoyances - Remove Ignore Frequent Niggles - ? Bug removal strategy Important Find & Fix Immediately Ignore Urgent

  28. Usability Testing • Graphical User Interfaces coupled with complex processes create usability issues • what’s clear and obvious to designer/programmer is meaningless to users • Doesn’t require a lot of people • 5 to 7 is adequate • Observe, and listen • understand how they see things • why they do what they do • Is your interface clear and consistent?

  29. Why usability is critical • It doesn’t matter if it processes everything right • if they don’t like it (can’t/won’t use it, it failed) • Prototype, prototype, prototype • Consistent vs. clever • at all times • Users will be forgiving with a highly usable system • ask Microsoft

  30. Hallway Testing • A simple form of usability testing • Open up development to interact more with users • Approach a person (in your organisation) at random • invite for 5-10 minute testing session • see what they do with your software from no background • is it understandable, usable

  31. The evil necessity • It’s all about finding bugs • A good test is a test that finds a bug • Be clear about what represents quality • situation by situation • Establish the context • and be guided by it • Good software testing is a challenging intellectual process. • Go for it!

More Related