Teaching about Software Testing that is NOT for Defect Detection

1. Teaching about Software Testing that is NOT for Defect Detection W. Morven Gentleman Dalhousie University Morven.Gentleman@dal.ca

2. Conventional Perspective • Software is tested on behalf of developer • Purpose of testing is to find (and fix) bugsThis is an important perspective, but not the only one

3. Other perspectives:how unusual are they? • Testing to meet regulatory or contractual requirements • Testing a software artifact to measure and model quantitative attributes of the software • Testing to measure human behaviour when interacting with the software under test • Testing where the investigation is open-ended, to find questions to ask

4. Measuring quantitative attributes • Benchmarking • Sizing and capacity planning • Tuning performance to observed load

5. Measuring human behaviour • Usability testing • Assessing training requirements • Assessing change management activities • Studying tool adoption

6. Open-ended investigation • Validating a system model or specification • Studying competitors’ products • Interoperability testing • Identifying ways to use a product

7. Observations on teaching • Students have great difficulty assimilating experimental techniques and experimental evidence. They cannot • conceive of interesting questions. • propose experiments.to answer such questions • suggest analyses of experimental data. • interpret experimental results. • recognize assumptions contradicted empirically • notice anomalies that indicate the unsuspected . • manage an investment of experimental effort.

8. Diagnosis • CS/SE/IS students get no exposure to the culture of empirical science • the fundamental precept that all asserted truth must ultimately be founded on experimental observation • theory should be seen as an approximation summarizing experimental evidence • They are given a mathematical model of a problem, and they are unable to distinguish that model from physical reality • Remedy?

9. Some topics students find novel • Practical instrumentation • Choosing what to measure and what scale to use • Characterizing load • Systematic sources of measurement artifact • Practical application of statistics • Statistical vs.. practical significance • Exploratory data analysis • Modeling • Experimental design

10. Real Industrial Example

11. Time to update in-memory Oracle Database – 1 try • Specification requires < 0.5 seconds • Vendor demonstration 0.472 seconds

12. Time to update in-memory Oracle Database – 100 tries

13. Time to update in-memory Oracle Database – 5000 tries

14. Another real example

15. Time to open an Excel file

16. View of all software testing as searching for defects • Does it help in designing experiments and analysis? • Are defect detection strategies appropriate in other situations?

17. Definition of defect • “Operationally, it is useful to work with 2 definitions of a defect: • From the producers viewpoint: a product requirement that has not been met or an attribute of a product that is not in the statement of requirements that define the product. • From the customers viewpoint: anything that causes customer dissatisfaction, whether in the statement of requirements or not”

18. Conclusion A broader perspective on software testing can be applicable to CS/SE/IS students in careers with soft6ware developers and software customers