CS 51 23 Software Validation and Quality Assurance

1. CS5123Software Validation and Quality Assurance Lecture 01 Introduction

2. Course Instructor Name: Dr. Xiaoyin Wang (Sean) Office: NPB 3.208 Email: xiaoyin.wang@utsa.edu Experiences Got my PhD from Peking University, China Did my postdoc in UC Berkeley Worked for Microsoft (.net project), and Ensighta (a startup company at Berkeley with 7-8 people) 2

3. Course Meetings, Web Pages, etc. Course Meetings: MW 6:00pm – 7:15pm MH 3.02.24 Office Hours: W 3:00pm - 3:30pm, 4:30pm – 5:00pm Course Web Page: http://xywang.100871.net/CS5123.htm 3

4. Course Textbooks Just reference books Software Testing and Quality Assurance: Theory and Practice, by Kshirasagar Naik and Priyadarshi Tripathy Software Testing: Principles and Practice, by Srinivasan Desikan and ‎Gopalaswamy Ramesh Introduction to Software Testing, by Paul Ammann and Jeff Offutt 4

5. Course Topics Software Testing Software Debugging Software Verification Anti-pattern Checking and Refactoring Bug Management Research Progress: Automatic Oracles Automatic Program Repair 5

6. Grading Scheme Exams: 20% * 2 = 40% Projects: 25% * 2 = 50% Course participation and presentations: 10% 6

7. Final Exam Time: Oct 16th: 6:00pm to 7:15pm Dec 6th: 6:00pm to 7:15pm Location: MH 3.02.24 Content Selected important topics in the course We will have a review class before the exam class 7

8. Project I: Unit Testing Write unit tests for a class in existing open source software Steps Understand the source code Write one or more unit tests for a list of public methods to achieve preset test coverage Deliverable The testing code with comments 8

9. Project II: Android UI Testing UI script testing for an Android App Steps Explore the Android App Write Testing Reports Write Scripts for Regression Testing Deliverable A testing report and the testing scripts 9

10. Now, let’s go to the real lecture … 10

11. Why software quality matters? Prevalent usage Billions of PCs and mobile phones Chatting, Surfing, Gaming, … About 1 million lines of code on a new car, MIT Tech Review 2013 Some software on almost all major or small appliances ATMs Counters at shopping centers, restaurants, … Garage Entrances … 11

12. Why software quality matters? We are so dependent on software for critical tasks Traffic Control Trains, air traffic, … Transportation Control Ship / plane / car / elevator /… Financial transactions Bank, stock, online shopping Power supplies Electricity, Power stations, gas, … Medical equipments 12

13. What can caused by software bugs From inconvenience to disasters Ugly presentation Not showing some results or output Unable to do operation Lost of a hour’s / day’s / week’s work… Privacy leaks, lost accounts Money loss Injuries, Life-threatening dangers Disasters 13

14. Some well-known disasters caused by software bugs Mars Climate Orbiter ($165M, 1998) Sent to Mars to relay signal from the Mars Lander Smashed to the planet: failing to convert between English measure to metric values Shooting down of Airbus A300 (290 death, 1988) US CG-49 shoot down a Airbus A300 Misleading output of the tracking software THERAC-25 Radiation Therapy (1985) 2 cancer patients at East Texas Cancer center received fatal overdoses Miss-handling of race condition of the software in the equipment 14

15. Facts about software bugs On average, 1-5 bugs per KLOC (thousand lines of code) In mature software More than 10 bugs in prototypes Windows2000 35MLOC 63K known bugs at the time of release 2 bugs per KLOC $59.5B loss due to bugs in US 2002 (estimation by NIST) It is not feasible to remove all bugs But try to reduce critical bugs 15

16. Why so many bugs in software? Errors can happen in any engineering discipline Software is one of the most error-prone products of all engineering areas Requirements are often vague Software can be really complex, undecidable problems are everywhere Result Most product are put into the market with no or very few errors Almost all software in the market has some number of bugs (we will see that later) 16

17. Aspects of software quality Dependability availability, reliability, safety Efficiency processing time, memory utilization, responsiveness Usability appropriate user interface and adequate documentation Maintainability ease of change 17

18. Dependability (I) Availability For how large proportion of time the software system is working Recover from a crash? Reliability For how large proportion of inputs the software is giving correct outputs Fault tolerance? 18

19. Dependability (II) Safety For critical software, e.g. insulin control To what extent the software is threatening people? Runtime fault detection? Security To what extent the software is able to protect itself from malicious inputs Input sanitization? 19

20. Efficiency Processing time for time-consuming tasks Algorithm Complexity Optimization Maximal Memory utilization Avoid memory leaks Release memory at proper time Responsiveness Interleave UI threads and back-end threads 20

21. Usability User Interface Looks attractive Easy to understand and convenient to use Help Documentation Easy to search and read Cover all the features of the software 21

22. Maintainability Ease to read Code quality Comments Ease to extend Design quality Suitable design patterns Ease to transplant Avoid hard coded elements 22

23. Absolute Software Quality Impossible Vague Requirements Free of bugs can never be verified even for some simple programs Practice on critical software The NASA example 23

24. Each execution handles $4Billion equipments, human lives, dreams No prototypes 420k lines of code, 17 errors in 11 versions (is it a large software?) Commercial equivalent would have at least 1000 bugs NASA Example 24

25. A third of the effort before coding starts Long specifications, written down, fully discussed 40k pages of specification (longer than the code) Change to add GPS support (2500 pages more specification) Specification is almost pseudo code NASA Example 25

26. When fixing bugs, fix what allowed the mistake Unclear API Insufficient tests Improper use of tools Validation and review at all levels 85% of bugs revealed before testing NASA Example 26

27. Cost 260 people $32 million A year TOO EXPENSIVE!!! Overkill for normal software NASA Example 27

28. Quality Economics Tradeoff of cost and requirements 28

29. Approaches to Enhance Software Quality Bug Prevention Prevent bugs during requirement collection, designing, coding Bug Removal Remove bugs after coding, but before release Bug Tolerance Runtime mechanisms to avoid known or unknown bugs in the software 29

30. Approach to prevent bugs Requirement Engineering Remove ambiguity in requirements Formal Specification Design Reduce dependency between modules Concentrate cross-cut concerns Coding Proper coding styles Well written comments Avoid misleading interfaces 30

31. Approach to remove bugs Testing Feed input to software and run it to see whether its behavior is as expected Limitations Impossible to cover all cases Test oracles (what is expected) Static checking Identify specific problems (e.g., memory leak) in the software by scanning the code or all possible paths Limitations Limited problem types False positives 31

32. Approach to remove bugs Formal Proof Formally prove that the program implements the specification Limitations Difficult to have a formal specification The proof cost a lot of human efforts Inspection Manually review the code to detect faults Limitations: Hard to evaluate Sometime hard to get progress 32

33. Approach to tolerate bugs Exception handling Specify what the program should do when unexpected results are generated Spend time on writing exception handlers Consider exceptions in the exception handler Redundant code Have redundant modules to avoid runtime bugs by checking consistency of results Voting for a correct results when bug happens 33

34. In our course Bug prevention: something you should learn in software engineering course, briefly introduce it at the end of our course Bug removal: The focus of this course Bug tolerance We will introduce guidance for exception handling at the end of our course, redundant code will not be introduced because it is rarely used 34

35. A short review Software bugs affect every aspect of our life Software Bugs may cause severe consequences Software Bugs are prevalent Aspects of software quality Dependability, Efficiency, Usability, Maintainability (Bad smells -> bugs) Approaches to enhance software quality Bug prevention, Bug removal, Bug avoiding 35