1 / 34

CS 47 23 Software Validation and Quality Assurance

CS 47 23 Software Validation and Quality Assurance. Lecture 01 Introduction. Course Instructor. Name: Dr. Xiaoyin Wang (Sean) Office: NPB 3.310 Email: xiaoyin.wang@utsa.edu Experiences Got my PhD from Peking University, China Did my postdoc in UC Berkeley

maiorano
Download Presentation

CS 47 23 Software Validation and Quality Assurance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS4723Software Validation and Quality Assurance Lecture 01 Introduction

  2. Course Instructor Name: Dr. Xiaoyin Wang (Sean) Office: NPB 3.310 Email: xiaoyin.wang@utsa.edu Experiences Got my PhD from Peking University, China Did my postdoc in UC Berkeley Worked for Microsoft (.net project), and Ensighta (a startup company at Berkeley with 7-8 people) 2

  3. Course Meetings, Web Pages, etc. Course Meetings: TR 11:30am – 12:45pm MH 3.02.18 Office Hours: TR 10am – 11:30am Course Web Page: http://xywang.100871.net/CS4723.htm 3

  4. Course Textbooks Just reference books Software Testing and Quality Assurance: Theory and Practice, by Kshirasagar Naik and Priyadarshi Tripathy Software Testing: Principles and Practice, by Srinivasan Desikan and ‎Gopalaswamy Ramesh Introduction to Software Testing, by Paul Ammann and Jeff Offutt 4

  5. Course Topics Software Testing Software Debugging Software Verification Anti-pattern Checking and Refactoring Bug Management Research Progress: Automatic Oracles Automatic Program Repair 5

  6. Grading Scheme Exams: 20% * 2 = 40% Mid-term project: 20% Final project: 20% Assignment: 10% Course participation and presentations: 10% 6

  7. Exams Time: Oct 2nd: 11:30am to 12:45pm Dec 4th: 11:30pm to 12:45pm Location: MH 3.02.18 Content Selected important topics in the course We will have a review class before exams 7

  8. Project I: Unit Testing Write unit tests for a class in existing open source software Steps Understand the source code Write one or more unit tests for a list of public methods to achieve preset test coverage Deliverable The testing code with comments 8

  9. Project II: Android UI Testing UI script testing for an Android App Steps Explore the Android App Write Testing Reports Write Scripts for Regression Testing Deliverable A testing report and the testing scripts 9

  10. Now, let’s go to the real lecture … 10

  11. Why software quality matters? Prevalent usage Billions of PCs and mobile phones Working, Chatting, Gaming, … About 1 million lines of code on a new car, MIT Tech Review 2013 Some software on almost all major or small appliances ATMs Counters at shopping centers, restaurants, … Garage Entrances … 11

  12. Why software quality matters? We are so dependent on software for critical tasks Traffic Control Trains, air traffic, … Transportation Control Ship / plane / car / elevator /… Financial transactions Bank, stock, online shopping Power supplies Electricity, Power stations, gas, … Medical equipments 12

  13. What can caused by software bugs From inconvenience to disasters Ugly presentation Not showing some results or output Unable to do operation Lost of a hour’s / day’s / week’s work… Privacy leaks, lost accounts Money loss Injuries, Life-threatening dangers Disasters 13

  14. Some well-known disasters caused by software bugs Mars Climate Orbiter ($165M, 1998) Sent to Mars to relay signal from the Mars Lander Smashed to the planet: failing to convert between English measure to metric values Shooting down of Airbus A300 (290 death, 1988) US CG-49 shoot down a Airbus A300 Misleading output of the tracking software THERAC-25 Radiation Therapy (1985) 2 cancer patients at East Texas Cancer center received fatal overdoses Miss-handling of race condition of the software in the equipment 14

  15. Facts about software bugs On average, 1-5 bugs per KLOC (thousand lines of code) In mature software More than 10 bugs in prototypes Windows2000 35MLOC 63K known bugs at the time of release 2 bugs per KLOC $59.5B loss due to bugs in US 2002 (estimation by NIST) It is not feasible to remove all bugs But try to reduce critical bugs 15

  16. Why so many bugs in software? Errors can happen in any engineering discipline Software is one of the most error-prone products of all engineering areas Requirements are often vague Software can be really complex, undecidable problems are everywhere Result Most product are put into the market with no or very few errors Almost all software in the market has some number of bugs (we will see that later) 16

  17. Absolute Software Quality Impossible Vague Requirements Free of bugs can never be verified even for some simple programs Practice on critical software The NASA example 17

  18. Each execution handles $4Billion equipments, human lives, dreams No prototypes 420k lines of code, 17 errors in 11 versions (is it a large software?) Commercial equivalent would have at least 1000 bugs NASA Example 18

  19. A third of the effort before coding starts Long specifications, written down, fully discussed 40k pages of specification (longer than the code) Change to add GPS support (2500 pages more specification) Specification is almost pseudo code NASA Example 19

  20. When fixing bugs, fix what allowed the mistake Unclear API Insufficient tests Improper use of tools Validation and review at all levels 85% of bugs revealed before testing NASA Example 20

  21. Cost 260 people $32 million A year TOO EXPENSIVE!!! Overkill for normal software NASA Example 21

  22. Quality Economics Tradeoff of cost and requirements 22

  23. Approaches to Enhance Software Quality Bug Prevention Prevent bugs during requirement collection, designing, coding Bug Removal Remove bugs after coding, but before release Bug Tolerance Runtime mechanisms to avoid known or unknown bugs in the software 23

  24. Approach to prevent bugs Requirement Engineering Remove ambiguity in requirements Formal Specification Design Reduce dependency between modules Concentrate cross-cut concerns Coding Proper coding styles Well written comments Avoid misleading interfaces 24

  25. Approach to remove bugs Testing Feed input to software and run it to see whether its behavior is as expected Limitations Impossible to cover all cases Test oracles (what is expected) Static checking Identify specific problems (e.g., memory leak) in the software by scanning the code or all possible paths Limitations Limited problem types False positives 25

  26. Approach to remove bugs Formal Proof Formally prove that the program implements the specification Limitations Difficult to have a formal specification The proof cost a lot of human efforts Inspection Manually review the code to detect faults Limitations: Hard to evaluate Sometime hard to get progress 26

  27. Approach to tolerate bugs Exception handling Specify what the program should do when unexpected results are generated Spend time on writing exception handlers Consider exceptions in the exception handler Redundant code Have redundant modules to avoid runtime bugs by checking consistency of results Voting for a correct results when bug happens 27

  28. In our course Bug prevention: something you should learn in software engineering course, briefly introduce it at the end of our course Bug removal: The focus of this course Bug tolerance We will introduce guidance for exception handling at the end of our course, redundant code will not be introduced because it is rarely used 28

  29. Aspects of software quality Dependability availability, reliability, safety Efficiency processing time, memory utilization, responsiveness Usability appropriate user interface and adequate documentation Maintainability ease of change 29

  30. Dependability (I) Availability For how large proportion of time the software system is working Recover from a crash? Reliability For how large proportion of inputs the software is giving correct outputs Fault tolerance? 30

  31. Dependability (II) Safety For critical software, e.g. insulin control To what extent the software is threatening people? Runtime fault detection? Security To what extent the software is able to protect itself from malicious inputs Input sanitization? 31

  32. Efficiency Processing time for time-consuming tasks Algorithm Complexity Optimization Maximal Memory utilization Avoid memory leaks Release memory at proper time Responsiveness Interleave UI threads and back-end threads 32

  33. Usability User Interface Looks attractive Easy to understand and convenient to use Help Documentation Easy to search and read Cover all the features of the software 33

  34. Maintainability Ease to read Code quality Comments Ease to extend Design quality Suitable design patterns Ease to transplant Avoid hard coded elements 34

More Related