Can We Trust the Computer?

1. Can We Trust the Computer? What Can Go Wrong? Case Study: The Therac-25 Increasing Reliability and Safety Perspectives on Failures, Dependence, Risk, and Progress Computer Models

2. What Can Go Wrong? • Facts About Computer Errors • Error-free software is not possible. • Errors are often caused by more than one factor. • Errors can be reduced by following good procedures and professional practices. Q: How do we distinguish between tolerable or unavoidable errors in software versus careless software development?

3. What Can Go Wrong? • The Roles of People in Computer-related Problems: • Computer User • At home or work, users should understand the limitations of computers and the need for proper training and responsible use. • Computer Professional • Understanding the source and consequences of computer failures is valuable when buying, developing, or managing a complex system. • Educated Member of Society • Personal decisions and political, social, and ethical decisions depend on understanding computer risks.

4. What Can Go Wrong? • Categories of Computer Errors and Failures • Problems for Individuals: • usually in their role as consumers. • who are incorrectly identified by inaccurate law enforcement databases. • System Failures: • affecting large numbers of people and/or costing large amounts of money. • Safety-Critical Applications: • where people may be injured or killed.

5. What Can Go Wrong? • Problems for Individuals • Billing Errors • Lack of tests for inconsistencies and inappropriate amounts. • Database Accuracy Problems • Incorrect information resulting in wrongful treatment or acts. Q: Describe a computer error or failure that has affected you.

6. What Can Go Wrong? • Problems for Individuals (cont’d) • Causes: • Large population. • Human common sense not part of automated processing. • Overconfidence in the accuracy of data from a computer. • Errors in data entry. • Information not updated or corrected. • Lack of accountability for errors.

7. What Can Go Wrong? • System Failures • Communications: • Telephone, online, and broadcast services. • Business: • Inventory and management software. • Financial: • Stock exchange, brokerages, banks, etc.. • Transportation: • Reservations, ticketing, and baggage handling.

8. What Can Go Wrong? • System Failures (cont’d) • Causes: • Insufficient testing and debugging time. • Significant changes in specifications (during and after project begun). • Overconfidence in system. • Mismanagement of the project. Q: Describe a recent system failure that affected many people or resulted in a great monetary loss.

9. What Can Go Wrong? • Safety-Critical Applications • Military • Power Plants • Aircraft • Trains • Automated Factories • Medicine • …others.

10. What Can Go Wrong? • Safety-Critical Applications • Causes: • Overconfidence. • Lack of override features. • Insufficient testing. • Sheer complexity of system. • Mismanagement. Q: What activities do you participate in that are controlled by safety-critical applications?

11. Case Study: The Therac-25 • The Therac-25 was a software-controlled radiation-therapy machine used to treat people with cancer. • Overdoses of radiation • Normal dosage is 100–200 rads. • It is estimated that 13,000 and 25,000 rads were given to six people. • Three of the six people died. Q: What determines whether the risks associated with a safety-critical application are acceptable?

12. Case Study: The Therac-25 • Therac-25 Radiation Overdose • Multiple Causes: • Poor safety design. • Insufficient testing and debugging. • Software errors. • Lack of safety interlocks. • Overconfidence. • Inadequate reporting and investigation of accidents.

13. Increasing Reliability and Safety • What Goes Wrong? • Computer Systems Fail Because: • The job they are doing is inherently difficult, and • The job is done poorly. • Compounding the Reliability Issue: • Developers and users exhibit overconfidence in the system. • Reused system software may not work in different environments. Q: Identify the elements needed as an incentive to increase reliability and safety.

14. Increasing Reliability and Safety • Professional Techniques • Follow good software practices. • Exhibit professional responsibility at all levels of development and use. • Construct well-designed user interfaces. • Take human factors into account. • Include built-in redundancy. • Incorporate self-checking where appropriate. • Follow good testing principals and techniques. Q: What human interface features should be considered for ordinary business applications?

15. Increasing Reliability and Safety • Law and Regulation • Criminal and Civil Penalties: • to recover loss from faulty or unsafe systems. • Liability and Civil Laws: • to provide incentives to produce reliable and safe systems. • Warranties: • to guarantee a certain level of quality. • Federal or State Regulations: • to protect the public.

16. Increasing Reliability and Safety • Law and Regulation (cont’d) • Database Accuracy Enforcement: • to protect the public from inaccurate information maintained by private companies and government. • Mandatory Licensing of Software Developers: • to ensure proper training, competency, and continuing education. Q: How can consumers protect themselves from faulty software?

17. Perspectives on Failures, Dependence, Risk, and Progress • Failures • What are acceptable rates of failures? • How accurate should software be? • Dependence • How dependent on computer systems are our ordinary activities? • How useful are computer systems to our ordinary activities? • Risk and Progress • How do new technologies become safer? • Can progress in software safety keep up with the pace of change in computer technology?

18. Computer Models • Points to Consider: • Models are simplifications of either physical or intangible systems. • Those who design and develop models must be honest and accurate with results. • Computer professionals and the general public must be able to evaluate the claims of the developers. Q: What problems in your community have been or could be studied with computer models?

19. Computer Models • Evaluating Models • Why Models Might Not Be Accurate: • Developers have incomplete knowledge of the system being modeled. • Data might be incomplete or inaccurate. • Power of the computer might be inadequate. • Variables are difficult to numerically quantify. • Political and economic motivation to distort results. Q: For each item above, give an example of a model or simulation that was inaccurate.

20. Computer Models • Evaluating Models (cont’d) • Regarding the Car-Crash Models Described in the Text: • How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? • What are the assumptions and simplifications in the model? • Do the results or predictions correspond with the real world?

21. Computer Models • Evaluating Models (cont’d) • Regarding the Climate Models Discussed in the Text: • How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? • What are the assumptions and simplifications in the model? • Do the results or predictions correspond with the real world?