Computers in Society

1. Computers in Society Lecture 7, part 2: Software Reliability

2. Today we will talk about software reliability and responsibility for software faults. I will focus on a case study of the Therac-25, a radiation treatment machine. The Therac-25 suffered from software faults that caused it to give patients large overdoses of radiation. Some patients died as a result of the overdoses. The fault was difficult to find, so the machine was used even after some patients had been seriously harmed. Assignment for Next Time

3. For Wednesday’s class session, I want each group to discuss the Therac-25 case, answer some questions, and come to class prepared to share your answers. You should address the following questions: • Many people involved with the Therac-25 made mistakes or made bad decisions. Who made mistakes/bad decisions? How much responsibility does each such person (or group of persons) have for the injuries and death that were caused? Assignment for Next Time (2)

4. If you were the chairman of AECL, the company that produced the Therac-25, what would you have done after the software bug was understood? How would you have compensated the victims? How would you have changed your organization as a result? • How would you use your knowledge of the Therac-25 case to organize software development to minimize the chances that such a situation could happen again? Assignment for Next Time (3)

5. On Wednesday each group will have 5 to 10 minutes to explain their answers to the rest of the class. Please try to keep your answers and justifications succinct so we have time for discussion. Assignment for Next Time (4)

6. The Therac-25 was a linear accelerator device used to treat cancer using radiation. It was built by Atomic Energy of Canada Limited (AECL). It could use either X-rays or electron radiation. Therac-25 was a next-generation machine based on two earlier machines that AECL had built in cooperation with CGR, a French company. The earlier machines were the Therac-6 and the Therac-20. The Therac-25

7. The Therac-6 and Therac-20 incorporated a computer (PDP-11) as a front end. The computer was a front end only. The linear accelerators could be operated independently of the computer. All safety features were built into hardware. The Therac-25 integrated the computer with the linear accelerator. The Therac-25 (2)

8. Two important changes created the possibility for problems: First, the Therac-25 reused code from the earlier machines. Second, some hardware safety features were replaced by software features. In machines with both electron and X-ray modes (dual-mode accelerators), a turntable rotates needed equipment into position to give proper doses of radiation. In older machines this was checked with hardware interlocks. Therac-25 checked this in software. The Therac-25 (3)

9. The Therac-25 (4) The Therac-25 went into service in 1983. Eleven systems were delivered. Problems began occurring in June 1985.

10. Accident 1: Marietta, Georgia, June 1985. Kennestone Regional Oncology Center (KROC) • A patient was burned by treatment and suffered crippling injuries. KROC contacted AECL and asked if the Therac-25 could have failed to diffuse the radiation beam. AECL said no. • The patient sued AECL and the hospital in October 1985. Therac-25 Problem History

11. Accident 2: Hamilton, Ontario, July 1985. Ontario Cancer Foundation. • A patient was burned during treatment. The machine shut down during treatment. The display indicated no treatment had been made. The operator tried to proceed with treatment multiple times until the machine suspended treatment. The patient complained of being burned, and was hospitalized for radiation overdose three days later. • The patient died of cancer in November 1985. Therac-25 Problem History

12. First AECL Investigation: July-September 1985 • AECL sent an engineer to investigate after the Ontario overdose. • The engineer discovered design problems related to a microswitch. • AECL introduced hardware and software changes to fix the microswitch problem. • A Canadian regulatory board requested a redesign of the handling of malfunction conditions, but AECL did not comply. Therac-25 Problem History

13. Accident 3: Yakima, Washington, December 1985. Yakima Valley Memorial Hospital. • A patient developed a pattern of striped burns as a result of treatment. The hospital staff suspected that the pattern was from slots in the accelerator’s blocking trays. • AECL claimed that neither the Therac-25 or operator error could have produced the damage. AECL also claimed that no similar accidents had been reported. • The patient survived, though she was left with scarring and a mild disability. Therac-25 Problem History

14. Accident 4: Tyler, Texas, March 1986. East Texas Cancer Center (ETCC). • During a treatment session the operator noticed she had entered an “X” (for X-ray) instead of an “E” (for electron) into the display. She quickly fixed the problem, moving the cursor to the field in error, changing it, and moving the cursor back to the bottom of the screen. The system was designed to detect that input was complete when the cursor was in the bottom right position. Therac-25 Problem History

15. Accident 4 continued: • Once the operator was ready, she started treatment. After a few seconds the system shut down and gave the error code “Malfunction 54”. The operator continued treatment. • The patient, who had had eight previous treatments, knew something was wrong because of pain he experienced. • The patient died five months later of a radiation overdose of between 80-100 times the prescribed dose. Therac-25 Problem History

16. Second AECL Investigation: March 1986. • ETCC shut down its Therac-25 after the accident and notified AECL. • AECL sent two engineers, who were unable to duplicate the problem. They claimed it was impossible for the Therac-25 to overdose a patient. They blamed the problems on the hospital’s electrical system. • ETCC found no problems with the electrical system, and put the Therac-25 back into service. Therac-25 Problem History

17. Accident 5: Tyler, Texas, April 1986. East Texas Cancer Center (ETCC). • This accident was virtually the same as accident 4. The same operator was at the controls, and made the same change. The same behavior and “Malfunction 54” occurred. ETCC shut down the machine and contacted AECL. • The patient received a massive overdose of radiation to his brain and died three weeks later. • After this incident, investigators were able to duplicate it, and the first major software bug was detected. Therac-25 Problem History

18. Therac-25 Declared Defective: May 2, 1986. • On May 2, 1986, the US Food and Drug Administration (FDA) declared the Therac-25 to be defective. • AECL was required to notify all Therac-25 customers. • To gain back FDA approval, AECL had to show how it would make the Therac-25 safe. Therac-25 Problem History

19. Accident 6: Yakima, Washington, January 1987. Yakima Valley Memorial Hospital. • A second patient developed a pattern of striped burns as a result of treatment. • The hospital staff was able to match the burn marks to the slots in the Therac-25’s blocking tray. • The patient died three months later. Therac-25 Problem History

20. Therac-25 Declared Defective: February 1987. • On February 10, 1987 the US Food and Drug Administration (FDA) declared the Therac-25 to be defective. It recommended that all machines be shut down. • To gain back FDA approval, AECL had to show how it would make the Therac-25 safe. Therac-25 Problem History

21. Therac-25 Declared Defective: February 1987. • It took five months and five plans to receive FDA approval. The final plan included hardware interlocks to prevent overdoses or activating the beam when the turntable was not in the correct position. • No accidents have been reported since. Therac-25 Problem History

22. One bug occurred because the system detected end of data entry when the cursor moved to the bottom right of the entry screen. At that point magnets for directing the beam would be positioned, which took a few seconds. After the magnets were positioned, the cursor was checked again. If it was at the bottom of the screen, no changes were detected. Therac-25 Software Bugs

23. If a fast-typing operator such as the one in Texas made a change while the magnets were moving and the restored the cursor to the bottom of the page, the changes would show on the page. However, the system would see the cursor at the bottom and not check for the changes. The previous (mistaken) data would be used. This is an example of a race condition. Therac-25 Software Bugs (2)

24. A second race condition produced the overdoses at the Yakima center. It occurred when the machine was moving the gun into position. A variable was supposed to be zero if the beam was in position to fire. Any other value meant that the beam should not fire. When the beam was not in position, the variable would be incremented steadily. Therac-25 Software Bugs (3)

25. The incrementing counter only held the values from 0-255. That meant that on occasion it reset to zero. If the operator pressed the button to fire the beam when the value reset from 255 to zero, the beam would fire even if it was not in position. This was a rare occurrence, but it could occur, and did on two occasions. Therac-25 Software Bugs (4)